simple_lmk: Introduce Simple Low Memory Killer for Android
Subsystems (14)
RFC 1/1 simple_lmk: Introduce Simple Low Memory Killer for Android
From: Sultan Alsawaf <[email protected]>
This is a complete low memory killer solution for Android that is small and simple. It kills the largest, least-important processes it can find whenever a page allocation has completely failed (right after direct reclaim). Processes are killed according to the priorities that Android gives them, so that the least important processes are always killed first. Killing larger processes is preferred in order to free the most memory possible in one go.
Simple LMK is integrated deeply into the page allocator in order to catch exactly when a page allocation fails and exactly when a page is freed. Failed page allocations that have invoked Simple LMK are placed on a queue and wait for Simple LMK to satisfy them. When a page is about to be freed, the failed page allocations are given priority over normal page allocations by Simple LMK to see if they can immediately use the freed page.
Additionally, processes are continuously killed by failed small-order page allocations until they are satisfied.
--- drivers/android/Kconfig | 28 ++++ drivers/android/Makefile | 1 + drivers/android/simple_lmk.c | 301 +++++++++++++++++++++++++++++++++++ include/linux/sched.h | 3 + include/linux/simple_lmk.h | 11 ++ kernel/fork.c | 3 + mm/page_alloc.c | 13 ++ 7 files changed, 360 insertions(+) create mode 100644 drivers/android/simple_lmk.c create mode 100644 include/linux/simple_lmk.h
drivers/android/Kconfig +28 -0
@@ -54,6 +54,34 @@ config ANDROID_BINDER_IPC_SELFTEST exhaustively with combinations of various buffer sizes and alignments. +config ANDROID_SIMPLE_LMK + bool "Simple Android Low Memory Killer" + depends on !MEMCG + ---help--- + This is a complete low memory killer solution for Android that is + small and simple. It is integrated deeply into the page allocator to + know exactly when a page allocation hits OOM and exactly when a page + is freed. Processes are killed according to the priorities that + Android gives them, so that the least important processes are always + killed first. + +if ANDROID_SIMPLE_LMK + +config ANDROID_SIMPLE_LMK_MINFREE + int "Minimum MiB of memory to free per reclaim" + default "64" + help + Simple LMK will free at least this many MiB of memory per reclaim. + +config ANDROID_SIMPLE_LMK_KILL_TIMEOUT + int "Kill timeout in milliseconds" + default "50" + help + Simple LMK will only perform memory reclaim at most once per this + amount of time. + +endif # if ANDROID_SIMPLE_LMK + endif # if ANDROID endmenu
drivers/android/Makefile +1 -0
@@ -3,3 +3,4 @@ ccflags-y += -I$(src) # needed for trace events obj-$(CONFIG_ANDROID_BINDERFS) += binderfs.o obj-$(CONFIG_ANDROID_BINDER_IPC) += binder.o binder_alloc.o obj-$(CONFIG_ANDROID_BINDER_IPC_SELFTEST) += binder_alloc_selftest.o +obj-$(CONFIG_ANDROID_SIMPLE_LMK) += simple_lmk.o
drivers/android/simple_lmk.c new file +301 -0
301 lines not on this page · raw message
include/linux/sched.h +3 -0
@@ -1199,6 +1199,9 @@ struct task_struct {unsigned long lowest_stack;unsigned long prev_lowest_stack;#endif +#ifdef CONFIG_ANDROID_SIMPLE_LMK +bool slmk_sigkill_sent; +#endif /* * New fields for task_struct should be added above here, so that
include/linux/simple_lmk.h new file +11 -0
@@ -0,0 +1,11 @@ +/* SPDX-License-Identifier: GPL-2.0 +* +* Copyright (C) 2019 Sultan Alsawaf <[email protected]>. +*/ +#ifndef _SIMPLE_LMK_H_ +#define _SIMPLE_LMK_H_ + +struct page *simple_lmk_oom_alloc (unsigned int order,int migratetype); +bool simple_lmk_page_in (struct page *page,unsigned int order,int migratetype); + +#endif /* _SIMPLE_LMK_H_ */
kernel/fork.c +3 -0
@@ -1881,6 +1881,9 @@ static __latent_entropy struct task_struct *copy_process( p->sequential_io =0 ; p->sequential_io_avg =0 ;#endif +#ifdef CONFIG_ANDROID_SIMPLE_LMK + p->slmk_sigkill_sent = false; +#endif /* Perform scheduler related setup. Assign this task to a CPU. */ retval =sched_fork (clone_flags, p);
mm/page_alloc.c +13 -0
@@ -67,6 +67,7 @@#include <linux/lockdep.h> #include <linux/nmi.h> #include <linux/psi.h> +#include <linux/simple_lmk.h> #include <asm/sections.h> #include <asm/tlbflush.h>
@@ -967,6 +968,11 @@ static inline void __free_one_page(struct page *page, } } +#ifdef CONFIG_ANDROID_SIMPLE_LMK +if (simple_lmk_page_in (page, order, migratetype)) +return ; +#endif +list_add (&page->lru, &zone->free_area[order].free_list[migratetype]); out: zone->free_area[order].nr_free++;
@@ -4427,6 +4433,13 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,if (costly_order && !(gfp_mask & __GFP_RETRY_MAYFAIL)) goto nopage; +#ifdef CONFIG_ANDROID_SIMPLE_LMK + page =simple_lmk_oom_alloc (order, ac->migratetype); +if (page) +prep_new_page (page, order, gfp_mask, alloc_flags); + goto got_pg; +#endif +if (should_reclaim_retry (gfp_mask, order, ac, alloc_flags, did_some_progress >0 , &no_progress_loops)) goto retry;
Signature · 1 line
2.21.0
Greg Kroah-Hartman
maintainerQuoted 18 lines · Sultan Alsawaf
From: Sultan Alsawaf <[email protected]>
This is a complete low memory killer solution for Android that is small and simple. It kills the largest, least-important processes it can find whenever a page allocation has completely failed (right after direct reclaim). Processes are killed according to the priorities that Android gives them, so that the least important processes are always killed first. Killing larger processes is preferred in order to free the most memory possible in one go.
Simple LMK is integrated deeply into the page allocator in order to catch exactly when a page allocation fails and exactly when a page is freed. Failed page allocations that have invoked Simple LMK are placed on a queue and wait for Simple LMK to satisfy them. When a page is about to be freed, the failed page allocations are given priority over normal page allocations by Simple LMK to see if they can immediately use the freed page.
Quoted 4 lines
Additionally, processes are continuously killed by failed small-order page allocations until they are satisfied.
Signed-off-by Sultan Alsawaf <[email protected]>
Wait, why? We just removed the in-kernel android memory killer, we don't want to add another one back again, right? Android Go devices work just fine with the userspace memory killer code, and those are "low memory" by design.
Why do we need kernel code here at all?
thanks,
greg k-h
Quoted 29 lines · Greg Kroah-Hartman
Quoted 18 lines · Sultan Alsawaf
From: Sultan Alsawaf <[email protected]>
This is a complete low memory killer solution for Android that is small and simple. It kills the largest, least-important processes it can find whenever a page allocation has completely failed (right after direct reclaim). Processes are killed according to the priorities that Android gives them, so that the least important processes are always killed first. Killing larger processes is preferred in order to free the most memory possible in one go.
Simple LMK is integrated deeply into the page allocator in order to catch exactly when a page allocation fails and exactly when a page is freed. Failed page allocations that have invoked Simple LMK are placed on a queue and wait for Simple LMK to satisfy them. When a page is about to be freed, the failed page allocations are given priority over normal page allocations by Simple LMK to see if they can immediately use the freed page.
Quoted 4 lines
Additionally, processes are continuously killed by failed small-order page allocations until they are satisfied.
Signed-off-by Sultan Alsawaf <[email protected]>Wait, why? We just removed the in-kernel android memory killer, we don't want to add another one back again, right? Android Go devices work just fine with the userspace memory killer code, and those are "low memory" by design.
Quoted 5 lines
Why do we need kernel code here at all?
thanks,
greg k-h
Hi Greg,
Thanks for replying. It has not been my experience and the experience of many others that Android's userspace low memory memory killer works "just fine." On my Pixel 3 XL with a meager 4GB of memory, the userspace killer has had issues with killing too many processes, which has resulted in a noticeably poor user experience for all Pixel owners. From the looks of lmkd on the master branch, there still isn't really any definitive solution for this, aside from a 100ms delay in between process kills.
I think that the userspace low memory killer is more complex than necessary, especially since in the kernel we can detect exactly when we run out of memory and react far more quickly than any userspace daemon.
The original reasoning behind why the old kernel low memory killer was removed is also a bit vague to me. It just seemed to be abandonware, and all of a sudden a userspace daemon was touted as the solution.
This driver is like an Android-flavored version of the kernel's oom killer, and has proven quite effective for me on my Pixel. Processes are killed exactly when a page allocation fails, so memory use is maximized. There is no complexity to try and estimate how full memory is either.
Thanks,
Sultan
On Sun, Mar 10, 2019 at 01:34:03PM -0700, Sultan Alsawaf wrote:
[...]
Quoted 28 lines
kernel/fork.c +0
-0/* Perform scheduler related setup. Assign this task to a CPU. */ retval =sched_fork (clone_flags, p);mm/page_alloc.c +6
-0@@ -67,6 +67,7 @@#include <linux/lockdep.h> #include <linux/nmi.h> #include <linux/psi.h> +#include <linux/simple_lmk.h> #include <asm/sections.h> #include <asm/tlbflush.h> @@ -967,6 +968,11 @@ static inline void __free_one_page(struct page *page, } } +#ifdef CONFIG_ANDROID_SIMPLE_LMK +if (simple_lmk_page_in (page, order, migratetype)) +return ; +#endif +list_add (&page->lru, &zone->free_area[order].free_list[migratetype]); out: zone->free_area[order].nr_free++;
mm/page_alloc.c __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, · line 4442
Quoted 10 lines
mm/page_alloc.c +7
-0@@ -4427,6 +4433,13 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,if (costly_order && !(gfp_mask & __GFP_RETRY_MAYFAIL)) goto nopage; +#ifdef CONFIG_ANDROID_SIMPLE_LMK + page =simple_lmk_oom_alloc (order, ac->migratetype); +if (page) +prep_new_page (page, order, gfp_mask, alloc_flags); + goto got_pg; +#endif +
Hacking generic MM code with Android-specific callback is probably a major issue with your patch. Also I CC'd -mm maintainers and lists since your patch touches page_alloc.c. Always run get_maintainer.pl before sending a patch. I added them this time.
Have you looked at the recent PSI work that Suren and Johannes have been doing [1]? As I understand, userspace lmkd may be migrated to use that at some point. Suren can provide more details. I am sure AOSP contributions to make LMKd better by using the PSI backend would be appreciated. Please consider collaborating on that and help out, thanks. Check the cover-letter of that patch [1] where LMKd is mentioned.
thanks,
- Joel
[1] https://www.mail-archive.com/[email protected]/msg1951257.html
Quoted 44 lines · Joel Fernandes
On Sun, Mar 10, 2019 at 01:34:03PM -0700, Sultan Alsawaf wrote:
[...]Quoted 28 lines
kernel/fork.c +0
-0/* Perform scheduler related setup. Assign this task to a CPU. */ retval =sched_fork (clone_flags, p);mm/page_alloc.c +6
-0@@ -67,6 +67,7 @@#include <linux/lockdep.h> #include <linux/nmi.h> #include <linux/psi.h> +#include <linux/simple_lmk.h> #include <asm/sections.h> #include <asm/tlbflush.h> @@ -967,6 +968,11 @@ static inline void __free_one_page(struct page *page, } } +#ifdef CONFIG_ANDROID_SIMPLE_LMK +if (simple_lmk_page_in (page, order, migratetype)) +return ; +#endif +list_add (&page->lru, &zone->free_area[order].free_list[migratetype]); out: zone->free_area[order].nr_free++;mm/page_alloc.c __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, · line 4442
Quoted 10 lines
mm/page_alloc.c +7
-0@@ -4427,6 +4433,13 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,if (costly_order && !(gfp_mask & __GFP_RETRY_MAYFAIL)) goto nopage; +#ifdef CONFIG_ANDROID_SIMPLE_LMK + page =simple_lmk_oom_alloc (order, ac->migratetype); +if (page) +prep_new_page (page, order, gfp_mask, alloc_flags); + goto got_pg; +#endif +Hacking generic MM code with Android-specific callback is probably a major issue with your patch.
Quoted 3 lines
Also I CC'd -mm maintainers and lists since your patch
touchespage_alloc.c. Always run get_maintainer.pl before sending a patch. I added them this time.
I see you CC'd linux-mm on your initial patch, so I apologize. Ignore this part of my reply. Thanks.
Quoted 6 lines
Have you looked at the recent PSI work that Suren and Johannes have been doing [1]? As I understand, userspace lmkd may be migrated to use that at some point. Suren can provide more details. I am sure AOSP contributions to make LMKd better by using the PSI backend would be appreciated. Please consider collaborating on that and help out, thanks. Check the cover-letter of that patch [1] where LMKd is mentioned.
Michal Hocko
reviewerQuoted 18 lines · Sultan Alsawaf
From: Sultan Alsawaf <[email protected]>
This is a complete low memory killer solution for Android that is small and simple. It kills the largest, least-important processes it can find whenever a page allocation has completely failed (right after direct reclaim). Processes are killed according to the priorities that Android gives them, so that the least important processes are always killed first. Killing larger processes is preferred in order to free the most memory possible in one go.
Simple LMK is integrated deeply into the page allocator in order to catch exactly when a page allocation fails and exactly when a page is freed. Failed page allocations that have invoked Simple LMK are placed on a queue and wait for Simple LMK to satisfy them. When a page is about to be freed, the failed page allocations are given priority over normal page allocations by Simple LMK to see if they can immediately use the freed page.
Quoted 2 lines
Additionally, processes are continuously killed by failed small-order page allocations until they are satisfied.
I am sorry but we are not going to maintain two different OOM implementations in the kernel. From a quick look the implementation is quite a hack which is not really suitable for anything but a very specific usecase. E.g. reusing a freed page for a waiting allocation sounds like an interesting idea but it doesn't really work for many reasons. E.g. any NUMA affinity is broken, zone protection doesn't work either. Not to mention how the code hooks into the allocator hot paths. This is simply no no.
Last but not least people have worked really hard to provide means (PSI) to do what you need in the userspace.
Quoted 453 lines
Signed-off-by Sultan Alsawaf <[email protected]>--- drivers/android/Kconfig | 28 ++++ drivers/android/Makefile | 1 + drivers/android/simple_lmk.c | 301 +++++++++++++++++++++++++++++++++++ include/linux/sched.h | 3 + include/linux/simple_lmk.h | 11 ++ kernel/fork.c | 3 + mm/page_alloc.c | 13 ++ 7 files changed, 360 insertions(+) create mode 100644 drivers/android/simple_lmk.c create mode 100644 include/linux/simple_lmk.hdrivers/android/Kconfig +28
-0@@ -54,6 +54,34 @@ config ANDROID_BINDER_IPC_SELFTEST exhaustively with combinations of various buffer sizes and alignments. +config ANDROID_SIMPLE_LMK + bool "Simple Android Low Memory Killer" + depends on !MEMCG + ---help--- + This is a complete low memory killer solution for Android that is + small and simple. It is integrated deeply into the page allocator to + know exactly when a page allocation hits OOM and exactly when a page + is freed. Processes are killed according to the priorities that + Android gives them, so that the least important processes are always + killed first. + +if ANDROID_SIMPLE_LMK + +config ANDROID_SIMPLE_LMK_MINFREE + int "Minimum MiB of memory to free per reclaim" + default "64" + help + Simple LMK will free at least this many MiB of memory per reclaim. + +config ANDROID_SIMPLE_LMK_KILL_TIMEOUT + int "Kill timeout in milliseconds" + default "50" + help + Simple LMK will only perform memory reclaim at most once per this + amount of time. + +endif # if ANDROID_SIMPLE_LMK + endif # if ANDROID endmenudrivers/android/Makefile +1
-0@@ -3,3 +3,4 @@ ccflags-y += -I$(src) # needed for trace events obj-$(CONFIG_ANDROID_BINDERFS) += binderfs.o obj-$(CONFIG_ANDROID_BINDER_IPC) += binder.o binder_alloc.o obj-$(CONFIG_ANDROID_BINDER_IPC_SELFTEST) += binder_alloc_selftest.o +obj-$(CONFIG_ANDROID_SIMPLE_LMK) += simple_lmk.odrivers/android/simple_lmk.c new file +301
-0@@ -0,0 +1,301 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Copyright (C) 2019 Sultan Alsawaf <[email protected]>. +*/ + +#define pr_fmt (fmt) "simple_lmk: " fmt + +#include <linux/mm.h> +#include <linux/moduleparam.h> +#include <linux/oom.h> +#include <linux/sched.h> +#include <linux/sizes.h> +#include <linux/sort.h> + +#define MIN_FREE_PAGES (CONFIG_ANDROID_SIMPLE_LMK_MINFREE * SZ_1M / PAGE_SIZE) + +struct oom_alloc_req { +struct page *page; +struct completion done; +struct list_head lh; +unsigned int order; +int migratetype; +}; + +struct victim_info { +struct task_struct *tsk; +unsigned long size; +}; + +enum { +DISABLED , +STARTING , +READY , +KILLING +}; + +/* Pulled from the Android framework */ +static const short int adj_prio[] = { +906 ,/* CACHED_APP_MAX_ADJ */ +905 ,/* Cached app */ +904 ,/* Cached app */ +903 ,/* Cached app */ +902 ,/* Cached app */ +901 ,/* Cached app */ +900 ,/* CACHED_APP_MIN_ADJ */ +800 ,/* SERVICE_B_ADJ */ +700 ,/* PREVIOUS_APP_ADJ */ +600 ,/* HOME_APP_ADJ */ +500 ,/* SERVICE_ADJ */ +400 ,/* HEAVY_WEIGHT_APP_ADJ */ +300 ,/* BACKUP_APP_ADJ */ +200 ,/* PERCEPTIBLE_APP_ADJ */ +100 ,/* VISIBLE_APP_ADJ */ +0 /* FOREGROUND_APP_ADJ */ +}; + +/* Make sure that PID_MAX_DEFAULT isn't too big, or these arrays will be huge */ +static struct victim_info victim_array[PID_MAX_DEFAULT ]; +static struct victim_info *victim_ptr_array[ARRAY_SIZE (victim_array)]; +static atomic_t simple_lmk_state =ATOMIC_INIT (DISABLED ); +static atomic_t oom_alloc_count =ATOMIC_INIT (0 ); +static unsigned long last_kill_expires; +static unsigned long kill_expires; +static DEFINE_SPINLOCK (oom_queue_lock); +static LIST_HEAD (oom_alloc_queue); + +static int victim_info_cmp (const void *lhs,const void *rhs) +{ +const struct victim_info **lhs_ptr = (typeof (lhs_ptr ))lhs; +const struct victim_info **rhs_ptr = (typeof (rhs_ptr ))rhs; + +if ((*lhs_ptr)->size > (*rhs_ptr)->size) +return -1 ; + +if ((*lhs_ptr)->size < (*rhs_ptr)->size) +return 1 ; + +return 0 ; +} + +static unsigned long scan_and_kill (int min_adj,int max_adj, +unsigned long pages_needed) +{ +unsigned long pages_freed =0 ; +unsigned int i, vcount =0 ; +struct task_struct *tsk; + +rcu_read_lock (); +for_each_process (tsk) { +struct task_struct *vtsk; +unsigned long tasksize; +short oom_score_adj; + +/* Don't commit suicide or kill kthreads */ +if (same_thread_group (tsk, current) || tsk->flags &PF_KTHREAD ) +continue ; + + vtsk =find_lock_task_mm (tsk); +if (!vtsk) +continue ; + +/* Don't kill tasks that have been killed or lack memory */ +if (vtsk->slmk_sigkill_sent || +test_tsk_thread_flag (vtsk,TIF_MEMDIE )) { +task_unlock (vtsk); +continue ; + } + + oom_score_adj = vtsk->signal->oom_score_adj; +if (oom_score_adj < min_adj || oom_score_adj > max_adj) { +task_unlock (vtsk); +continue ; + } + + tasksize =get_mm_rss (vtsk->mm); +task_unlock (vtsk); +if (!tasksize) +continue ; + +/* Store this potential victim away for later */ +get_task_struct (vtsk); + victim_array[vcount].tsk = vtsk; + victim_array[vcount].size = tasksize; + victim_ptr_array[vcount] = &victim_array[vcount]; + vcount++; + +/* The victim array is so big that this should never happen */ +if (unlikely (vcount ==ARRAY_SIZE (victim_array))) +break ; + } +rcu_read_unlock (); + +/* No potential victims for this adj range means no pages freed */ +if (!vcount) +return 0 ; + +/* +* Sort the victims in descending order of size in order to target the +* largest ones first. +*/ +sort (victim_ptr_array, vcount,sizeof (victim_ptr_array[0 ]), + victim_info_cmp,NULL ); + +for (i =0 ; i < vcount; i++) { +struct victim_info *victim = victim_ptr_array[i]; +struct task_struct *vtsk = victim->tsk; + +if (pages_freed >= pages_needed) { +put_task_struct (vtsk); +continue ; + } + +pr_info ("killing %s with adj %d to free %lu MiB\n" , + vtsk->comm, vtsk->signal->oom_score_adj, + victim->size *PAGE_SIZE /SZ_1M ); + +if (!do_send_sig_info (SIGKILL ,SEND_SIG_PRIV , vtsk, true)) + pages_freed += victim->size; + +/* Unconditionally mark task as killed so it isn't reused */ + vtsk->slmk_sigkill_sent = true; +put_task_struct (vtsk); + } + +return pages_freed; +} + +static void kill_processes (unsigned long pages_needed) +{ +unsigned long pages_freed =0 ; +int i; + +for (i =1 ; i <ARRAY_SIZE (adj_prio); i++) { + pages_freed +=scan_and_kill (adj_prio[i], adj_prio[i -1 ], + pages_needed - pages_freed); +if (pages_freed >= pages_needed) +break ; + } +} + +static void do_memory_reclaim (void ) +{ +/* Only one reclaim can occur at a time */ +if (atomic_cmpxchg (&simple_lmk_state,READY ,KILLING ) !=READY ) +return ; + +if (time_after_eq (jiffies, last_kill_expires)) { +kill_processes (MIN_FREE_PAGES ); + last_kill_expires = jiffies + kill_expires; + } + +atomic_set (&simple_lmk_state,READY ); +} + +static long reclaim_once_or_more (struct completion *done,unsigned int order) +{ +long ret; + +/* Don't allow costly allocations to do memory reclaim more than once */ +if (order >PAGE_ALLOC_COSTLY_ORDER ) { +do_memory_reclaim (); +return wait_for_completion_killable (done); + } + +do { +do_memory_reclaim (); + ret =wait_for_completion_killable_timeout (done, kill_expires); + }while (!ret); + +return ret; +} + +struct page *simple_lmk_oom_alloc (unsigned int order,int migratetype) +{ +struct oom_alloc_req page_req = { + .done =COMPLETION_INITIALIZER_ONSTACK (page_req.done), + .order = order, + .migratetype = migratetype + }; +long ret; + +if (atomic_read (&simple_lmk_state) <=STARTING ) +return NULL ; + +spin_lock (&oom_queue_lock); +list_add_tail (&page_req.lh, &oom_alloc_queue); +spin_unlock (&oom_queue_lock); + +atomic_inc (&oom_alloc_count); + +/* Do memory reclaim and wait */ + ret =reclaim_once_or_more (&page_req.done, order); +if (ret == -ERESTARTSYS ) { +/* Give up since this process is dying */ +spin_lock (&oom_queue_lock); +if (!page_req.page) +list_del (&page_req.lh); +spin_unlock (&oom_queue_lock); + } + +atomic_dec (&oom_alloc_count); + +return page_req.page; +} + +bool simple_lmk_page_in (struct page *page,unsigned int order,int migratetype) +{ +struct oom_alloc_req *page_req; +bool matched = false; +int try_order; + +if (atomic_read (&simple_lmk_state) <=STARTING || + !atomic_read (&oom_alloc_count)) +return false; + +/* Try to match this free page with an OOM allocation request */ +spin_lock (&oom_queue_lock); +for (try_order = order; try_order >=0 ; try_order--) { +list_for_each_entry (page_req, &oom_alloc_queue, lh) { +if (page_req->order == try_order && + page_req->migratetype == migratetype) { + matched = true; +break ; + } + } + +if (matched) +break ; + } + +if (matched) { +__ClearPageBuddy (page); + page_req->page = page; +list_del (&page_req->lh); +complete (&page_req->done); + } +spin_unlock (&oom_queue_lock); + +return matched; +} + +/* Enable Simple LMK when LMKD in Android writes to the minfree parameter */ +static int simple_lmk_init_set (const char *val,const struct kernel_param *kp) +{ +if (atomic_cmpxchg (&simple_lmk_state,DISABLED ,STARTING ) !=DISABLED ) +return 0 ; + +/* Store the calculated kill timeout jiffies for frequent reuse */ + kill_expires =msecs_to_jiffies (CONFIG_ANDROID_SIMPLE_LMK_KILL_TIMEOUT ); +atomic_set (&simple_lmk_state,READY ); +return 0 ; +} + +static const struct kernel_param_ops simple_lmk_init_ops = { + .set = simple_lmk_init_set +}; + +/* Needed to prevent Android from thinking there's no LMK and thus rebooting */ +#undef MODULE_PARAM_PREFIX +#define MODULE_PARAM_PREFIX "lowmemorykiller." +module_param_cb (minfree, &simple_lmk_init_ops,NULL ,0200 );include/linux/sched.h +3
-0@@ -1199,6 +1199,9 @@ struct task_struct {unsigned long lowest_stack;unsigned long prev_lowest_stack;#endif +#ifdef CONFIG_ANDROID_SIMPLE_LMK +bool slmk_sigkill_sent; +#endif /* * New fields for task_struct should be added above here, so that include/linux/simple_lmk.h new file +11
-0@@ -0,0 +1,11 @@ +/* SPDX-License-Identifier: GPL-2.0 +* +* Copyright (C) 2019 Sultan Alsawaf <[email protected]>. +*/ +#ifndef _SIMPLE_LMK_H_ +#define _SIMPLE_LMK_H_ + +struct page *simple_lmk_oom_alloc (unsigned int order,int migratetype); +bool simple_lmk_page_in (struct page *page,unsigned int order,int migratetype); + +#endif /* _SIMPLE_LMK_H_ */ kernel/fork.c +3
-0@@ -1881,6 +1881,9 @@ static __latent_entropy struct task_struct *copy_process( p->sequential_io =0 ; p->sequential_io_avg =0 ;#endif +#ifdef CONFIG_ANDROID_SIMPLE_LMK + p->slmk_sigkill_sent = false; +#endif /* Perform scheduler related setup. Assign this task to a CPU. */ retval =sched_fork (clone_flags, p);mm/page_alloc.c +13
-0@@ -67,6 +67,7 @@#include <linux/lockdep.h> #include <linux/nmi.h> #include <linux/psi.h> +#include <linux/simple_lmk.h> #include <asm/sections.h> #include <asm/tlbflush.h> @@ -967,6 +968,11 @@ static inline void __free_one_page(struct page *page, } } +#ifdef CONFIG_ANDROID_SIMPLE_LMK +if (simple_lmk_page_in (page, order, migratetype)) +return ; +#endif +list_add (&page->lru, &zone->free_area[order].free_list[migratetype]); out: zone->free_area[order].nr_free++;@@ -4427,6 +4433,13 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,if (costly_order && !(gfp_mask & __GFP_RETRY_MAYFAIL)) goto nopage; +#ifdef CONFIG_ANDROID_SIMPLE_LMK + page =simple_lmk_oom_alloc (order, ac->migratetype); +if (page) +prep_new_page (page, order, gfp_mask, alloc_flags); + goto got_pg; +#endif +if (should_reclaim_retry (gfp_mask, order, ac, alloc_flags, did_some_progress >0 , &no_progress_loops)) goto retry;Signature · 1 line
2.21.0
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 11 lines · Michal Hocko
I am sorry but we are not going to maintain two different OOM implementations in the kernel. From a quick look the implementation is quite a hack which is not really suitable for anything but a very specific usecase. E.g. reusing a freed page for a waiting allocation sounds like an interesting idea but it doesn't really work for many reasons. E.g. any NUMA affinity is broken, zone protection doesn't work either. Not to mention how the code hooks into the allocator hot paths. This is simply no no.
Last but not least people have worked really hard to provide means (PSI) to do what you need in the userspace.
Hi Michal,
Thanks for the feedback. I had no doubt that this would be vehemently rejected
on the mailing list, but I wanted feedback/opinions on it and thus sent it as anRFC. At best I thought perhaps the mechanisms I've employed might serve as inspiration for LMKD improvements in Android, since this hacky OOM killer I've
devised does work quite well for the very specific usecase it is set out to
address. The NUMA affinity and zone protection bits are helpful insights too.
I'll take a look at PSI which Joel mentioned as well.
Thanks,
Sultan Alsawaf
Suren Baghdasaryan
reviewerHi Sultan,
Quoted 11 lines · Sultan Alsawaf
Quoted 9 lines · Michal Hocko
I am sorry but we are not going to maintain two different OOM implementations in the kernel. From a quick look the implementation is quite a hack which is not really suitable for anything but a very specific usecase. E.g. reusing a freed page for a waiting allocation sounds like an interesting idea but it doesn't really work for many reasons. E.g. any NUMA affinity is broken, zone protection doesn't work either. Not to mention how the code hooks into the allocator hot paths. This is simply no no.
Quoted 7 lines
Quoted 2 lines
Last but not least people have worked really hard to provide means (PSI) to do what you need in the userspace.
Hi Michal,
Thanks for the feedback. I had no doubt that this would be vehemently rejected on the mailing list, but I wanted feedback/opinions on it and thus sent it as anRFC.
Thanks for the proposal. I think Michal and Joel already answered why in-kernel LMK will not be accepted and that was one of the reasons the lowmemorykiller driver was removed in 4.12.
Quoted 4 lines
At best I thought perhaps the mechanisms I've employed might serve as inspiration for LMKD improvements in Android, since this hacky OOM killer I've devised does work quite well for the very specific usecase it is set out to address. The NUMA affinity and zone protection bits are helpful insights too.
The idea seems interesting although I need to think about this a bit more. Killing processes based on failed page allocation might backfire during transient spikes in memory usage.
AFAIKT the biggest issue with using this approach in userspace is that it's not practically implementable without heavy in-kernel support. How to implement such interaction between kernel and userspace would be an interesting discussion which I would be happy to participate in.
Quoted 4 lines
I'll take a look at PSI which Joel mentioned as well.
Thanks,
Sultan Alsawaf
Thanks,
Suren.
Quoted 3 lines · Suren Baghdasaryan
The idea seems interesting although I need to think about this a bit more. Killing processes based on failed page allocation might backfire during transient spikes in memory usage.
This issue could be alleviated if tasks could be killed and have their pages reaped faster. Currently, Linux takes a _very_ long time to free a task's memory after an initial privileged SIGKILL is sent to a task, even with the task's priority being set to the highest possible (so unwanted scheduler preemption starving dying tasks of CPU time is not the issue at play here). I've frequently measured the difference in time between when a SIGKILL is sent for a task and when free_task() is called for that task to be hundreds of milliseconds, which is incredibly long. AFAIK, this is a problem that LMKD suffers from as well, and perhaps any OOM killer implementation in Linux, since you cannot evaluate effect you've had on memory pressure by killing a process for at least several tens of milliseconds.
Quoted 4 lines
AFAIKT the biggest issue with using this approach in userspace is that it's not practically implementable without heavy in-kernel support. How to implement such interaction between kernel and userspace would be an interesting discussion which I would be happy to participate in.
You could signal a lightweight userspace process that has maximum scheduler priority and have it kill the tasks it'd like.
Thanks,
Sultan
Quoted 7 lines · Sultan Alsawaf
Quoted 3 lines · Suren Baghdasaryan
The idea seems interesting although I need to think about this a bit more. Killing processes based on failed page allocation might backfire during transient spikes in memory usage.
This issue could be alleviated if tasks could be killed and have their pages reaped faster.
But the point is that a transient temporary memory spike should not be a signal to kill _any_ process. The reaction to kill shouldn't be so spontaneous that unwanted tasks are killed because the system went into panic mode. It should be averaged out which I believe is what PSI does.
thanks,
- Joel
Quoted 4 lines · Joel Fernandes
But the point is that a transient temporary memory spike should not be a signal to kill _any_ process. The reaction to kill shouldn't be so spontaneous that unwanted tasks are killed because the system went into panic mode. It should be averaged out which I believe is what PSI does.
In my patch from the first email, I implemented the decision to kill a process at the same time that the existing kernel OOM killer decides to kill a process. If the kernel's OOM killer were susceptible to killing processes due to transient memory spikes, then I think there would have been several complaints about this behavior regardless of which userspace or architecture is in use. I think the existing OOM killer has this done right.
The decision to kill a process occurs after the page allocator has tried _very_ hard to satisfy a page allocation via alternative means, such as utilizing compaction, flushing file-backed pages to disk via kswapd, and direct reclaim. Once all of those means have failed, it is quite reasonable to kill a process to free memory. Trying to wait out the memory starvation at this point would be futile.
Thanks,
Sultan
Suren Baghdasaryan
reviewerQuoted 6 lines · Sultan Alsawaf
Quoted 3 lines · Suren Baghdasaryan
The idea seems interesting although I need to think about this a bit more. Killing processes based on failed page allocation might backfire during transient spikes in memory usage.
Quoted 11 lines
This issue could be alleviated if tasks could be killed and have their pages reaped faster. Currently, Linux takes a _very_ long time to free a task's memory after an initial privileged SIGKILL is sent to a task, even with the task's priority being set to the highest possible (so unwanted scheduler preemption starving dying tasks of CPU time is not the issue at play here). I've frequently measured the difference in time between when a SIGKILL is sent for a task and when
free_task()is called for that task to be hundreds of milliseconds, which is incredibly long. AFAIK, this is a problem that LMKD suffers from as well, and perhaps any OOM killer implementation in Linux, since you cannot evaluate effect you've had on memory pressure by killing a process for at least several tens of milliseconds.
Yeah, killing speed is a well-known problem which we are considering in LMKD. For example the recent LMKD change to assign process being killed to a cpuset cgroup containing big cores cuts the kill time considerably. This is not ideal and we are thinking about better ways to expedite the cleanup process.
Quoted 7 lines
Quoted 4 lines
AFAIKT the biggest issue with using this approach in userspace is that it's not practically implementable without heavy in-kernel support. How to implement such interaction between kernel and userspace would be an interesting discussion which I would be happy to participate in.
You could signal a lightweight userspace process that has maximum scheduler priority and have it kill the tasks it'd like.
This what LMKD currently is - a userspace RT process. My point was that this page allocation queue that you implemented can't be implemented in userspace, at least not without extensive communication with kernel.
Quoted 2 lines
Thanks,
Sultan
Thanks,
Suren.
Quoted 4 lines · Suren Baghdasaryan
This what LMKD currently is - a userspace RT process. My point was that this page allocation queue that you implemented can't be implemented in userspace, at least not without extensive communication with kernel.
Oh, that's easy to address. My page allocation queue and the decision on when to kill a process are orthogonal. In fact, the page allocation queue could be touched up a bit to factor in the issues Michal mentioned, and it can be implemented as an improvement to the existing OOM killer. The point of it is just to ensure that page allocation requests that have gone OOM are given priority over other allocation requests when free pages start to trickle in.
Userspace doesn't need to know about the page allocation queue, and the queue is not necessary to implement the method of determining when to kill processes that I've proposed. It's an optimization, not a necessity.
Thanks,
Sultan
Michal Hocko
reviewerQuoted 19 lines · Suren Baghdasaryan
Quoted 6 lines · Sultan Alsawaf
Quoted 3 lines · Suren Baghdasaryan
The idea seems interesting although I need to think about this a bit more. Killing processes based on failed page allocation might backfire during transient spikes in memory usage.
Quoted 11 lines
This issue could be alleviated if tasks could be killed and have their pages reaped faster. Currently, Linux takes a _very_ long time to free a task's memory after an initial privileged SIGKILL is sent to a task, even with the task's priority being set to the highest possible (so unwanted scheduler preemption starving dying tasks of CPU time is not the issue at play here). I've frequently measured the difference in time between when a SIGKILL is sent for a task and when
free_task()is called for that task to be hundreds of milliseconds, which is incredibly long. AFAIK, this is a problem that LMKD suffers from as well, and perhaps any OOM killer implementation in Linux, since you cannot evaluate effect you've had on memory pressure by killing a process for at least several tens of milliseconds.
Quoted 5 lines
Yeah, killing speed is a well-known problem which we are considering in LMKD. For example the recent LMKD change to assign process being killed to a cpuset cgroup containing big cores cuts the kill time considerably. This is not ideal and we are thinking about better ways to expedite the cleanup process.
If you design is relies on the speed of killing then it is fundamentally flawed AFAICT. You cannot assume anything about how quickly a task dies. It might be blocked in an uninterruptible sleep or performin an operation which takes some time. Sure, oom_reaper might help here but still.
The only way to control the OOM behavior pro-actively is to throttle allocation speed. We have memcg high limit for that purpose. Along with PSI, I can imagine a reasonably working user space early oom notifications and reasonable acting upon that.
Signature · 2 lines
Michal Hocko SUSE Labs
Suren Baghdasaryan
reviewerQuoted 27 lines · Michal Hocko
Quoted 19 lines · Suren Baghdasaryan
Quoted 6 lines · Sultan Alsawaf
Quoted 3 lines · Suren Baghdasaryan
The idea seems interesting although I need to think about this a bit more. Killing processes based on failed page allocation might backfire during transient spikes in memory usage.
Quoted 11 lines
This issue could be alleviated if tasks could be killed and have their pages reaped faster. Currently, Linux takes a _very_ long time to free a task's memory after an initial privileged SIGKILL is sent to a task, even with the task's priority being set to the highest possible (so unwanted scheduler preemption starving dying tasks of CPU time is not the issue at play here). I've frequently measured the difference in time between when a SIGKILL is sent for a task and when
free_task()is called for that task to be hundreds of milliseconds, which is incredibly long. AFAIK, this is a problem that LMKD suffers from as well, and perhaps any OOM killer implementation in Linux, since you cannot evaluate effect you've had on memory pressure by killing a process for at least several tens of milliseconds.Quoted 5 lines
Yeah, killing speed is a well-known problem which we are considering in LMKD. For example the recent LMKD change to assign process being killed to a cpuset cgroup containing big cores cuts the kill time considerably. This is not ideal and we are thinking about better ways to expedite the cleanup process.
Quoted 5 lines
If you design is relies on the speed of killing then it is fundamentally flawed AFAICT. You cannot assume anything about how quickly a task dies. It might be blocked in an uninterruptible sleep or performin an operation which takes some time. Sure,
oom_reapermight help here but still.
That's what I was considering. This is not a silver bullet but increased speed would not hurt.
Quoted 4 lines
The only way to control the OOM behavior pro-actively is to throttle allocation speed. We have memcg high limit for that purpose. Along with PSI, I can imagine a reasonably working user space early oom notifications and reasonable acting upon that.
That makes sense and we are working in this direction.
Quoted 3 lines
--
Michal Hocko
SUSE Labs
Thanks,
Suren.
Quoted 7 lines · Michal Hocko
Quoted 5 lines · Suren Baghdasaryan
Yeah, killing speed is a well-known problem which we are considering in LMKD. For example the recent LMKD change to assign process being killed to a cpuset cgroup containing big cores cuts the kill time considerably. This is not ideal and we are thinking about better ways to expedite the cleanup process.
Quoted 5 lines
If you design is relies on the speed of killing then it is fundamentally flawed AFAICT. You cannot assume anything about how quickly a task dies. It might be blocked in an uninterruptible sleep or performin an operation which takes some time. Sure,
oom_reapermight help here but still.
Many UNINTERRUPTIBLE sleeps can be converted to KILLABLE sleeps. It just needs someone to do the work.
Michal Hocko
reviewerQuoted 8 lines · Matthew Wilcox
Quoted 7 lines · Michal Hocko
Quoted 5 lines · Suren Baghdasaryan
Yeah, killing speed is a well-known problem which we are considering in LMKD. For example the recent LMKD change to assign process being killed to a cpuset cgroup containing big cores cuts the kill time considerably. This is not ideal and we are thinking about better ways to expedite the cleanup process.
Quoted 8 lines
Quoted 5 lines
If you design is relies on the speed of killing then it is fundamentally flawed AFAICT. You cannot assume anything about how quickly a task dies. It might be blocked in an uninterruptible sleep or performin an operation which takes some time. Sure,
oom_reapermight help here but still.Many UNINTERRUPTIBLE sleeps can be converted to KILLABLE sleeps. It just needs someone to do the work.
They can and should as much as possible. No question about that. But not all of them can and that is why nobody should be relying on that. That is the whole point of having the oom_reaper and async oom victim tear down.
Signature · 2 lines
Michal Hocko SUSE Labs
Michal Hocko
reviewerQuoted 15 lines · Michal Hocko
Quoted 14 lines · Matthew Wilcox
Quoted 12 lines · Michal Hocko
Quoted 5 lines · Suren Baghdasaryan
Yeah, killing speed is a well-known problem which we are considering in LMKD. For example the recent LMKD change to assign process being killed to a cpuset cgroup containing big cores cuts the kill time considerably. This is not ideal and we are thinking about better ways to expedite the cleanup process.
If you design is relies on the speed of killing then it is fundamentally flawed AFAICT. You cannot assume anything about how quickly a task dies. It might be blocked in an uninterruptible sleep or performin an operation which takes some time. Sure,
oom_reapermight help here but still.
Quoted 7 lines
Quoted 2 lines
Many UNINTERRUPTIBLE sleeps can be converted to KILLABLE sleeps. It just needs someone to do the work.
They can and should as much as possible. No question about that. But not all of them can and that is why nobody should be relying on that. That is the whole point of having the
oom_reaperand async oom victim tear down.
Let me clarify a bit. LMK obviously doesn't need any guarantee like the core oom killer because it is more of a pro-active measure than the last resort. I merely wanted to say that relying on a design which assumes anything about time victim needs to exit is flawed and it will fail under different workloads. On the other hand this might work good enough on very specific workloads to be usable. I am not questioning that. The point is that this is not generic enough to be accepted to the upstream kernel.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 4 lines · Michal Hocko
The only way to control the OOM behavior pro-actively is to throttle allocation speed. We have memcg high limit for that purpose. Along with PSI, I can imagine a reasonably working user space early oom notifications and reasonable acting upon that.
The issue with pro-active memory management that prompted me to create this was poor memory utilization. All of the alternative means of reclaiming pages in the page allocator's slow path turn out to be very useful for maximizing memory utilization, which is something that we would have to forgo by relying on a purely pro-active solution. I have not had a chance to look at PSI yet, but unless a PSI-enabled solution allows allocations to reach the same point as when the OOM killer is invoked (which is contradictory to what it sets out to do), then it cannot take advantage of all of the alternative memory-reclaim means employed in the slowpath, and will result in killing a process before it is _really_ necessary.
Quoted 5 lines
If you design is relies on the speed of killing then it is fundamentally flawed AFAICT. You cannot assume anything about how quickly a task dies. It might be blocked in an uninterruptible sleep or performin an operation which takes some time. Sure,
oom_reapermight help here but still.
In theory we could instantly zap any process that is not trapped in the kernel at the time that the OOM killer is invoked without any consequences though, no?
Thanks,
Sultan
Michal Hocko
reviewerQuoted 6 lines · Sultan Alsawaf
Quoted 4 lines · Michal Hocko
The only way to control the OOM behavior pro-actively is to throttle allocation speed. We have memcg high limit for that purpose. Along with PSI, I can imagine a reasonably working user space early oom notifications and reasonable acting upon that.
Quoted 10 lines
The issue with pro-active memory management that prompted me to create this was poor memory utilization. All of the alternative means of reclaiming pages in the page allocator's slow path turn out to be very useful for maximizing memory utilization, which is something that we would have to forgo by relying on a purely pro-active solution. I have not had a chance to look at PSI yet, but unless a PSI-enabled solution allows allocations to reach the same point as when the OOM killer is invoked (which is contradictory to what it sets out to do), then it cannot take advantage of all of the alternative memory-reclaim means employed in the slowpath, and will result in killing a process before it is _really_ necessary.
If you really want to reach the real OOM situation then you can very well rely on the in-kernel OOM killer. The only reason you want a customized oom killer is the tasks clasification. And that is a different story. User space hints on the victim selection has been a topic for quite while. It never get to any conclusion as interested parties have always lost an interest because it got hairy quickly.
Quoted 8 lines
Quoted 5 lines
If you design is relies on the speed of killing then it is fundamentally flawed AFAICT. You cannot assume anything about how quickly a task dies. It might be blocked in an uninterruptible sleep or performin an operation which takes some time. Sure,
oom_reapermight help here but still.In theory we could instantly zap any process that is not trapped in the kernel at the time that the OOM killer is invoked without any consequences though, no?
No, this is not so simple. Have a look at the oom_reaper and hops it has to go through.
Signature · 2 lines
Michal Hocko SUSE Labs
Michal Hocko
reviewerQuoted 6 lines · Sultan Alsawaf
I have not had a chance to look at PSI yet, but
unless a PSI-enabled solution allows allocations to reach the same point as when the OOM killer is invoked (which is contradictory to what it sets out to do), then it cannot take advantage of all of the alternative memory-reclaim means employed in the slowpath, and will result in killing a process before it is _really_ necessary.
One more note. The above is true, but you can also hit one of the thrashing reclaim behaviors and reclaim last few pages again and again with the whole system really sluggish. That is what PSI is trying to help with.
Signature · 2 lines
Michal Hocko SUSE Labs
Suren Baghdasaryan
reviewerQuoted 5 lines · Michal Hocko
Quoted 3 lines · Sultan Alsawaf
I have not had a chance to look at PSI yet, but
unless a PSI-enabled solution allows allocations to reach the same point as when the OOM killer is invoked (which is contradictory to what it sets out to do),
LMK's job is to relieve memory pressure before we reach the boiling point at which OOM killer has to be invoked. If we wait that long it will definitely affect user experience. There might be usecases when you might not care about this but on interactive systems like Android it is important.
Quoted 3 lines
Quoted 3 lines
then it cannot take advantage of all of the alternative memory-reclaim means employed in the slowpath, and will result in killing a process before it is _really_ necessary.
I guess it's a matter of defining when is it _really_ necessary to kill. In Android case that should be when the user starts suffering from the delays caused by memory contention and that delay is exactly what PSI is measuring.
Quoted 7 lines
One more note. The above is true, but you can also hit one of the thrashing reclaim behaviors and reclaim last few pages again and again with the whole system really sluggish. That is what PSI is trying to help with.
--
Michal Hocko
SUSE Labs
Quoted 7 lines · Sultan Alsawaf
Quoted 4 lines · Michal Hocko
The only way to control the OOM behavior pro-actively is to throttle allocation speed. We have memcg high limit for that purpose. Along with PSI, I can imagine a reasonably working user space early oom notifications and reasonable acting upon that.
Quoted 10 lines
The issue with pro-active memory management that prompted me to create this was poor memory utilization. All of the alternative means of reclaiming pages in the page allocator's slow path turn out to be very useful for maximizing memory utilization, which is something that we would have to forgo by relying on a purely pro-active solution. I have not had a chance to look at PSI yet, but unless a PSI-enabled solution allows allocations to reach the same point as when the OOM killer is invoked (which is contradictory to what it sets out to do), then it cannot take advantage of all of the alternative memory-reclaim means employed in the slowpath, and will result in killing a process before it is _really_ necessary.
There are two essential parts of a lowmemorykiller implementation:
when to kill and how to kill.
There are a million possible approaches to decide when to kill an unimportant process. They usually trade off between the same two failure modes depending on the workload.
If you kill too aggressively, a transient spike that could be imperceptibly absorbed by evicting some file pages or moving some pages to ZRAM will result in killing processes, which then get started up later and have a performance/battery cost.
If you don't kill aggressively enough, you will encounter a workload that thrashes the page cache, constantly evicting and reloading file pages and moving things in and out of ZRAM, which makes the system unusable when a process should have been killed instead.
As far as I've seen, any methodology that uses single points in time to decide when to kill without completely biasing toward one or the other is susceptible to both. The minfree approach used by lowmemorykiller/lmkd certainly is; it is both too aggressive for some workloads and not aggressive enough for other workloads. My guess is that simple LMK won't kill on transient spikes but will be extremely susceptible to page cache thrashing. This is not an improvement; page cache thrashing manifests as the entire system running very slowly.
What you actually want from lowmemorykiller/lmkd on Android is to only kill once it becomes clear that the system will continue to try to reclaim memory to the extent that it could impact what the user actually cares about. That means tracking how much time is spent in reclaim/paging operations and the like, and that's exactly what PSI does. lmkd has had support for using PSI as a replacement for vmpressure for use as a wakeup trigger (to check current memory levels against the minfree thresholds) since early February. It works fine; unsurprisingly it's better than vmpressure at avoiding false wakeups.
Longer term, there's a lot of work to be done in lmkd to turn PSI into a kill trigger and remove minfree entirely. It's tricky (mainly because of the "when to kill another process" problem discussed later), but I believe it's feasible.
How to kill is similarly messy. The latency of reclaiming memory post SIGKILL can be severe (usually tens of milliseconds, occasionally
Quoted 1 line
100ms). The latency we see on Android usually isn't because those
threads are blocked in uninterruptible sleep, it's because times of memory pressure are also usually times of significant CPU contention and these are overwhelmingly CFS threads, some of which may be assigned a very low priority. lmkd now sets priorities and resets cpusets upon killing a process, and we have seen improved reclaim latency because of this. oom reaper might be a good approach to avoid this latency (I think some in-kernel lowmemorykiller implementations rely on it), but we can't use it from userspace. Something for future consideration.
A non-obvious consequence of both of these concerns is that when to kill a second process is a distinct and more difficult problem than when to kill the first. A second process should be killed if reclaim from the first process has finished and there has been insufficient memory reclaimed to avoid perceptible impact. Identifying whether memory pressure continues at the same level can probably be handled through multiple PSI monitors with different thresholds and window lengths, but this is an area of future work.
Knowing whether a SIGKILL'd process has finished reclaiming is as far as I know not possible without something like procfds. That's where the 100ms timeout in lmkd comes in. lowmemorykiller and lmkd both attempt to wait up to 100ms for reclaim to finish by checking for the continued existence of the thread that received the SIGKILL, but this really means that they wait up to 100ms for the _thread_ to finish, which doesn't tell you anything about the memory used by that process. If those threads terminate early and lowmemorykiller/lmkd get a signal to kill again, then there may be two processes competing for CPU time to reclaim memory. That doesn't reclaim any faster and may be an unnecessary kill.
So, in summary, the impactful LMK improvements seem like
- get lmkd and PSI to the point that lmkd can use PSI signals as a kill trigger and remove all static memory thresholds from lmkd completely. I think this is mostly on the lmkd side, but there may be some PSI or PSI monitor changes that would help
- give userspace some path to start reclaiming memory without waiting for every thread in a process to be scheduled--could be oom reaper, could be something else
- offer a way to wait for process termination so lmkd can tell when reclaim has finished and know when killing another process is appropriate
Quoted 7 lines · Tim Murray
Knowing whether a SIGKILL'd process has finished reclaiming is as far as I know not possible without something like procfds. That's where the 100ms timeout in lmkd comes in. lowmemorykiller and lmkd both attempt to wait up to 100ms for reclaim to finish by checking for the continued existence of the thread that received the SIGKILL, but this really means that they wait up to 100ms for the _thread_ to finish, which doesn't tell you anything about the memory used by that process.
Quoted 8 lines
If those threads terminate early and lowmemorykiller/lmkd get a signal to kill again, then there may be two processes competing for CPU time to reclaim memory. That doesn't reclaim any faster and may be an unnecessary kill.
...
- offer a way to wait for process termination so lmkd can tell when reclaim has finished and know when killing another process is appropriate
Should be pretty easy with something like this:
include/linux/sched.h +1 -0
@@ -1199,6 +1199,7 @@ struct task_struct {unsigned long lowest_stack;unsigned long prev_lowest_stack;#endif +ktime_t sigkill_time;/* * New fields for task_struct should be added above here, so that
kernel/fork.c +6 -0
@@ -435,6 +435,8 @@ void put_task_stack(struct task_struct *tsk)void free_task (struct task_struct *tsk) { +ktime_t sigkill_time = tsk->sigkill_time; +pid_t pid = tsk->pid;#ifndef CONFIG_THREAD_INFO_IN_TASK /* * The task is finally done with both the stack and thread_info,
@@ -455,6 +457,9 @@ void free_task(struct task_struct *tsk)if (tsk->flags &PF_KTHREAD )free_kthread_struct (tsk);free_task_struct (tsk); +if (sigkill_time) +printk ("%d killed after %lld us\n" , pid, +ktime_us_delta (ktime_get (), sigkill_time)); }EXPORT_SYMBOL (free_task);
@@ -1881,6 +1886,7 @@ static __latent_entropy struct task_struct *copy_process( p->sequential_io =0 ; p->sequential_io_avg =0 ;#endif + p->sigkill_time =0 ;/* Perform scheduler related setup. Assign this task to a CPU. */ retval =sched_fork (clone_flags, p);
kernel/signal.c +2 -0
@@ -1168,6 +1168,8 @@ static int __send_signal(int sig, struct kernel_siginfo *info, struct task_struc } out_set: +if (sig ==SIGKILL ) + t->sigkill_time =ktime_get ();signalfd_notify (t, sig);sigaddset (&pending->signal, sig);
Quoted 18 lines · Sultan Alsawaf
Quoted 7 lines · Tim Murray
Knowing whether a SIGKILL'd process has finished reclaiming is as far as I know not possible without something like procfds. That's where the 100ms timeout in lmkd comes in. lowmemorykiller and lmkd both attempt to wait up to 100ms for reclaim to finish by checking for the continued existence of the thread that received the SIGKILL, but this really means that they wait up to 100ms for the _thread_ to finish, which doesn't tell you anything about the memory used by that process.
Quoted 8 lines
If those threads terminate early and lowmemorykiller/lmkd get a signal to kill again, then there may be two processes competing for CPU time to reclaim memory. That doesn't reclaim any faster and may be an unnecessary kill.
...
- offer a way to wait for process termination so lmkd can tell when reclaim has finished and know when killing another process is appropriate
Quoted 1 line
Should be pretty easy with something like this:
Yeah, that's in the spirit of what I was suggesting, but there are lot of edge cases around how to get that data out efficiently and PID reuse (it's a real issue--often the Android apps that are causing memory pressure are also constantly creating/destroying threads).
I believe procfds or a similar mechanism will be a good solution to this.
Christian Brauner
maintainerQuoted 20 lines · Tim Murray
Quoted 18 lines · Sultan Alsawaf
Quoted 15 lines · Tim Murray
Knowing whether a SIGKILL'd process has finished reclaiming is as far as I know not possible without something like procfds. That's where the 100ms timeout in lmkd comes in. lowmemorykiller and lmkd both attempt to wait up to 100ms for reclaim to finish by checking for the continued existence of the thread that received the SIGKILL, but this really means that they wait up to 100ms for the _thread_ to finish, which doesn't tell you anything about the memory used by that process. If those threads terminate early and lowmemorykiller/lmkd get a signal to kill again, then there may be two processes competing for CPU time to reclaim memory. That doesn't reclaim any faster and may be an unnecessary kill.
...
- offer a way to wait for process termination so lmkd can tell when reclaim has finished and know when killing another process is appropriate
Quoted 8 lines
Quoted 1 line
Should be pretty easy with something like this:
Yeah, that's in the spirit of what I was suggesting, but there are lot of edge cases around how to get that data out efficiently and PID reuse (it's a real issue--often the Android apps that are causing memory pressure are also constantly creating/destroying threads).
I believe procfds or a similar mechanism will be a good solution to this.
Fwiw, I am working on this and have send a PR for inclusion in 5.1:
https://lore.kernel.org/lkml/[email protected]/
There's also a tree to track this work.
Christian
Hi Tim,
Thanks for the detailed and excellent write-up. It will serve as a good future reference for low memory killer requirements. I made some comments below on the "how to kill" part.
Quoted 64 lines · Tim Murray
Quoted 7 lines · Sultan Alsawaf
Quoted 4 lines · Michal Hocko
The only way to control the OOM behavior pro-actively is to throttle allocation speed. We have memcg high limit for that purpose. Along with PSI, I can imagine a reasonably working user space early oom notifications and reasonable acting upon that.
Quoted 10 lines
The issue with pro-active memory management that prompted me to create this was poor memory utilization. All of the alternative means of reclaiming pages in the page allocator's slow path turn out to be very useful for maximizing memory utilization, which is something that we would have to forgo by relying on a purely pro-active solution. I have not had a chance to look at PSI yet, but unless a PSI-enabled solution allows allocations to reach the same point as when the OOM killer is invoked (which is contradictory to what it sets out to do), then it cannot take advantage of all of the alternative memory-reclaim means employed in the slowpath, and will result in killing a process before it is _really_ necessary.
There are two essential parts of a lowmemorykiller implementation:
when to kill and how to kill.There are a million possible approaches to decide when to kill an unimportant process. They usually trade off between the same two failure modes depending on the workload.
If you kill too aggressively, a transient spike that could be imperceptibly absorbed by evicting some file pages or moving some pages to ZRAM will result in killing processes, which then get started up later and have a performance/battery cost.
If you don't kill aggressively enough, you will encounter a workload that thrashes the page cache, constantly evicting and reloading file pages and moving things in and out of ZRAM, which makes the system unusable when a process should have been killed instead.
As far as I've seen, any methodology that uses single points in time to decide when to kill without completely biasing toward one or the other is susceptible to both. The minfree approach used by lowmemorykiller/lmkd certainly is; it is both too aggressive for some workloads and not aggressive enough for other workloads. My guess is that simple LMK won't kill on transient spikes but will be extremely susceptible to page cache thrashing. This is not an improvement; page cache thrashing manifests as the entire system running very slowly.
What you actually want from lowmemorykiller/lmkd on Android is to only kill once it becomes clear that the system will continue to try to reclaim memory to the extent that it could impact what the user actually cares about. That means tracking how much time is spent in reclaim/paging operations and the like, and that's exactly what PSI does. lmkd has had support for using PSI as a replacement for vmpressure for use as a wakeup trigger (to check current memory levels against the minfree thresholds) since early February. It works fine; unsurprisingly it's better than vmpressure at avoiding false wakeups.
Longer term, there's a lot of work to be done in lmkd to turn PSI into a kill trigger and remove minfree entirely. It's tricky (mainly because of the "when to kill another process" problem discussed later), but I believe it's feasible.
How to kill is similarly messy. The latency of reclaiming memory post SIGKILL can be severe (usually tens of milliseconds, occasionally
Quoted 1 line
100ms). The latency we see on Android usually isn't because those
Quoted 10 lines
threads are blocked in uninterruptible sleep, it's because times of memory pressure are also usually times of significant CPU contention and these are overwhelmingly CFS threads, some of which may be assigned a very low priority. lmkd now sets priorities and resets cpusets upon killing a process, and we have seen improved reclaim latency because of this. oom reaper might be a good approach to avoid this latency (I think some in-kernel lowmemorykiller implementations rely on it), but we can't use it from userspace. Something for future consideration.
This makes sense. If the process receiving the SIGKILL does not get CPU time, then the kernel may not be able to execute the unconditional signal handling paths in the context of the victim process to free the memory.
I don't see how proc-fds approach will solve this though. Say you have process L (which is LMKd) which sends a SIGKILL to process V(which is a victim). Now L sends SIGKILL to V. Unless V executes the signal-handling code in kernel context and is scheduled at high enough priority to get CPU time, I don't think the SIGKILL will be processed.
The exact path that the process being killed executes to free its memory is: do_signal-> get_signal-> do_group_exit-> do_exit-> mmput. And this needs to execute in the context of V which needs to get CPU-time to do such execution.
So my point is to be notified of process death, you still need SIGKILL to be processed. So you may still need to make sure the task is at a high enough priority and scheduler puts it on the CPU. Only *after that* can he proc-fds notification mechanism (or whichever) notification mechanism can kick in.
Speaking of which I wonder if the scheduler should special case SIGKILLed threads as higher priority automatically so that they get CPU time, but don't know if this can cause denial of service kind of attacks. I don't know if it does something like this already. Peter should know this right off the bat and he is on CC so he can comment more.
About the 100ms latency, I wonder whether it is that high because of the way Android's lmkd is observing that a process has died. There is a gap between when a process memory is freed and when it disappears from the process-table. Once a process is SIGKILLed, it becomes a zombie. Its memory is freed instantly during the SIGKILL delivery (I traced this so that's how I know), but until it is reaped by its parent thread, it will still exist in /proc/<pid> . So if testing the existence of /proc/<pid> is how Android is observing that the process died, then there can be a large latency where it takes a very long time for the parent to actually reap the child way after its memory was long freed. A quicker way to know if a process's memory is freed before it is reaped could be to read back /proc/<pid>/maps in userspace of the victim <pid>, and that file will be empty for zombie processes. So then one does not need wait for the parent to reap it. I wonder how much of that 100ms you mentioned is actually the "Waiting while Parent is reaping the child", than "memory freeing time". So yeah for this second problem, the procfds work will help.
By the way another approach that can provide a quick and asynchronous notification of when the process memory is freed, is to monitor sched_process_exit trace event using eBPF. You can tell eBPF the PID that you want to monitor before the SIGKILL. As soon as the process dies and its memory is freed, the eBPF program can send a notification to user space (using the perf_events polling infra). The sched_process_exit fires just after the mmput() happens so it is quite close to when the memory is reclaimed. This also doesn't need any kernel changes. I could come up with a prototype for this and benchmark it on Android, if you want. Just let me know.
thanks,
- Joel
Quoted 33 lines
A non-obvious consequence of both of these concerns is that when to kill a second process is a distinct and more difficult problem than when to kill the first. A second process should be killed if reclaim from the first process has finished and there has been insufficient memory reclaimed to avoid perceptible impact. Identifying whether memory pressure continues at the same level can probably be handled through multiple PSI monitors with different thresholds and window lengths, but this is an area of future work.
Knowing whether a SIGKILL'd process has finished reclaiming is as far as I know not possible without something like procfds. That's where the 100ms timeout in lmkd comes in. lowmemorykiller and lmkd both attempt to wait up to 100ms for reclaim to finish by checking for the continued existence of the thread that received the SIGKILL, but this really means that they wait up to 100ms for the _thread_ to finish, which doesn't tell you anything about the memory used by that process. If those threads terminate early and lowmemorykiller/lmkd get a signal to kill again, then there may be two processes competing for CPU time to reclaim memory. That doesn't reclaim any faster and may be an unnecessary kill.
So, in summary, the impactful LMK improvements seem like
- get lmkd and PSI to the point that lmkd can use PSI signals as a kill trigger and remove all static memory thresholds from lmkd completely. I think this is mostly on the lmkd side, but there may be some PSI or PSI monitor changes that would help
- give userspace some path to start reclaiming memory without waiting for every thread in a process to be scheduled--could be oom reaper, could be something else
- offer a way to wait for process termination so lmkd can tell when reclaim has finished and know when killing another process is appropriate
Quoted 18 lines · Joel Fernandes
About the 100ms latency, I wonder whether it is that high because of the way Android's lmkd is observing that a process has died. There is a gap between when a process memory is freed and when it disappears from the process-table. Once a process is SIGKILLed, it becomes a zombie. Its memory is freed instantly during the SIGKILL delivery (I traced this so that's how I know), but until it is reaped by its parent thread, it will still exist in /proc/<pid> . So if testing the existence of /proc/<pid> is how Android is observing that the process died, then there can be a large latency where it takes a very long time for the parent to actually reap the child way after its memory was long freed. A quicker way to know if a process's memory is freed before it is reaped could be to read back /proc/<pid>/maps in userspace of the victim <pid>, and that file will be empty for zombie processes. So then one does not need wait for the parent to reap it. I wonder how much of that 100ms you mentioned is actually the "Waiting while Parent is reaping the child", than "memory freeing time". So yeah for this second problem, the procfds work will help.
Quoted 10 lines
By the way another approach that can provide a quick and asynchronous notification of when the process memory is freed, is to monitor
sched_process_exittrace event using eBPF. You can tell eBPF the PID that you want to monitor before the SIGKILL. As soon as the process dies and its memory is freed, the eBPF program can send a notification to user space (using theperf_eventspolling infra). Thesched_process_exitfires just after themmput()happens so it is quite close to when the memory is reclaimed. This also doesn't need any kernel changes. I could come up with a prototype for this and benchmark it on Android, if you want. Just let me know.
Perhaps I'm missing something, but if you want to know when a process has died after sending a SIGKILL to it, then why not just make the SIGKILL optionally block until the process has died completely? It'd be rather trivial to just store a pointer to an onstack completion inside the victim process' task_struct, and then complete it in free_task().
Thanks,
Sultan
Quoted 30 lines · Sultan Alsawaf
Quoted 18 lines · Joel Fernandes
About the 100ms latency, I wonder whether it is that high because of the way Android's lmkd is observing that a process has died. There is a gap between when a process memory is freed and when it disappears from the process-table. Once a process is SIGKILLed, it becomes a zombie. Its memory is freed instantly during the SIGKILL delivery (I traced this so that's how I know), but until it is reaped by its parent thread, it will still exist in /proc/<pid> . So if testing the existence of /proc/<pid> is how Android is observing that the process died, then there can be a large latency where it takes a very long time for the parent to actually reap the child way after its memory was long freed. A quicker way to know if a process's memory is freed before it is reaped could be to read back /proc/<pid>/maps in userspace of the victim <pid>, and that file will be empty for zombie processes. So then one does not need wait for the parent to reap it. I wonder how much of that 100ms you mentioned is actually the "Waiting while Parent is reaping the child", than "memory freeing time". So yeah for this second problem, the procfds work will help.
Quoted 10 lines
By the way another approach that can provide a quick and asynchronous notification of when the process memory is freed, is to monitor
sched_process_exittrace event using eBPF. You can tell eBPF the PID that you want to monitor before the SIGKILL. As soon as the process dies and its memory is freed, the eBPF program can send a notification to user space (using theperf_eventspolling infra). Thesched_process_exitfires just after themmput()happens so it is quite close to when the memory is reclaimed. This also doesn't need any kernel changes. I could come up with a prototype for this and benchmark it on Android, if you want. Just let me know.
Quoted 5 lines
Perhaps I'm missing something, but if you want to know when a process has died after sending a SIGKILL to it, then why not just make the SIGKILL optionally block until the process has died completely? It'd be rather trivial to just store a pointer to an onstack completion inside the victim process'
task_struct, and then complete it infree_task().
I'm not sure if that makes much semantic sense for how the signal handling is supposed to work. Imagine a parent sends SIGKILL to its child, and then does a wait(2). Because the SIGKILL blocks in your idea, then the wait cannot execute, and because the wait cannot execute, the zombie task will not get reaped and so the SIGKILL senders never gets unblocked and the whole thing just gets locked up. No? I don't know it just feels incorrect.
Further, in your idea adding stuff to task_struct will simply bloat it - when this task can easily be handled using eBPF without making any kernel changes. Either by probing sched_process_free or sched_process_exit tracepoints. Scheduler maintainers generally frown on adding stuff to task_struct pointlessly there's a good reason since bloating it effects the performance etc, and something like this would probably never be ifdef'd out behind a CONFIG.
thanks,
- Joel
Quoted 6 lines · Joel Fernandes
I'm not sure if that makes much semantic sense for how the signal handling is supposed to work. Imagine a parent sends SIGKILL to its child, and then does a wait(2). Because the SIGKILL blocks in your idea, then the wait cannot execute, and because the wait cannot execute, the zombie task will not get reaped and so the SIGKILL senders never gets unblocked and the whole thing just gets locked up. No? I don't know it just feels incorrect.
Block until the victim becomes a zombie instead.
Quoted 7 lines
Further, in your idea adding stuff to
task_structwill simply bloat it - when this task can easily be handled using eBPF without making any kernel changes. Either by probingsched_process_freeorsched_process_exittracepoints. Scheduler maintainers generally frown on adding stuff totask_structpointlessly there's a good reason since bloating it effects the performance etc, and something like this would probably never be ifdef'd out behind a CONFIG.
Adding something to task_struct is just the easiest way to test things for experimentation. This can be avoided in my suggestion by passing the pointer to a completion via the relevant functions, and then completing it at the time the victim transitions to a zombie state. I understand it's possible to use eBPF for this, but it seems kind of messy since this functionality is something that I think others would want provided by the kernel (i.e., anyone using PSI to implement their own OOM killer daemon similar to LMKD).
Thanks,
Sultan
Steven Rostedt
reviewerQuoted 5 lines · On Thu, 14 Mar 2019 13:49:11 -0700 Sultan Alsawaf <[email protected]>…
Perhaps I'm missing something, but if you want to know when a process has died after sending a SIGKILL to it, then why not just make the SIGKILL optionally block until the process has died completely? It'd be rather trivial to just store a pointer to an onstack completion inside the victim process'
task_struct, and then complete it infree_task().
How would you implement such a method in userspace? kill() doesn't take any parameters but the pid of the process you want to send a signal to, and the signal to send. This would require a new system call, and be quite a bit of work. If you can solve this with an ebpf program, I strongly suggest you do that instead.
-- Steve
Quoted 5 lines · Steven Rostedt
How would you implement such a method in userspace?
kill()doesn't take any parameters but the pid of the process you want to send a signal to, and the signal to send. This would require a new system call, and be quite a bit of work. If you can solve this with an ebpf program, I strongly suggest you do that instead.
This can be done by introducing a new signal number that provides SIGKILL functionality while blocking (maybe SIGKILLBLOCK?).
Thanks,
Sultan
Quoted 10 lines · Steven Rostedt
Quoted 5 lines · On Thu, 14 Mar 2019 13:49:11 -0700 Sultan Alsawaf <[email protected]>…
Perhaps I'm missing something, but if you want to know when a process has died after sending a SIGKILL to it, then why not just make the SIGKILL optionally block until the process has died completely? It'd be rather trivial to just store a pointer to an onstack completion inside the victim process'
task_struct, and then complete it infree_task().
Quoted 4 lines
How would you implement such a method in userspace?
kill()doesn't take any parameters but the pid of the process you want to send a signal to, and the signal to send. This would require a new system call, and be quite a bit of work.
That's what the pidfd work is for. Please read the original threads about the motivation and design of that facility.
Quoted 2 lines
If you can solve this with an ebpf program, I
strongly suggest you do that instead.
Regarding process death notification: I will absolutely not support putting aBPF and perf trace events on the critical path of core system memory management functionality. Tracing and monitoring facilities are great for learning about the system, but they were never intended to be load-bearing. The proposed eBPF process-monitoring approach is just a variant of the netlink proposal we discussed previously on the pidfd threads; it has all of its drawbacks. We really need a core system call --- really, we've needed robust process management since the creation of unix --- and I'm glad that we're finally getting it. Adding new system calls is not expensive; going to great lengths to avoid adding one is like calling a helicopter to avoid crossing the street. I don't think we should present an abuse of the debugging and performance monitoring infrastructure as an alternative to a robust and desperately-needed bit of core functionality that's neither hard to add nor complex to implement nor expensive to use.
Regarding the proposal for a new kernel-side lmkd: when possible, the kernel should provide mechanism, not policy. Putting the low memory killer back into the kernel after we've spent significant effort making it possible for userspace to do that job. Compared to kernel code, more easily understood, more easily debuggable, more easily updated, and much safer. If we *can* move something out of the kernel, we should. This patch moves us in exactly the wrong direction. Yes, we need *something* that sits synchronously astride the page allocation path and does *something* to stop a busy beaver allocator that eats all the available memory before lmkd, even mlocked and realtime, can respond. The OOM killer is adequate for this very rare case.
With respect to kill timing: Tim is right about the need for two levels of policy: first, a high-level process prioritization and memory-demand balancing scheme (which is what OOM score adjustment code in ActivityManager amounts to); and second, a low-level process-killing methodology that maximizes sustainable memory reclaim and minimizes unwanted side effects while killing those processes that should be dead. Both of these policies belong in userspace --- because they *can* be in userspace --- and userspace needs only a few tools, most of which already exist, to do a perfectly adequate job.
We do want killed processes to die promptly. That's why I support boosting a process's priority somehow when lmkd is about to kill it. The precise way in which we do that --- involving not only actual priority, but scheduler knobs, cgroup assignment, core affinity, and so on --- is a complex topic best left to userspace. lmkd already has all the knobs it needs to implement whatever priority boosting policy it wants.
Hell, once we add a pidfd_wait --- which I plan to work on, assuming nobody beats me to it, after pidfd_send_signal lands --- you can imagine a general-purpose priority inheritance mechanism expediting process death when a high-priority process waits on a pidfd_wait handle for a condemned process. You know you're on the right track design-wise when you start seeing this kind of elegant constructive interference between seemingly-unrelated features. What we don't need is some kind of blocking SIGKILL alternative or backdoor event delivery system.
We definitely don't want to have to wait for a process's parent to reap it. Instead, we want to wait for it to become a zombie. That's why I designed my original exithand patch to fire death notification upon transition to the zombie state, not upon process table removal, and I expect pidfd_wait (or whatever we call it) to act the same way.
In any case, there's a clear path forward here --- general-purpose, cheap, and elegant --- and we should just focus on doing that instead of more complex proposals with few advantages.
On Thu, Mar 14, 2019 at 09:36:43PM -0700, Daniel Colascione wrote:
[snip]
Quoted 18 lines
Quoted 2 lines
If you can solve this with an ebpf program, I
strongly suggest you do that instead.Regarding process death notification: I will absolutely not support putting aBPF and perf trace events on the critical path of core system memory management functionality. Tracing and monitoring facilities are great for learning about the system, but they were never intended to be load-bearing. The proposed eBPF process-monitoring approach is just a variant of the netlink proposal we discussed previously on the pidfd threads; it has all of its drawbacks. We really need a core system call --- really, we've needed robust process management since the creation of unix --- and I'm glad that we're finally getting it. Adding new system calls is not expensive; going to great lengths to avoid adding one is like calling a helicopter to avoid crossing the street. I don't think we should present an abuse of the debugging and performance monitoring infrastructure as an alternative to a robust and desperately-needed bit of core functionality that's neither hard to add nor complex to implement nor expensive to use.
The eBPF-based solution to this would be just as simple while avoiding any kernel changes. I don't know why you think it is not load-bearing. However, I agree the proc/pidfd approach is better and can be simpler for some users who don't want to deal with eBPF - especially since something like this has many usecases. I was just suggesting the eBPF solution as a better alternative to the task_struct surgery idea from Sultan since that sounded to me quite hackish (that could just be my opinion).
Quoted 40 lines
Regarding the proposal for a new kernel-side lmkd: when possible, the kernel should provide mechanism, not policy. Putting the low memory killer back into the kernel after we've spent significant effort making it possible for userspace to do that job. Compared to kernel code, more easily understood, more easily debuggable, more easily updated, and much safer. If we *can* move something out of the kernel, we should. This patch moves us in exactly the wrong direction. Yes, we need *something* that sits synchronously astride the page allocation path and does *something* to stop a busy beaver allocator that eats all the available memory before lmkd, even mlocked and realtime, can respond. The OOM killer is adequate for this very rare case.
With respect to kill timing: Tim is right about the need for two levels of policy: first, a high-level process prioritization and memory-demand balancing scheme (which is what OOM score adjustment code in ActivityManager amounts to); and second, a low-level process-killing methodology that maximizes sustainable memory reclaim and minimizes unwanted side effects while killing those processes that should be dead. Both of these policies belong in userspace --- because they *can* be in userspace --- and userspace needs only a few tools, most of which already exist, to do a perfectly adequate job.
We do want killed processes to die promptly. That's why I support boosting a process's priority somehow when lmkd is about to kill it. The precise way in which we do that --- involving not only actual priority, but scheduler knobs, cgroup assignment, core affinity, and so on --- is a complex topic best left to userspace. lmkd already has all the knobs it needs to implement whatever priority boosting policy it wants.
Hell, once we add a
pidfd_wait--- which I plan to work on, assuming nobody beats me to it, afterpidfd_send_signallands --- you can imagine a general-purpose priority inheritance mechanism expediting process death when a high-priority process waits on apidfd_waithandle for a condemned process. You know you're on the right track design-wise when you start seeing this kind of elegant constructive interference between seemingly-unrelated features. What we don't need is some kind of blocking SIGKILL alternative or backdoor event delivery system.
Quoted 5 lines
We definitely don't want to have to wait for a process's parent to reap it. Instead, we want to wait for it to become a zombie. That's why I designed my original exithand patch to fire death notification upon transition to the zombie state, not upon process table removal, and I expect
pidfd_wait(or whatever we call it) to act the same way.
Agreed. Looking forward to the patches. :)
thanks,
- Joel
Suren Baghdasaryan
reviewerQuoted 69 lines · Daniel Colascione
Quoted 10 lines · Steven Rostedt
Quoted 5 lines · On Thu, 14 Mar 2019 13:49:11 -0700 Sultan Alsawaf <[email protected]>…
Perhaps I'm missing something, but if you want to know when a process has died after sending a SIGKILL to it, then why not just make the SIGKILL optionally block until the process has died completely? It'd be rather trivial to just store a pointer to an onstack completion inside the victim process'
task_struct, and then complete it infree_task().Quoted 4 lines
How would you implement such a method in userspace?
kill()doesn't take any parameters but the pid of the process you want to send a signal to, and the signal to send. This would require a new system call, and be quite a bit of work.That's what the pidfd work is for. Please read the original threads about the motivation and design of that facility.
Quoted 2 lines
If you can solve this with an ebpf program, I
strongly suggest you do that instead.Regarding process death notification: I will absolutely not support putting aBPF and perf trace events on the critical path of core system memory management functionality. Tracing and monitoring facilities are great for learning about the system, but they were never intended to be load-bearing. The proposed eBPF process-monitoring approach is just a variant of the netlink proposal we discussed previously on the pidfd threads; it has all of its drawbacks. We really need a core system call --- really, we've needed robust process management since the creation of unix --- and I'm glad that we're finally getting it. Adding new system calls is not expensive; going to great lengths to avoid adding one is like calling a helicopter to avoid crossing the street. I don't think we should present an abuse of the debugging and performance monitoring infrastructure as an alternative to a robust and desperately-needed bit of core functionality that's neither hard to add nor complex to implement nor expensive to use.
Regarding the proposal for a new kernel-side lmkd: when possible, the kernel should provide mechanism, not policy. Putting the low memory killer back into the kernel after we've spent significant effort making it possible for userspace to do that job. Compared to kernel code, more easily understood, more easily debuggable, more easily updated, and much safer. If we *can* move something out of the kernel, we should. This patch moves us in exactly the wrong direction. Yes, we need *something* that sits synchronously astride the page allocation path and does *something* to stop a busy beaver allocator that eats all the available memory before lmkd, even mlocked and realtime, can respond. The OOM killer is adequate for this very rare case.
With respect to kill timing: Tim is right about the need for two levels of policy: first, a high-level process prioritization and memory-demand balancing scheme (which is what OOM score adjustment code in ActivityManager amounts to); and second, a low-level process-killing methodology that maximizes sustainable memory reclaim and minimizes unwanted side effects while killing those processes that should be dead. Both of these policies belong in userspace --- because they *can* be in userspace --- and userspace needs only a few tools, most of which already exist, to do a perfectly adequate job.
We do want killed processes to die promptly. That's why I support boosting a process's priority somehow when lmkd is about to kill it. The precise way in which we do that --- involving not only actual priority, but scheduler knobs, cgroup assignment, core affinity, and so on --- is a complex topic best left to userspace. lmkd already has all the knobs it needs to implement whatever priority boosting policy it wants.
Quoted 9 lines
Hell, once we add a
pidfd_wait--- which I plan to work on, assuming nobody beats me to it, afterpidfd_send_signallands --- you can imagine a general-purpose priority inheritance mechanism expediting process death when a high-priority process waits on apidfd_waithandle for a condemned process. You know you're on the right track design-wise when you start seeing this kind of elegant constructive interference between seemingly-unrelated features. What we don't need is some kind of blocking SIGKILL alternative or backdoor event delivery system.
When talking about pidfd_wait functionality do you mean something like
this: https://lore.kernel.org/patchwork/patch/345098/ ? I missed the discussion about it, could you please point me to it?
Quoted 9 lines
We definitely don't want to have to wait for a process's parent to reap it. Instead, we want to wait for it to become a zombie. That's why I designed my original exithand patch to fire death notification upon transition to the zombie state, not upon process table removal, and I expect
pidfd_wait(or whatever we call it) to act the same way.In any case, there's a clear path forward here --- general-purpose, cheap, and elegant --- and we should just focus on doing that instead of more complex proposals with few advantages.
Quoted 81 lines · Suren Baghdasaryan
Quoted 69 lines · Daniel Colascione
Quoted 10 lines · Steven Rostedt
Quoted 5 lines · On Thu, 14 Mar 2019 13:49:11 -0700 Sultan Alsawaf <[email protected]>…
Perhaps I'm missing something, but if you want to know when a process has died after sending a SIGKILL to it, then why not just make the SIGKILL optionally block until the process has died completely? It'd be rather trivial to just store a pointer to an onstack completion inside the victim process'
task_struct, and then complete it infree_task().Quoted 4 lines
How would you implement such a method in userspace?
kill()doesn't take any parameters but the pid of the process you want to send a signal to, and the signal to send. This would require a new system call, and be quite a bit of work.That's what the pidfd work is for. Please read the original threads about the motivation and design of that facility.
Quoted 2 lines
If you can solve this with an ebpf program, I
strongly suggest you do that instead.Regarding process death notification: I will absolutely not support putting aBPF and perf trace events on the critical path of core system memory management functionality. Tracing and monitoring facilities are great for learning about the system, but they were never intended to be load-bearing. The proposed eBPF process-monitoring approach is just a variant of the netlink proposal we discussed previously on the pidfd threads; it has all of its drawbacks. We really need a core system call --- really, we've needed robust process management since the creation of unix --- and I'm glad that we're finally getting it. Adding new system calls is not expensive; going to great lengths to avoid adding one is like calling a helicopter to avoid crossing the street. I don't think we should present an abuse of the debugging and performance monitoring infrastructure as an alternative to a robust and desperately-needed bit of core functionality that's neither hard to add nor complex to implement nor expensive to use.
Regarding the proposal for a new kernel-side lmkd: when possible, the kernel should provide mechanism, not policy. Putting the low memory killer back into the kernel after we've spent significant effort making it possible for userspace to do that job. Compared to kernel code, more easily understood, more easily debuggable, more easily updated, and much safer. If we *can* move something out of the kernel, we should. This patch moves us in exactly the wrong direction. Yes, we need *something* that sits synchronously astride the page allocation path and does *something* to stop a busy beaver allocator that eats all the available memory before lmkd, even mlocked and realtime, can respond. The OOM killer is adequate for this very rare case.
With respect to kill timing: Tim is right about the need for two levels of policy: first, a high-level process prioritization and memory-demand balancing scheme (which is what OOM score adjustment code in ActivityManager amounts to); and second, a low-level process-killing methodology that maximizes sustainable memory reclaim and minimizes unwanted side effects while killing those processes that should be dead. Both of these policies belong in userspace --- because they *can* be in userspace --- and userspace needs only a few tools, most of which already exist, to do a perfectly adequate job.
We do want killed processes to die promptly. That's why I support boosting a process's priority somehow when lmkd is about to kill it. The precise way in which we do that --- involving not only actual priority, but scheduler knobs, cgroup assignment, core affinity, and so on --- is a complex topic best left to userspace. lmkd already has all the knobs it needs to implement whatever priority boosting policy it wants.
Quoted 9 lines
Hell, once we add a
pidfd_wait--- which I plan to work on, assuming nobody beats me to it, afterpidfd_send_signallands --- you can imagine a general-purpose priority inheritance mechanism expediting process death when a high-priority process waits on apidfd_waithandle for a condemned process. You know you're on the right track design-wise when you start seeing this kind of elegant constructive interference between seemingly-unrelated features. What we don't need is some kind of blocking SIGKILL alternative or backdoor event delivery system.
Quoted 3 lines
When talking about
pidfd_waitfunctionality do you mean something like
this: https://lore.kernel.org/patchwork/patch/345098/ ? I missed the discussion about it, could you please point me to it?
That directory-polling approach came up in the discussion. It's a bad idea, mostly for API reasons. I'm talking about something more like
https://lore.kernel.org/lkml/[email protected]/,
albeit in system call form instead of in the form of a new per-task proc file.
Steven Rostedt
reviewerQuoted 11 lines · On Thu, 14 Mar 2019 21:36:43 -0700 Daniel Colascione <[email protected]> wrote:
Quoted 10 lines · Steven Rostedt
Quoted 5 lines · On Thu, 14 Mar 2019 13:49:11 -0700 Sultan Alsawaf <[email protected]>…
Perhaps I'm missing something, but if you want to know when a process has died after sending a SIGKILL to it, then why not just make the SIGKILL optionally block until the process has died completely? It'd be rather trivial to just store a pointer to an onstack completion inside the victim process'
task_struct, and then complete it infree_task().
Quoted 7 lines
Quoted 4 lines
How would you implement such a method in userspace?
kill()doesn't take any parameters but the pid of the process you want to send a signal to, and the signal to send. This would require a new system call, and be quite a bit of work.That's what the pidfd work is for. Please read the original threads about the motivation and design of that facility.
I wasn't Cc'd on the original work, so I haven't read them.
Quoted 31 lines
Quoted 3 lines
If you can solve this with an ebpf program, I
strongly suggest you do that instead.We do want killed processes to die promptly. That's why I support boosting a process's priority somehow when lmkd is about to kill it. The precise way in which we do that --- involving not only actual priority, but scheduler knobs, cgroup assignment, core affinity, and so on --- is a complex topic best left to userspace. lmkd already has all the knobs it needs to implement whatever priority boosting policy it wants.
Hell, once we add a
pidfd_wait--- which I plan to work on, assuming nobody beats me to it, afterpidfd_send_signallands --- you can imagine a general-purpose priority inheritance mechanism expediting process death when a high-priority process waits on apidfd_waithandle for a condemned process. You know you're on the right track design-wise when you start seeing this kind of elegant constructive interference between seemingly-unrelated features. What we don't need is some kind of blocking SIGKILL alternative or backdoor event delivery system.We definitely don't want to have to wait for a process's parent to reap it. Instead, we want to wait for it to become a zombie. That's why I designed my original exithand patch to fire death notification upon transition to the zombie state, not upon process table removal, and I expect
pidfd_wait(or whatever we call it) to act the same way.
Quoted 3 lines
In any case, there's a clear path forward here --- general-purpose, cheap, and elegant --- and we should just focus on doing that instead of more complex proposals with few advantages.
If you add new pidfd systemcalls then making a new way to send a signal and block till it does die or whatever is more acceptable than adding a new signal that changes the semantics of sending signals, which is what I was against.
I do agree with Joel about bloating task_struct too. If anything, have a wait queue you add, where you can allocate a descriptor with the task dieing and task killing, and just search this queue on dying. We could add a TIF flag to the task as well to let the exiting of this task know it should do such an operation.
-- Steve
Quoted 56 lines · Steven Rostedt
Quoted 11 lines · On Thu, 14 Mar 2019 21:36:43 -0700 Daniel Colascione <[email protected]> wrote:
Quoted 10 lines · Steven Rostedt
Quoted 5 lines · On Thu, 14 Mar 2019 13:49:11 -0700 Sultan Alsawaf <[email protected]>…
Perhaps I'm missing something, but if you want to know when a process has died after sending a SIGKILL to it, then why not just make the SIGKILL optionally block until the process has died completely? It'd be rather trivial to just store a pointer to an onstack completion inside the victim process'
task_struct, and then complete it infree_task().Quoted 7 lines
Quoted 4 lines
How would you implement such a method in userspace?
kill()doesn't take any parameters but the pid of the process you want to send a signal to, and the signal to send. This would require a new system call, and be quite a bit of work.That's what the pidfd work is for. Please read the original threads about the motivation and design of that facility.
I wasn't Cc'd on the original work, so I haven't read them.
Quoted 31 lines
Quoted 3 lines
If you can solve this with an ebpf program, I
strongly suggest you do that instead.We do want killed processes to die promptly. That's why I support boosting a process's priority somehow when lmkd is about to kill it. The precise way in which we do that --- involving not only actual priority, but scheduler knobs, cgroup assignment, core affinity, and so on --- is a complex topic best left to userspace. lmkd already has all the knobs it needs to implement whatever priority boosting policy it wants.
Hell, once we add a
pidfd_wait--- which I plan to work on, assuming nobody beats me to it, afterpidfd_send_signallands --- you can imagine a general-purpose priority inheritance mechanism expediting process death when a high-priority process waits on apidfd_waithandle for a condemned process. You know you're on the right track design-wise when you start seeing this kind of elegant constructive interference between seemingly-unrelated features. What we don't need is some kind of blocking SIGKILL alternative or backdoor event delivery system.We definitely don't want to have to wait for a process's parent to reap it. Instead, we want to wait for it to become a zombie. That's why I designed my original exithand patch to fire death notification upon transition to the zombie state, not upon process table removal, and I expect
pidfd_wait(or whatever we call it) to act the same way.
Quoted 6 lines
Quoted 3 lines
In any case, there's a clear path forward here --- general-purpose, cheap, and elegant --- and we should just focus on doing that instead of more complex proposals with few advantages.
If you add new pidfd systemcalls then making a new way to send a signal and block till it does die or whatever is
Right. And we shouldn't couple the killing and the waiting: while we now have a good race-free way to kill processes using pidfd_send_signal, but we still have no good facility for waiting for the death of a process that isn't a child of the waiter. Any kind of unified "kill and wait for death" primitive precludes the killing thread waiting for things other than death at the same time! Instead, if we allow waiting for an arbitrary process's death using general-purpose wait primitives like select/poll/epoll/io_submit/etc., then synchronous killing becomes just another sleep that composes in useful and predictable ways.
Quoted 3 lines
more acceptable than adding a
new signal that changes the semantics of sending signals, which is what I was against.
Agreed. Even if it were possible to easily add signals without breaking everyone, a special kind of signal with delivery semantics different from those of existing signals is a bad idea, and not really a signal at all, but just a new system call in disguise.
Quoted 5 lines
I do agree with Joel about bloating
task_structtoo. If anything, have a wait queue you add, where you can allocate a descriptor with the task dieing and task killing, and just search this queue on dying. We could add a TIF flag to the task as well to let the exiting of this task know it should do such an operation.
That's my basic plan. I think we need one link from struct signal or something so we don't end up doing some kind of *global* search on process death, but let's see how it goes.
Christian Brauner
maintainerQuoted 68 lines · Daniel Colascione
Quoted 10 lines · Steven Rostedt
Quoted 5 lines · On Thu, 14 Mar 2019 13:49:11 -0700 Sultan Alsawaf <[email protected]>…
Perhaps I'm missing something, but if you want to know when a process has died after sending a SIGKILL to it, then why not just make the SIGKILL optionally block until the process has died completely? It'd be rather trivial to just store a pointer to an onstack completion inside the victim process'
task_struct, and then complete it infree_task().Quoted 4 lines
How would you implement such a method in userspace?
kill()doesn't take any parameters but the pid of the process you want to send a signal to, and the signal to send. This would require a new system call, and be quite a bit of work.That's what the pidfd work is for. Please read the original threads about the motivation and design of that facility.
Quoted 2 lines
If you can solve this with an ebpf program, I
strongly suggest you do that instead.Regarding process death notification: I will absolutely not support putting aBPF and perf trace events on the critical path of core system memory management functionality. Tracing and monitoring facilities are great for learning about the system, but they were never intended to be load-bearing. The proposed eBPF process-monitoring approach is just a variant of the netlink proposal we discussed previously on the pidfd threads; it has all of its drawbacks. We really need a core system call --- really, we've needed robust process management since the creation of unix --- and I'm glad that we're finally getting it. Adding new system calls is not expensive; going to great lengths to avoid adding one is like calling a helicopter to avoid crossing the street. I don't think we should present an abuse of the debugging and performance monitoring infrastructure as an alternative to a robust and desperately-needed bit of core functionality that's neither hard to add nor complex to implement nor expensive to use.
Regarding the proposal for a new kernel-side lmkd: when possible, the kernel should provide mechanism, not policy. Putting the low memory killer back into the kernel after we've spent significant effort making it possible for userspace to do that job. Compared to kernel code, more easily understood, more easily debuggable, more easily updated, and much safer. If we *can* move something out of the kernel, we should. This patch moves us in exactly the wrong direction. Yes, we need *something* that sits synchronously astride the page allocation path and does *something* to stop a busy beaver allocator that eats all the available memory before lmkd, even mlocked and realtime, can respond. The OOM killer is adequate for this very rare case.
With respect to kill timing: Tim is right about the need for two levels of policy: first, a high-level process prioritization and memory-demand balancing scheme (which is what OOM score adjustment code in ActivityManager amounts to); and second, a low-level process-killing methodology that maximizes sustainable memory reclaim and minimizes unwanted side effects while killing those processes that should be dead. Both of these policies belong in userspace --- because they *can* be in userspace --- and userspace needs only a few tools, most of which already exist, to do a perfectly adequate job.
We do want killed processes to die promptly. That's why I support boosting a process's priority somehow when lmkd is about to kill it. The precise way in which we do that --- involving not only actual priority, but scheduler knobs, cgroup assignment, core affinity, and so on --- is a complex topic best left to userspace. lmkd already has all the knobs it needs to implement whatever priority boosting policy it wants.
Quoted 2 lines
Hell, once we add a
pidfd_wait--- which I plan to work on, assuming nobody beats me to it, afterpidfd_send_signallands --- you can
Daniel,
I've just been talking to Joel.
I actually "expected" you to work pidfd_wait() after prior conversations we had on the pidfd_send_signal() patchsets. :) That's why I got a separate git tree on kernel.org since I expect a lot more work to come. I hope that Linus still decides to pull pidfd_send_signal() before Sunday (For the ones who have missed the link in a prior response of mine:
https://lkml.org/lkml/2019/3/12/439
This is the first merge window I sent this PR.
The pidfd tree has a branch for-next that is already tracked by Stephen in linux-next since the 5.0 merge window. The patches for pidfd_send_signal() sit in the pidfd branch.
I'd be happy to share the tree with you and Joel (We can rename it if you prefer I don't care).
I would really like to centralize this work so that we sort of have a "united front" and end up with a coherent api and can send PRs from a centralized place:
https://git.kernel.org/pub/scm/linux/kernel/git/brauner/linux.git/
Christian
Quoted 17 lines
imagine a general-purpose priority inheritance mechanism expediting process death when a high-priority process waits on a
pidfd_waithandle for a condemned process. You know you're on the right track design-wise when you start seeing this kind of elegant constructive interference between seemingly-unrelated features. What we don't need is some kind of blocking SIGKILL alternative or backdoor event delivery system.We definitely don't want to have to wait for a process's parent to reap it. Instead, we want to wait for it to become a zombie. That's why I designed my original exithand patch to fire death notification upon transition to the zombie state, not upon process table removal, and I expect
pidfd_wait(or whatever we call it) to act the same way.In any case, there's a clear path forward here --- general-purpose, cheap, and elegant --- and we should just focus on doing that instead of more complex proposals with few advantages.
Quoted 85 lines · Christian Brauner
Quoted 68 lines · Daniel Colascione
Quoted 10 lines · Steven Rostedt
Quoted 5 lines · On Thu, 14 Mar 2019 13:49:11 -0700 Sultan Alsawaf <[email protected]>…
Perhaps I'm missing something, but if you want to know when a process has died after sending a SIGKILL to it, then why not just make the SIGKILL optionally block until the process has died completely? It'd be rather trivial to just store a pointer to an onstack completion inside the victim process'
task_struct, and then complete it infree_task().Quoted 4 lines
How would you implement such a method in userspace?
kill()doesn't take any parameters but the pid of the process you want to send a signal to, and the signal to send. This would require a new system call, and be quite a bit of work.That's what the pidfd work is for. Please read the original threads about the motivation and design of that facility.
Quoted 2 lines
If you can solve this with an ebpf program, I
strongly suggest you do that instead.Regarding process death notification: I will absolutely not support putting aBPF and perf trace events on the critical path of core system memory management functionality. Tracing and monitoring facilities are great for learning about the system, but they were never intended to be load-bearing. The proposed eBPF process-monitoring approach is just a variant of the netlink proposal we discussed previously on the pidfd threads; it has all of its drawbacks. We really need a core system call --- really, we've needed robust process management since the creation of unix --- and I'm glad that we're finally getting it. Adding new system calls is not expensive; going to great lengths to avoid adding one is like calling a helicopter to avoid crossing the street. I don't think we should present an abuse of the debugging and performance monitoring infrastructure as an alternative to a robust and desperately-needed bit of core functionality that's neither hard to add nor complex to implement nor expensive to use.
Regarding the proposal for a new kernel-side lmkd: when possible, the kernel should provide mechanism, not policy. Putting the low memory killer back into the kernel after we've spent significant effort making it possible for userspace to do that job. Compared to kernel code, more easily understood, more easily debuggable, more easily updated, and much safer. If we *can* move something out of the kernel, we should. This patch moves us in exactly the wrong direction. Yes, we need *something* that sits synchronously astride the page allocation path and does *something* to stop a busy beaver allocator that eats all the available memory before lmkd, even mlocked and realtime, can respond. The OOM killer is adequate for this very rare case.
With respect to kill timing: Tim is right about the need for two levels of policy: first, a high-level process prioritization and memory-demand balancing scheme (which is what OOM score adjustment code in ActivityManager amounts to); and second, a low-level process-killing methodology that maximizes sustainable memory reclaim and minimizes unwanted side effects while killing those processes that should be dead. Both of these policies belong in userspace --- because they *can* be in userspace --- and userspace needs only a few tools, most of which already exist, to do a perfectly adequate job.
We do want killed processes to die promptly. That's why I support boosting a process's priority somehow when lmkd is about to kill it. The precise way in which we do that --- involving not only actual priority, but scheduler knobs, cgroup assignment, core affinity, and so on --- is a complex topic best left to userspace. lmkd already has all the knobs it needs to implement whatever priority boosting policy it wants.
Quoted 2 lines
Hell, once we add a
pidfd_wait--- which I plan to work on, assuming nobody beats me to it, afterpidfd_send_signallands --- you canDaniel,
I've just been talking to Joel.
I actually "expected" you to workpidfd_wait()after prior conversations we had on thepidfd_send_signal()patchsets. :) That's why I got a separate git tree on kernel.org since I expect a lot more work to come. I hope that Linus still decides to pullpidfd_send_signal()before Sunday (For the ones who have missed the link in a prior response of mine:
https://lkml.org/lkml/2019/3/12/439This is the first merge window I sent this PR.
Quoted 9 lines
The pidfd tree has a branch for-next that is already tracked by Stephen in linux-next since the 5.0 merge window. The patches for
pidfd_send_signal()sit in the pidfd branch.
I'd be happy to share the tree with you and Joel (We can rename it if you prefer I don't care).
I would really like to centralize this work so that we sort of have a "united front" and end up with a coherent api and can send PRs from a centralized place:
https://git.kernel.org/pub/scm/linux/kernel/git/brauner/linux.git/
I am totally onboard with working together / reviewing this work with you all on a common tree somewhere (Christian's pidfd tree is fine). I was curious, why do we want to add a new syscall (pidfd_wait) though? Why not just use standard poll/epoll interface on the proc fd like Daniel was suggesting. AFAIK, once the proc file is opened, the struct pid is essentially pinned even though the proc number may be reused. Then the caller can just poll. We can add a waitqueue to struct pid, and wake up any waiters on process death (A quick look shows task_struct can be mapped to its struct pid) and also possibly optimize it using Steve's TIF flag idea. No new syscall is needed then, let me know if I missed something?
thanks,
- Joel
Christian Brauner
maintainerQuoted 96 lines · Joel Fernandes
Quoted 85 lines · Christian Brauner
Quoted 68 lines · Daniel Colascione
Quoted 10 lines · Steven Rostedt
Quoted 5 lines · On Thu, 14 Mar 2019 13:49:11 -0700 Sultan Alsawaf <[email protected]>…
Perhaps I'm missing something, but if you want to know when a process has died after sending a SIGKILL to it, then why not just make the SIGKILL optionally block until the process has died completely? It'd be rather trivial to just store a pointer to an onstack completion inside the victim process'
task_struct, and then complete it infree_task().Quoted 4 lines
How would you implement such a method in userspace?
kill()doesn't take any parameters but the pid of the process you want to send a signal to, and the signal to send. This would require a new system call, and be quite a bit of work.That's what the pidfd work is for. Please read the original threads about the motivation and design of that facility.
Quoted 2 lines
If you can solve this with an ebpf program, I
strongly suggest you do that instead.Regarding process death notification: I will absolutely not support putting aBPF and perf trace events on the critical path of core system memory management functionality. Tracing and monitoring facilities are great for learning about the system, but they were never intended to be load-bearing. The proposed eBPF process-monitoring approach is just a variant of the netlink proposal we discussed previously on the pidfd threads; it has all of its drawbacks. We really need a core system call --- really, we've needed robust process management since the creation of unix --- and I'm glad that we're finally getting it. Adding new system calls is not expensive; going to great lengths to avoid adding one is like calling a helicopter to avoid crossing the street. I don't think we should present an abuse of the debugging and performance monitoring infrastructure as an alternative to a robust and desperately-needed bit of core functionality that's neither hard to add nor complex to implement nor expensive to use.
Regarding the proposal for a new kernel-side lmkd: when possible, the kernel should provide mechanism, not policy. Putting the low memory killer back into the kernel after we've spent significant effort making it possible for userspace to do that job. Compared to kernel code, more easily understood, more easily debuggable, more easily updated, and much safer. If we *can* move something out of the kernel, we should. This patch moves us in exactly the wrong direction. Yes, we need *something* that sits synchronously astride the page allocation path and does *something* to stop a busy beaver allocator that eats all the available memory before lmkd, even mlocked and realtime, can respond. The OOM killer is adequate for this very rare case.
With respect to kill timing: Tim is right about the need for two levels of policy: first, a high-level process prioritization and memory-demand balancing scheme (which is what OOM score adjustment code in ActivityManager amounts to); and second, a low-level process-killing methodology that maximizes sustainable memory reclaim and minimizes unwanted side effects while killing those processes that should be dead. Both of these policies belong in userspace --- because they *can* be in userspace --- and userspace needs only a few tools, most of which already exist, to do a perfectly adequate job.
We do want killed processes to die promptly. That's why I support boosting a process's priority somehow when lmkd is about to kill it. The precise way in which we do that --- involving not only actual priority, but scheduler knobs, cgroup assignment, core affinity, and so on --- is a complex topic best left to userspace. lmkd already has all the knobs it needs to implement whatever priority boosting policy it wants.
Quoted 2 lines
Hell, once we add a
pidfd_wait--- which I plan to work on, assuming nobody beats me to it, afterpidfd_send_signallands --- you canDaniel,
I've just been talking to Joel.
I actually "expected" you to workpidfd_wait()after prior conversations we had on thepidfd_send_signal()patchsets. :) That's why I got a separate git tree on kernel.org since I expect a lot more work to come. I hope that Linus still decides to pullpidfd_send_signal()before Sunday (For the ones who have missed the link in a prior response of mine:
https://lkml.org/lkml/2019/3/12/439This is the first merge window I sent this PR.
Quoted 9 lines
The pidfd tree has a branch for-next that is already tracked by Stephen in linux-next since the 5.0 merge window. The patches for
pidfd_send_signal()sit in the pidfd branch.
I'd be happy to share the tree with you and Joel (We can rename it if you prefer I don't care).
I would really like to centralize this work so that we sort of have a "united front" and end up with a coherent api and can send PRs from a centralized place:
https://git.kernel.org/pub/scm/linux/kernel/git/brauner/linux.git/
Quoted 2 lines
I am totally onboard with working together / reviewing this work with you all on a common tree somewhere (Christian's pidfd tree is fine). I was curious,
Excellent.
Quoted 8 lines
why do we want to add a new syscall (
pidfd_wait) though? Why not just use standard poll/epoll interface on the proc fd like Daniel was suggesting. AFAIK, once the proc file is opened, the struct pid is essentially pinned even though the proc number may be reused. Then the caller can just poll. We can add a waitqueue to struct pid, and wake up any waiters on process death (A quick look showstask_structcan be mapped to its struct pid) and also possibly optimize it using Steve's TIF flag idea. No new syscall is needed then, let me know if I missed something?
Huh, I thought that Daniel was against the poll/epoll solution? I have no clear opinion on what is better at the moment since I have been mostly concerned with getting pidfd_send_signal() into shape and was reluctant to put more ideas/work into this if it gets shutdown. Once we have pidfd_send_signal() the wait discussion makes sense.
Thanks!
Christian
On Fri, Mar 15, 2019 at 07:24:28PM +0100, Christian Brauner wrote:
[..]
Quoted 10 lines
Quoted 8 lines
why do we want to add a new syscall (
pidfd_wait) though? Why not just use standard poll/epoll interface on the proc fd like Daniel was suggesting. AFAIK, once the proc file is opened, the struct pid is essentially pinned even though the proc number may be reused. Then the caller can just poll. We can add a waitqueue to struct pid, and wake up any waiters on process death (A quick look showstask_structcan be mapped to its struct pid) and also possibly optimize it using Steve's TIF flag idea. No new syscall is needed then, let me know if I missed something?Huh, I thought that Daniel was against the poll/epoll solution?
Hmm, going through earlier threads, I believe so now. Here was Daniel's reasoning about avoiding a notification about process death through proc directory fd: http://lkml.iu.edu/hypermail/linux/kernel/1811.0/00232.html
May be a dedicated syscall for this would be cleaner after all.
Quoted 4 lines
I have no clear opinion on what is better at the moment since I have been mostly concerned with getting
pidfd_send_signal()into shape and was reluctant to put more ideas/work into this if it gets shutdown. Once we havepidfd_send_signal()the wait discussion makes sense.
Ok. It looks like that is almost in though (fingers crossed :)).
thanks,
- Joel
Suren Baghdasaryan
reviewerQuoted 12 lines · Joel Fernandes
On Fri, Mar 15, 2019 at 07:24:28PM +0100, Christian Brauner wrote:
[..]Quoted 9 lines
Quoted 8 lines
why do we want to add a new syscall (
pidfd_wait) though? Why not just use standard poll/epoll interface on the proc fd like Daniel was suggesting. AFAIK, once the proc file is opened, the struct pid is essentially pinned even though the proc number may be reused. Then the caller can just poll. We can add a waitqueue to struct pid, and wake up any waiters on process death (A quick look showstask_structcan be mapped to its struct pid) and also possibly optimize it using Steve's TIF flag idea. No new syscall is needed then, let me know if I missed something?
Quoted 7 lines
Quoted 1 line
Huh, I thought that Daniel was against the poll/epoll solution?
Hmm, going through earlier threads, I believe so now. Here was Daniel's reasoning about avoiding a notification about process death through proc directory fd: http://lkml.iu.edu/hypermail/linux/kernel/1811.0/00232.html
May be a dedicated syscall for this would be cleaner after all.
Ah, I wish I've seen that discussion before...
syscall makes sense and it can be non-blocking and we can use select/poll/epoll if we use eventfd. I would strongly advocate for non-blocking version or at least to have a non-blocking option. Something like this:
evfd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
// register eventfd to receive death notificationpidfd_wait(pid_to_kill, evfd);
// kill the processpidfd_send_signal(pid_to_kill, ...)
// tend to other things
...
// wait for the process to diepoll_wait(evfd, ...);
This simplifies userspace, allows it to wait for multiple events using epoll and I think kernel implementation will be also quite simple because it already implements eventfd_signal() that takes care of waitqueue handling.
If pidfd_send_signal could be extended to have an optional eventfd parameter then we would not even have to add a new syscall.
Quoted 11 lines
Quoted 4 lines
I have no clear opinion on what is better at the moment since I have been mostly concerned with getting
pidfd_send_signal()into shape and was reluctant to put more ideas/work into this if it gets shutdown. Once we havepidfd_send_signal()the wait discussion makes sense.Ok. It looks like that is almost in though (fingers crossed :)).
thanks,
- Joel
Quoted 20 lines · Suren Baghdasaryan
Quoted 18 lines · Joel Fernandes
On Fri, Mar 15, 2019 at 07:24:28PM +0100, Christian Brauner wrote:
[..]Quoted 10 lines
Quoted 8 lines
why do we want to add a new syscall (
pidfd_wait) though? Why not just use standard poll/epoll interface on the proc fd like Daniel was suggesting. AFAIK, once the proc file is opened, the struct pid is essentially pinned even though the proc number may be reused. Then the caller can just poll. We can add a waitqueue to struct pid, and wake up any waiters on process death (A quick look showstask_structcan be mapped to its struct pid) and also possibly optimize it using Steve's TIF flag idea. No new syscall is needed then, let me know if I missed something?Huh, I thought that Daniel was against the poll/epoll solution?
Hmm, going through earlier threads, I believe so now. Here was Daniel's reasoning about avoiding a notification about process death through proc directory fd: http://lkml.iu.edu/hypermail/linux/kernel/1811.0/00232.html
Quoted 5 lines
Quoted 1 line
May be a dedicated syscall for this would be cleaner after all.
Ah, I wish I've seen that discussion before... syscall makes sense and it can be non-blocking and we can use select/poll/epoll if we use eventfd.
Thanks for taking a look.
Quoted 2 lines
I would strongly advocate for
non-blocking version or at least to have a non-blocking option.
Waiting for FD readiness is *already* blocking or non-blocking according to the caller's desire --- users can pass options they want to poll(2) or whatever. There's no need for any kind of special configuration knob or non-blocking option. We already *have* a non-blocking option that works universally for everything.
As I mentioned in the linked thread, waiting for process exit should work just like waiting for bytes to appear on a pipe. Process exit status is just another blob of bytes that a process might receive. A process exit handle ought to be just another information source. The reason the unix process API is so awful is that for whatever reason the original designers treated processes as some kind of special kind of resource instead of fitting them into the otherwise general-purpose unix data-handling API. Let's not repeat that mistake.
Quoted 8 lines
Something like this:
evfd =
eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
// register eventfd to receive death notificationpidfd_wait(pid_to_kill, evfd);
// kill the processpidfd_send_signal(pid_to_kill, ...)
// tend to other things
Now you've lost me. pidfd_wait should return a *new* FD, not wire up an eventfd.
Why? Because the new type FD can report process exit *status* information (via read(2) after readability signal) as well as this binary yes-or-no signal *that* a process exited, and this capability is useful if you want to the pidfd interface to be a good general-purpose process management facility to replace the awful wait() family of functions. You can't get an exit status from an eventfd. Wiring up an eventfd the way you've proposed also complicates wait-causality information, complicating both tracing and any priority inheritance we might want in the future (because all the wakeups gets mixed into the eventfd and you can't unscramble an egg). And for what? What do we gain by using an eventfd? Is the reason that exit.c would be able to use eventfd_signal instead of poking a waitqueue directly? How is that better? With an eventfd, you've increased path length on process exit *and* complicated the API for no reason.
Quoted 5 lines
...
// wait for the process to diepoll_wait(evfd, ...);This simplifies userspace
Not relative to an exit handle it doesn't.
Quoted 2 lines
, allows it to wait for multiple events using
epoll
So does a process exit status handle.
Quoted 3 lines
and I think kernel implementation will be also quite simple because it already implements
eventfd_signal()that takes care of waitqueue handling.
What if there are multiple eventfds registered for the death of a process? In any case, you need some mechanism to find, upon process death, a list of waiters, then wake each of them up. That's either a global search or a search in some list rooted in a task-related structure (either struct task or one of its friends). Using an eventfd here adds nothing, since upon death, you need this list search regardless, and as I mentioned above, eventfd-wiring just makes the API worse.
Quoted 2 lines
If
pidfd_send_signalcould be extended to have an optional eventfd parameter then we would not even have to add a new syscall.
There is nothing wrong with adding a new system call. I don't know why there's this idea circulating that adding system calls is something we should bend over backwards to avoid. It's cheap, and support-wise, kernel interface is kernel interface. Sending a signal has *nothing* to do with wiring up some kind of notification and there's no reason to mingle it with some kind of event registration.
Christian Brauner
maintainerQuoted 103 lines · Daniel Colascione
Quoted 20 lines · Suren Baghdasaryan
Quoted 18 lines · Joel Fernandes
On Fri, Mar 15, 2019 at 07:24:28PM +0100, Christian Brauner wrote:
[..]Quoted 10 lines
Quoted 8 lines
why do we want to add a new syscall (
pidfd_wait) though? Why not just use standard poll/epoll interface on the proc fd like Daniel was suggesting. AFAIK, once the proc file is opened, the struct pid is essentially pinned even though the proc number may be reused. Then the caller can just poll. We can add a waitqueue to struct pid, and wake up any waiters on process death (A quick look showstask_structcan be mapped to its struct pid) and also possibly optimize it using Steve's TIF flag idea. No new syscall is needed then, let me know if I missed something?Huh, I thought that Daniel was against the poll/epoll solution?
Hmm, going through earlier threads, I believe so now. Here was Daniel's reasoning about avoiding a notification about process death through proc directory fd: http://lkml.iu.edu/hypermail/linux/kernel/1811.0/00232.html
Quoted 5 lines
Quoted 1 line
May be a dedicated syscall for this would be cleaner after all.
Ah, I wish I've seen that discussion before... syscall makes sense and it can be non-blocking and we can use select/poll/epoll if we use eventfd.
Thanks for taking a look.
Quoted 2 lines
I would strongly advocate for
non-blocking version or at least to have a non-blocking option.Waiting for FD readiness is *already* blocking or non-blocking according to the caller's desire --- users can pass options they want to poll(2) or whatever. There's no need for any kind of special configuration knob or non-blocking option. We already *have* a non-blocking option that works universally for everything.
As I mentioned in the linked thread, waiting for process exit should work just like waiting for bytes to appear on a pipe. Process exit status is just another blob of bytes that a process might receive. A process exit handle ought to be just another information source. The reason the unix process API is so awful is that for whatever reason the original designers treated processes as some kind of special kind of resource instead of fitting them into the otherwise general-purpose unix data-handling API. Let's not repeat that mistake.
Quoted 8 lines
Something like this:
evfd =
eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
// register eventfd to receive death notificationpidfd_wait(pid_to_kill, evfd);
// kill the processpidfd_send_signal(pid_to_kill, ...)
// tend to other thingsNow you've lost me.
pidfd_waitshould return a *new* FD, not wire up an eventfd.Why? Because the new type FD can report process exit *status* information (via read(2) after readability signal) as well as this binary yes-or-no signal *that* a process exited, and this capability is useful if you want to the pidfd interface to be a good general-purpose process management facility to replace the awful
wait()family of functions. You can't get an exit status from an eventfd. Wiring up an eventfd the way you've proposed also complicates wait-causality information, complicating both tracing and any priority inheritance we might want in the future (because all the wakeups gets mixed into the eventfd and you can't unscramble an egg). And for what? What do we gain by using an eventfd? Is the reason thatexit.cwould be able to useeventfd_signalinstead of poking a waitqueue directly? How is that better? With an eventfd, you've increased path length on process exit *and* complicated the API for no reason.Quoted 5 lines
...
// wait for the process to diepoll_wait(evfd, ...);This simplifies userspace
Not relative to an exit handle it doesn't.
Quoted 2 lines
, allows it to wait for multiple events using
epollSo does a process exit status handle.
Quoted 3 lines
and I think kernel implementation will be also quite simple because it already implements
eventfd_signal()that takes care of waitqueue handling.What if there are multiple eventfds registered for the death of a process? In any case, you need some mechanism to find, upon process death, a list of waiters, then wake each of them up. That's either a global search or a search in some list rooted in a task-related structure (either struct task or one of its friends). Using an eventfd here adds nothing, since upon death, you need this list search regardless, and as I mentioned above, eventfd-wiring just makes the API worse.
Quoted 2 lines
If
pidfd_send_signalcould be extended to have an optional eventfd parameter then we would not even have to add a new syscall.
Quoted 6 lines
There is nothing wrong with adding a new system call. I don't know why there's this idea circulating that adding system calls is something we should bend over backwards to avoid. It's cheap, and support-wise, kernel interface is kernel interface. Sending a signal has *nothing* to do with wiring up some kind of notification and there's no reason to mingle it with some kind of event registration.
I agree with Daniel.
One design goal is to not stuff clearly delinated tasks related to process management into the same syscall. That will just leave us with a confusing api. Sending signals is part of managing a process while it is running. Waiting on a process to end is clearly separate from that. It's important to keep in mind that the goal of the pidfd work is to end up with an api that is of use to all of user space concerned with process management not just a specific project.
Suren Baghdasaryan
reviewerQuoted 58 lines · Christian Brauner
Quoted 56 lines · Daniel Colascione
Quoted 20 lines · Suren Baghdasaryan
Quoted 18 lines · Joel Fernandes
On Fri, Mar 15, 2019 at 07:24:28PM +0100, Christian Brauner wrote:
[..]Quoted 10 lines
Quoted 8 lines
why do we want to add a new syscall (
pidfd_wait) though? Why not just use standard poll/epoll interface on the proc fd like Daniel was suggesting. AFAIK, once the proc file is opened, the struct pid is essentially pinned even though the proc number may be reused. Then the caller can just poll. We can add a waitqueue to struct pid, and wake up any waiters on process death (A quick look showstask_structcan be mapped to its struct pid) and also possibly optimize it using Steve's TIF flag idea. No new syscall is needed then, let me know if I missed something?Huh, I thought that Daniel was against the poll/epoll solution?
Hmm, going through earlier threads, I believe so now. Here was Daniel's reasoning about avoiding a notification about process death through proc directory fd: http://lkml.iu.edu/hypermail/linux/kernel/1811.0/00232.html
Quoted 5 lines
Quoted 1 line
May be a dedicated syscall for this would be cleaner after all.
Ah, I wish I've seen that discussion before... syscall makes sense and it can be non-blocking and we can use select/poll/epoll if we use eventfd.
Thanks for taking a look.
Quoted 2 lines
I would strongly advocate for
non-blocking version or at least to have a non-blocking option.Waiting for FD readiness is *already* blocking or non-blocking according to the caller's desire --- users can pass options they want to poll(2) or whatever. There's no need for any kind of special configuration knob or non-blocking option. We already *have* a non-blocking option that works universally for everything.
As I mentioned in the linked thread, waiting for process exit should work just like waiting for bytes to appear on a pipe. Process exit status is just another blob of bytes that a process might receive. A process exit handle ought to be just another information source. The reason the unix process API is so awful is that for whatever reason the original designers treated processes as some kind of special kind of resource instead of fitting them into the otherwise general-purpose unix data-handling API. Let's not repeat that mistake.
Quoted 8 lines
Something like this:
evfd =
eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
// register eventfd to receive death notificationpidfd_wait(pid_to_kill, evfd);
// kill the processpidfd_send_signal(pid_to_kill, ...)
// tend to other things
Quoted 3 lines
Quoted 3 lines
Now you've lost me.
pidfd_waitshould return a *new* FD, not wire up an eventfd.
Ok, I probably misunderstood your post linked by Joel. I though your original proposal was based on being able to poll a file under /proc/pid and then you changed your mind to have a separate syscall which I assumed would be a blocking one to wait for process exit. Maybe you can describe the new interface you are thinking about in terms of userspace usage like I did above? Several lines of code would explain more than paragraphs of text.
Quoted 52 lines
Quoted 44 lines
Why? Because the new type FD can report process exit *status* information (via read(2) after readability signal) as well as this binary yes-or-no signal *that* a process exited, and this capability is useful if you want to the pidfd interface to be a good general-purpose process management facility to replace the awful
wait()family of functions. You can't get an exit status from an eventfd. Wiring up an eventfd the way you've proposed also complicates wait-causality information, complicating both tracing and any priority inheritance we might want in the future (because all the wakeups gets mixed into the eventfd and you can't unscramble an egg). And for what? What do we gain by using an eventfd? Is the reason thatexit.cwould be able to useeventfd_signalinstead of poking a waitqueue directly? How is that better? With an eventfd, you've increased path length on process exit *and* complicated the API for no reason.Quoted 5 lines
...
// wait for the process to diepoll_wait(evfd, ...);This simplifies userspace
Not relative to an exit handle it doesn't.
Quoted 2 lines
, allows it to wait for multiple events using
epollSo does a process exit status handle.
Quoted 3 lines
and I think kernel implementation will be also quite simple because it already implements
eventfd_signal()that takes care of waitqueue handling.What if there are multiple eventfds registered for the death of a process? In any case, you need some mechanism to find, upon process death, a list of waiters, then wake each of them up. That's either a global search or a search in some list rooted in a task-related structure (either struct task or one of its friends). Using an eventfd here adds nothing, since upon death, you need this list search regardless, and as I mentioned above, eventfd-wiring just makes the API worse.
Quoted 2 lines
If
pidfd_send_signalcould be extended to have an optional eventfd parameter then we would not even have to add a new syscall.Quoted 6 lines
There is nothing wrong with adding a new system call. I don't know why there's this idea circulating that adding system calls is something we should bend over backwards to avoid. It's cheap, and support-wise, kernel interface is kernel interface. Sending a signal has *nothing* to do with wiring up some kind of notification and there's no reason to mingle it with some kind of event registration.
Quoted 8 lines
I agree with Daniel.
One design goal is to not stuff clearly delinated tasks related to process management into the same syscall. That will just leave us with a confusing api. Sending signals is part of managing a process while it is running. Waiting on a process to end is clearly separate from that. It's important to keep in mind that the goal of the pidfd work is to end up with an api that is of use to all of user space concerned with process management not just a specific project.
I'm not bent on adding or not adding a new syscall as long as functionality is there.
Thanks!
Quoted 63 lines · Suren Baghdasaryan
Quoted 58 lines · Christian Brauner
Quoted 56 lines · Daniel Colascione
Quoted 20 lines · Suren Baghdasaryan
Quoted 18 lines · Joel Fernandes
On Fri, Mar 15, 2019 at 07:24:28PM +0100, Christian Brauner wrote:
[..]Quoted 10 lines
Quoted 8 lines
why do we want to add a new syscall (
pidfd_wait) though? Why not just use standard poll/epoll interface on the proc fd like Daniel was suggesting. AFAIK, once the proc file is opened, the struct pid is essentially pinned even though the proc number may be reused. Then the caller can just poll. We can add a waitqueue to struct pid, and wake up any waiters on process death (A quick look showstask_structcan be mapped to its struct pid) and also possibly optimize it using Steve's TIF flag idea. No new syscall is needed then, let me know if I missed something?Huh, I thought that Daniel was against the poll/epoll solution?
Hmm, going through earlier threads, I believe so now. Here was Daniel's reasoning about avoiding a notification about process death through proc directory fd: http://lkml.iu.edu/hypermail/linux/kernel/1811.0/00232.html
Quoted 5 lines
Quoted 1 line
May be a dedicated syscall for this would be cleaner after all.
Ah, I wish I've seen that discussion before... syscall makes sense and it can be non-blocking and we can use select/poll/epoll if we use eventfd.
Thanks for taking a look.
Quoted 2 lines
I would strongly advocate for
non-blocking version or at least to have a non-blocking option.Waiting for FD readiness is *already* blocking or non-blocking according to the caller's desire --- users can pass options they want to poll(2) or whatever. There's no need for any kind of special configuration knob or non-blocking option. We already *have* a non-blocking option that works universally for everything.
As I mentioned in the linked thread, waiting for process exit should work just like waiting for bytes to appear on a pipe. Process exit status is just another blob of bytes that a process might receive. A process exit handle ought to be just another information source. The reason the unix process API is so awful is that for whatever reason the original designers treated processes as some kind of special kind of resource instead of fitting them into the otherwise general-purpose unix data-handling API. Let's not repeat that mistake.
Quoted 8 lines
Something like this:
evfd =
eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
// register eventfd to receive death notificationpidfd_wait(pid_to_kill, evfd);
// kill the processpidfd_send_signal(pid_to_kill, ...)
// tend to other thingsQuoted 3 lines
Quoted 3 lines
Now you've lost me.
pidfd_waitshould return a *new* FD, not wire up an eventfd.
Quoted 7 lines
Ok, I probably misunderstood your post linked by Joel. I though your original proposal was based on being able to poll a file under
/proc/pidand then you changed your mind to have a separate syscall which I assumed would be a blocking one to wait for process exit. Maybe you can describe the new interface you are thinking about in terms of userspace usage like I did above? Several lines of code would explain more than paragraphs of text.
Hey, Thanks Suren for the eventfd idea. I agree with Daniel on this. The idea from Daniel here is to wait for process death and exit events by just referring to a stable fd, independent of whatever is going on in /proc.
What is needed is something like this (in highly pseudo-code form):
pidfd = opendir("/proc/<pid>",..);wait_fd = pidfd_wait(pidfd);
read or poll wait_fd (non-blocking or blocking whichever)
wait_fd will block until the task has either died or reaped. In both these cases, it can return a suitable string such as "dead" or "reaped" although an integer with some predefined meaning is also Ok.
What that guarantees is, even if the task's PID has been reused, or the task has already died or already died + reaped, all of these events cannot race with the code above and the information passed to the user is race-free and stable / guaranteed.
An eventfd seems to not fit well, because AFAICS passing the raw PID to eventfd as in your example would still race since the PID could have been reused by another process by the time the eventfd is created.
Also Andy's idea in [1] seems to use poll flags to communicate various tihngs which is still not as explicit about the PID's status so that's a poor API choice compared to the explicit syscall.
I am planning to work on a prototype patch based on Daniel's idea and post something soon (chatted with Daniel about it and will reference him in the posting as well), during this posting I will also summarize all the previous discussions and come up with some tests as well. I hope to have something soon.
Let me know if I hit all the points correctly and I hope we are all on the same page.
Thanks!
- Joel
[1] http://lkml.iu.edu/hypermail//linux/kernel/1212.0/00808.html
Christian Brauner
maintainerQuoted 95 lines · Joel Fernandes
Quoted 63 lines · Suren Baghdasaryan
Quoted 58 lines · Christian Brauner
Quoted 56 lines · Daniel Colascione
Quoted 20 lines · Suren Baghdasaryan
Quoted 18 lines · Joel Fernandes
On Fri, Mar 15, 2019 at 07:24:28PM +0100, Christian Brauner wrote:
[..]Quoted 10 lines
Quoted 8 lines
why do we want to add a new syscall (
pidfd_wait) though? Why not just use standard poll/epoll interface on the proc fd like Daniel was suggesting. AFAIK, once the proc file is opened, the struct pid is essentially pinned even though the proc number may be reused. Then the caller can just poll. We can add a waitqueue to struct pid, and wake up any waiters on process death (A quick look showstask_structcan be mapped to its struct pid) and also possibly optimize it using Steve's TIF flag idea. No new syscall is needed then, let me know if I missed something?Huh, I thought that Daniel was against the poll/epoll solution?
Hmm, going through earlier threads, I believe so now. Here was Daniel's reasoning about avoiding a notification about process death through proc directory fd: http://lkml.iu.edu/hypermail/linux/kernel/1811.0/00232.html
Quoted 5 lines
Quoted 1 line
May be a dedicated syscall for this would be cleaner after all.
Ah, I wish I've seen that discussion before... syscall makes sense and it can be non-blocking and we can use select/poll/epoll if we use eventfd.
Thanks for taking a look.
Quoted 2 lines
I would strongly advocate for
non-blocking version or at least to have a non-blocking option.Waiting for FD readiness is *already* blocking or non-blocking according to the caller's desire --- users can pass options they want to poll(2) or whatever. There's no need for any kind of special configuration knob or non-blocking option. We already *have* a non-blocking option that works universally for everything.
As I mentioned in the linked thread, waiting for process exit should work just like waiting for bytes to appear on a pipe. Process exit status is just another blob of bytes that a process might receive. A process exit handle ought to be just another information source. The reason the unix process API is so awful is that for whatever reason the original designers treated processes as some kind of special kind of resource instead of fitting them into the otherwise general-purpose unix data-handling API. Let's not repeat that mistake.
Quoted 8 lines
Something like this:
evfd =
eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
// register eventfd to receive death notificationpidfd_wait(pid_to_kill, evfd);
// kill the processpidfd_send_signal(pid_to_kill, ...)
// tend to other thingsQuoted 3 lines
Quoted 3 lines
Now you've lost me.
pidfd_waitshould return a *new* FD, not wire up an eventfd.Quoted 7 lines
Ok, I probably misunderstood your post linked by Joel. I though your original proposal was based on being able to poll a file under
/proc/pidand then you changed your mind to have a separate syscall which I assumed would be a blocking one to wait for process exit. Maybe you can describe the new interface you are thinking about in terms of userspace usage like I did above? Several lines of code would explain more than paragraphs of text.Hey, Thanks Suren for the eventfd idea. I agree with Daniel on this. The idea from Daniel here is to wait for process death and exit events by just referring to a stable fd, independent of whatever is going on in /proc.
What is needed is something like this (in highly pseudo-code form):
pidfd =
opendir("/proc/<pid>",..);wait_fd=pidfd_wait(pidfd);
read or pollwait_fd(non-blocking or blocking whichever)
wait_fdwill block until the task has either died or reaped. In both these cases, it can return a suitable string such as "dead" or "reaped" although an integer with some predefined meaning is also Ok.What that guarantees is, even if the task's PID has been reused, or the task has already died or already died + reaped, all of these events cannot race with the code above and the information passed to the user is race-free and stable / guaranteed.
An eventfd seems to not fit well, because AFAICS passing the raw PID to eventfd as in your example would still race since the PID could have been reused by another process by the time the eventfd is created.
Quoted 8 lines
Also Andy's idea in [1] seems to use poll flags to communicate various tihngs which is still not as explicit about the PID's status so that's a poor API choice compared to the explicit syscall.
I am planning to work on a prototype patch based on Daniel's idea and post something soon (chatted with Daniel about it and will reference him in the posting as well), during this posting I will also summarize all the previous discussions and come up with some tests as well. I hope to have something soon.
Having pidfd_wait() return another fd will make the syscall harder to swallow for a lot of people I reckon.
What exactly prevents us from making the pidfd itself readable/pollable for the exit staus? They are "special" fds anyway. I would really like to avoid polluting the api with multiple different types of fds if possible.
ret = pidfd_wait(pidfd);
read or poll pidfd
(Note that I'm traveling so my responses might be delayed quite a bit.) (Ccing a few people that might have an opinion here.)
Christian
Quoted 80 lines · Christian Brauner
Quoted 78 lines · Joel Fernandes
Quoted 63 lines · Suren Baghdasaryan
Quoted 58 lines · Christian Brauner
Quoted 56 lines · Daniel Colascione
Quoted 20 lines · Suren Baghdasaryan
Quoted 18 lines · Joel Fernandes
On Fri, Mar 15, 2019 at 07:24:28PM +0100, Christian Brauner wrote:
[..]Quoted 10 lines
Quoted 8 lines
why do we want to add a new syscall (
pidfd_wait) though? Why not just use standard poll/epoll interface on the proc fd like Daniel was suggesting. AFAIK, once the proc file is opened, the struct pid is essentially pinned even though the proc number may be reused. Then the caller can just poll. We can add a waitqueue to struct pid, and wake up any waiters on process death (A quick look showstask_structcan be mapped to its struct pid) and also possibly optimize it using Steve's TIF flag idea. No new syscall is needed then, let me know if I missed something?Huh, I thought that Daniel was against the poll/epoll solution?
Hmm, going through earlier threads, I believe so now. Here was Daniel's reasoning about avoiding a notification about process death through proc directory fd: http://lkml.iu.edu/hypermail/linux/kernel/1811.0/00232.html
Quoted 5 lines
Quoted 1 line
May be a dedicated syscall for this would be cleaner after all.
Ah, I wish I've seen that discussion before... syscall makes sense and it can be non-blocking and we can use select/poll/epoll if we use eventfd.
Thanks for taking a look.
Quoted 2 lines
I would strongly advocate for
non-blocking version or at least to have a non-blocking option.Waiting for FD readiness is *already* blocking or non-blocking according to the caller's desire --- users can pass options they want to poll(2) or whatever. There's no need for any kind of special configuration knob or non-blocking option. We already *have* a non-blocking option that works universally for everything.
As I mentioned in the linked thread, waiting for process exit should work just like waiting for bytes to appear on a pipe. Process exit status is just another blob of bytes that a process might receive. A process exit handle ought to be just another information source. The reason the unix process API is so awful is that for whatever reason the original designers treated processes as some kind of special kind of resource instead of fitting them into the otherwise general-purpose unix data-handling API. Let's not repeat that mistake.
Quoted 8 lines
Something like this:
evfd =
eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
// register eventfd to receive death notificationpidfd_wait(pid_to_kill, evfd);
// kill the processpidfd_send_signal(pid_to_kill, ...)
// tend to other thingsQuoted 3 lines
Quoted 3 lines
Now you've lost me.
pidfd_waitshould return a *new* FD, not wire up an eventfd.Quoted 7 lines
Ok, I probably misunderstood your post linked by Joel. I though your original proposal was based on being able to poll a file under
/proc/pidand then you changed your mind to have a separate syscall which I assumed would be a blocking one to wait for process exit. Maybe you can describe the new interface you are thinking about in terms of userspace usage like I did above? Several lines of code would explain more than paragraphs of text.Hey, Thanks Suren for the eventfd idea. I agree with Daniel on this. The idea from Daniel here is to wait for process death and exit events by just referring to a stable fd, independent of whatever is going on in /proc.
What is needed is something like this (in highly pseudo-code form):
Quoted 7 lines
Quoted 7 lines
pidfd =
opendir("/proc/<pid>",..);wait_fd=pidfd_wait(pidfd);
read or pollwait_fd(non-blocking or blocking whichever)
wait_fdwill block until the task has either died or reaped. In both these cases, it can return a suitable string such as "dead" or "reaped" although an integer with some predefined meaning is also Ok.
I want to return a siginfo_t: we already use this structure in other contexts to report exit status.
Quoted 9 lines
Quoted 9 lines
What that guarantees is, even if the task's PID has been reused, or the task has already died or already died + reaped, all of these events cannot race with the code above and the information passed to the user is race-free and stable / guaranteed.
An eventfd seems to not fit well, because AFAICS passing the raw PID to eventfd as in your example would still race since the PID could have been reused by another process by the time the eventfd is created.
Quoted 8 lines
Quoted 8 lines
Also Andy's idea in [1] seems to use poll flags to communicate various tihngs which is still not as explicit about the PID's status so that's a poor API choice compared to the explicit syscall.
I am planning to work on a prototype patch based on Daniel's idea and post something soon (chatted with Daniel about it and will reference him in the posting as well), during this posting I will also summarize all the previous discussions and come up with some tests as well. I hope to have something soon.
Thanks.
Quoted 5 lines
Having
pidfd_wait()return another fd will make the syscall harder to swallow for a lot of people I reckon.
What exactly prevents us from making the pidfd itself readable/pollable for the exit staus? They are "special" fds anyway. I would really like to avoid polluting the api with multiple different types of fds if possible.
If pidfds had been their own file type, I'd agree with you. But pidfds are directories, which means that we're beholden to make them behave like directories normally do. I'd rather introduce another FD than heavily overload the semantics of a directory FD in one particular context. In no other circumstances are directory FDs also weird IO-data sources. Our providing a facility to get a new FD to which we *can* give pipe-like behavior does no harm and *usage* cleaner and easier to reason about.
Christian Brauner
maintainerQuoted 118 lines · Daniel Colascione
Quoted 80 lines · Christian Brauner
Quoted 78 lines · Joel Fernandes
Quoted 63 lines · Suren Baghdasaryan
Quoted 58 lines · Christian Brauner
Quoted 56 lines · Daniel Colascione
Quoted 20 lines · Suren Baghdasaryan
Quoted 18 lines · Joel Fernandes
On Fri, Mar 15, 2019 at 07:24:28PM +0100, Christian Brauner wrote:
[..]Quoted 10 lines
Quoted 8 lines
why do we want to add a new syscall (
pidfd_wait) though? Why not just use standard poll/epoll interface on the proc fd like Daniel was suggesting. AFAIK, once the proc file is opened, the struct pid is essentially pinned even though the proc number may be reused. Then the caller can just poll. We can add a waitqueue to struct pid, and wake up any waiters on process death (A quick look showstask_structcan be mapped to its struct pid) and also possibly optimize it using Steve's TIF flag idea. No new syscall is needed then, let me know if I missed something?Huh, I thought that Daniel was against the poll/epoll solution?
Hmm, going through earlier threads, I believe so now. Here was Daniel's reasoning about avoiding a notification about process death through proc directory fd: http://lkml.iu.edu/hypermail/linux/kernel/1811.0/00232.html
Quoted 5 lines
Quoted 1 line
May be a dedicated syscall for this would be cleaner after all.
Ah, I wish I've seen that discussion before... syscall makes sense and it can be non-blocking and we can use select/poll/epoll if we use eventfd.
Thanks for taking a look.
Quoted 2 lines
I would strongly advocate for
non-blocking version or at least to have a non-blocking option.Waiting for FD readiness is *already* blocking or non-blocking according to the caller's desire --- users can pass options they want to poll(2) or whatever. There's no need for any kind of special configuration knob or non-blocking option. We already *have* a non-blocking option that works universally for everything.
As I mentioned in the linked thread, waiting for process exit should work just like waiting for bytes to appear on a pipe. Process exit status is just another blob of bytes that a process might receive. A process exit handle ought to be just another information source. The reason the unix process API is so awful is that for whatever reason the original designers treated processes as some kind of special kind of resource instead of fitting them into the otherwise general-purpose unix data-handling API. Let's not repeat that mistake.
Quoted 8 lines
Something like this:
evfd =
eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
// register eventfd to receive death notificationpidfd_wait(pid_to_kill, evfd);
// kill the processpidfd_send_signal(pid_to_kill, ...)
// tend to other thingsQuoted 3 lines
Quoted 3 lines
Now you've lost me.
pidfd_waitshould return a *new* FD, not wire up an eventfd.Quoted 7 lines
Ok, I probably misunderstood your post linked by Joel. I though your original proposal was based on being able to poll a file under
/proc/pidand then you changed your mind to have a separate syscall which I assumed would be a blocking one to wait for process exit. Maybe you can describe the new interface you are thinking about in terms of userspace usage like I did above? Several lines of code would explain more than paragraphs of text.Hey, Thanks Suren for the eventfd idea. I agree with Daniel on this. The idea from Daniel here is to wait for process death and exit events by just referring to a stable fd, independent of whatever is going on in /proc.
What is needed is something like this (in highly pseudo-code form):
Quoted 7 lines
Quoted 7 lines
pidfd =
opendir("/proc/<pid>",..);wait_fd=pidfd_wait(pidfd);
read or pollwait_fd(non-blocking or blocking whichever)
wait_fdwill block until the task has either died or reaped. In both these cases, it can return a suitable string such as "dead" or "reaped" although an integer with some predefined meaning is also Ok.I want to return a
siginfo_t: we already use this structure in other contexts to report exit status.Quoted 9 lines
Quoted 9 lines
What that guarantees is, even if the task's PID has been reused, or the task has already died or already died + reaped, all of these events cannot race with the code above and the information passed to the user is race-free and stable / guaranteed.
An eventfd seems to not fit well, because AFAICS passing the raw PID to eventfd as in your example would still race since the PID could have been reused by another process by the time the eventfd is created.
Quoted 8 lines
Quoted 8 lines
Also Andy's idea in [1] seems to use poll flags to communicate various tihngs which is still not as explicit about the PID's status so that's a poor API choice compared to the explicit syscall.
I am planning to work on a prototype patch based on Daniel's idea and post something soon (chatted with Daniel about it and will reference him in the posting as well), during this posting I will also summarize all the previous discussions and come up with some tests as well. I hope to have something soon.
Thanks.
Quoted 5 lines
Having
pidfd_wait()return another fd will make the syscall harder to swallow for a lot of people I reckon.
What exactly prevents us from making the pidfd itself readable/pollable for the exit staus? They are "special" fds anyway. I would really like to avoid polluting the api with multiple different types of fds if possible.
Quoted 8 lines
If pidfds had been their own file type, I'd agree with you. But pidfds are directories, which means that we're beholden to make them behave like directories normally do. I'd rather introduce another FD than heavily overload the semantics of a directory FD in one particular context. In no other circumstances are directory FDs also weird IO-data sources. Our providing a facility to get a new FD to which we *can* give pipe-like behavior does no harm and *usage* cleaner and easier to reason about.
I have two things I'm currently working on:
- hijacking translate_pid()
- pidfd_clone() essentially
My first goal is to talk to Eric about taking the translate_pid() syscall that has been sitting in his tree and expanding it. translate_pid() currently allows you to either get an fd for the pid namespace a pid resides in or the pid number of a given process in another pid namespace relative to a passed in pid namespace fd. I would like to make it possible for this syscall to also give us back pidfds. One question I'm currently struggling with is exactly what you said
above: what type of file descriptor these are going to give back to us. It seems that a regular file instead of directory would make the most sense and would lead to a nicer API and I'm very much leaning towards that.
Christian
Quoted 86 lines · Christian Brauner
Quoted 85 lines · Daniel Colascione
Quoted 84 lines · Christian Brauner
Quoted 82 lines · Joel Fernandes
Quoted 63 lines · Suren Baghdasaryan
Quoted 58 lines · Christian Brauner
Quoted 56 lines · Daniel Colascione
Quoted 20 lines · Suren Baghdasaryan
Quoted 18 lines · Joel Fernandes
On Fri, Mar 15, 2019 at 07:24:28PM +0100, Christian Brauner wrote:
[..]Quoted 10 lines
Quoted 8 lines
why do we want to add a new syscall (
pidfd_wait) though? Why not just use standard poll/epoll interface on the proc fd like Daniel was suggesting. AFAIK, once the proc file is opened, the struct pid is essentially pinned even though the proc number may be reused. Then the caller can just poll. We can add a waitqueue to struct pid, and wake up any waiters on process death (A quick look showstask_structcan be mapped to its struct pid) and also possibly optimize it using Steve's TIF flag idea. No new syscall is needed then, let me know if I missed something?Huh, I thought that Daniel was against the poll/epoll solution?
Hmm, going through earlier threads, I believe so now. Here was Daniel's reasoning about avoiding a notification about process death through proc directory fd: http://lkml.iu.edu/hypermail/linux/kernel/1811.0/00232.html
Quoted 5 lines
Quoted 1 line
May be a dedicated syscall for this would be cleaner after all.
Ah, I wish I've seen that discussion before... syscall makes sense and it can be non-blocking and we can use select/poll/epoll if we use eventfd.
Thanks for taking a look.
Quoted 2 lines
I would strongly advocate for
non-blocking version or at least to have a non-blocking option.Waiting for FD readiness is *already* blocking or non-blocking according to the caller's desire --- users can pass options they want to poll(2) or whatever. There's no need for any kind of special configuration knob or non-blocking option. We already *have* a non-blocking option that works universally for everything.
As I mentioned in the linked thread, waiting for process exit should work just like waiting for bytes to appear on a pipe. Process exit status is just another blob of bytes that a process might receive. A process exit handle ought to be just another information source. The reason the unix process API is so awful is that for whatever reason the original designers treated processes as some kind of special kind of resource instead of fitting them into the otherwise general-purpose unix data-handling API. Let's not repeat that mistake.
Quoted 8 lines
Something like this:
evfd =
eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
// register eventfd to receive death notificationpidfd_wait(pid_to_kill, evfd);
// kill the processpidfd_send_signal(pid_to_kill, ...)
// tend to other thingsQuoted 3 lines
Quoted 3 lines
Now you've lost me.
pidfd_waitshould return a *new* FD, not wire up an eventfd.Quoted 7 lines
Ok, I probably misunderstood your post linked by Joel. I though your original proposal was based on being able to poll a file under
/proc/pidand then you changed your mind to have a separate syscall which I assumed would be a blocking one to wait for process exit. Maybe you can describe the new interface you are thinking about in terms of userspace usage like I did above? Several lines of code would explain more than paragraphs of text.Hey, Thanks Suren for the eventfd idea. I agree with Daniel on this. The idea from Daniel here is to wait for process death and exit events by just referring to a stable fd, independent of whatever is going on in /proc.
What is needed is something like this (in highly pseudo-code form):
pidfd =
opendir("/proc/<pid>",..);wait_fd=pidfd_wait(pidfd);
read or pollwait_fd(non-blocking or blocking whichever)
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
wait_fdwill block until the task has either died or reaped. In both these cases, it can return a suitable string such as "dead" or "reaped" although an integer with some predefined meaning is also Ok.I want to return a
siginfo_t: we already use this structure in other contexts to report exit status.
Fine with me. I did a prototype (code is below) as a string but I can change that to siginfo_t in the future.
Quoted 19 lines
Quoted 6 lines
Quoted 5 lines
Having
pidfd_wait()return another fd will make the syscall harder to swallow for a lot of people I reckon.
What exactly prevents us from making the pidfd itself readable/pollable for the exit staus? They are "special" fds anyway. I would really like to avoid polluting the api with multiple different types of fds if possible.Quoted 8 lines
If pidfds had been their own file type, I'd agree with you. But pidfds are directories, which means that we're beholden to make them behave like directories normally do. I'd rather introduce another FD than heavily overload the semantics of a directory FD in one particular context. In no other circumstances are directory FDs also weird IO-data sources. Our providing a facility to get a new FD to which we *can* give pipe-like behavior does no harm and *usage* cleaner and easier to reason about.
I have two things I'm currently working on:
- hijackingtranslate_pid()
-pidfd_clone()essentially
Quoted 5 lines
My first goal is to talk to Eric about taking the
translate_pid()syscall that has been sitting in his tree and expanding it.translate_pid()currently allows you to either get an fd for the pid namespace a pid resides in or the pid number of a given process in another pid namespace relative to a passed in pid namespace fd.
That's good to know. More comments below:
Quoted 7 lines
I would
like to make it possible for this syscall to also give us back pidfds. One question I'm currently struggling with is exactly what you said
above: what type of file descriptor these are going to give back to us. It seems that a regular file instead of directory would make the most sense and would lead to a nicer API and I'm very much leaning towards that.
How about something like the following? We can plumb the new file as a pseudo file that is invisible and linked to the fd. This is extremely rough (does not do error handling, synchronizatoin etc) but just wanted to share the idea of what the "frontend" could look like. It is also missing all the actual pid status messages. It just takes care of the creating new fd from the pidfd part and providing file read ops returning the "status" string. It is also written in signal.c and should likely go into proc fs files under fs. Appreciate any suggestions (a test program did prove it works).
Also, I was able to translate a pidfd to a pid_namespace by referring to some existing code but perhaps you may be able to suggest something better for such translation..
---8<-----------------------
From: Joel Fernandes <[email protected]>
Subject: [PATCH] Partial skeleton prototype of pidfd_wait frontend
--- arch/x86/entry/syscalls/syscall_32.tbl | 1 + arch/x86/entry/syscalls/syscall_64.tbl | 1 + include/linux/syscalls.h | 1 + include/uapi/asm-generic/unistd.h | 4 +- kernel/signal.c | 62 ++++++++++++++++++++++++++ kernel/sys_ni.c | 3 ++ 6 files changed, 71 insertions(+), 1 deletion(-)
arch/x86/entry/syscalls/syscall_32.tbl +1 -0
@@ -433,3 +433,4 @@ 425 i386 io_uring_setup sys_io_uring_setup __ia32_sys_io_uring_setup 426 i386 io_uring_enter sys_io_uring_enter __ia32_sys_io_uring_enter 427 i386 io_uring_register sys_io_uring_register __ia32_sys_io_uring_register +428 i386 pidfd_wait sys_pidfd_wait __ia32_sys_pidfd_wait
arch/x86/entry/syscalls/syscall_64.tbl +1 -0
@@ -349,6 +349,7 @@ 425 common io_uring_setup __x64_sys_io_uring_setup 426 common io_uring_enter __x64_sys_io_uring_enter 427 common io_uring_register __x64_sys_io_uring_register +428 common pidfd_wait __x64_sys_pidfd_wait # # x32-specific system call numbers start at 512 to avoid cache impact
include/linux/syscalls.h +1 -0
@@ -988,6 +988,7 @@ asmlinkage long sys_rseq(struct rseq __user *rseq, uint32_t rseq_len,asmlinkage long sys_pidfd_send_signal (int pidfd,int sig,siginfo_t __user *info,unsigned int flags); +asmlinkage long sys_pidfd_wait (int pidfd);/* * Architecture-specific system calls
include/uapi/asm-generic/unistd.h +3 -1
@@ -832,9 +832,11 @@ __SYSCALL(__NR_io_uring_setup, sys_io_uring_setup) __SYSCALL(__NR_io_uring_enter, sys_io_uring_enter)#define __NR_io_uring_register 427 __SYSCALL(__NR_io_uring_register, sys_io_uring_register) +#define __NR_pidfd_wait 428 +__SYSCALL(__NR_pidfd_wait, sys_pidfd_wait)#undef __NR_syscalls -#define __NR_syscalls 428 +#define __NR_syscalls 429/* * 32 bit systems traditionally used different
kernel/signal.c +62 -0
68 lines not on this page · raw message
kernel/sys_ni.c +3 -0
@@ -450,3 +450,6 @@ COND_SYSCALL(setuid16);/* restartable sequence */ COND_SYSCALL (rseq); + +/* pidfd */ +COND_SYSCALL (pidfd_wait);
Signature · 1 line
2.21.0.225.g810b269d1ac-goog
Christian Brauner
maintainerQuoted 117 lines · Joel Fernandes
Quoted 86 lines · Christian Brauner
Quoted 85 lines · Daniel Colascione
Quoted 84 lines · Christian Brauner
Quoted 82 lines · Joel Fernandes
Quoted 63 lines · Suren Baghdasaryan
Quoted 58 lines · Christian Brauner
Quoted 56 lines · Daniel Colascione
Quoted 20 lines · Suren Baghdasaryan
Quoted 18 lines · Joel Fernandes
On Fri, Mar 15, 2019 at 07:24:28PM +0100, Christian Brauner wrote:
[..]Quoted 10 lines
Quoted 8 lines
why do we want to add a new syscall (
pidfd_wait) though? Why not just use standard poll/epoll interface on the proc fd like Daniel was suggesting. AFAIK, once the proc file is opened, the struct pid is essentially pinned even though the proc number may be reused. Then the caller can just poll. We can add a waitqueue to struct pid, and wake up any waiters on process death (A quick look showstask_structcan be mapped to its struct pid) and also possibly optimize it using Steve's TIF flag idea. No new syscall is needed then, let me know if I missed something?Huh, I thought that Daniel was against the poll/epoll solution?
Hmm, going through earlier threads, I believe so now. Here was Daniel's reasoning about avoiding a notification about process death through proc directory fd: http://lkml.iu.edu/hypermail/linux/kernel/1811.0/00232.html
Quoted 5 lines
Quoted 1 line
May be a dedicated syscall for this would be cleaner after all.
Ah, I wish I've seen that discussion before... syscall makes sense and it can be non-blocking and we can use select/poll/epoll if we use eventfd.
Thanks for taking a look.
Quoted 2 lines
I would strongly advocate for
non-blocking version or at least to have a non-blocking option.Waiting for FD readiness is *already* blocking or non-blocking according to the caller's desire --- users can pass options they want to poll(2) or whatever. There's no need for any kind of special configuration knob or non-blocking option. We already *have* a non-blocking option that works universally for everything.
As I mentioned in the linked thread, waiting for process exit should work just like waiting for bytes to appear on a pipe. Process exit status is just another blob of bytes that a process might receive. A process exit handle ought to be just another information source. The reason the unix process API is so awful is that for whatever reason the original designers treated processes as some kind of special kind of resource instead of fitting them into the otherwise general-purpose unix data-handling API. Let's not repeat that mistake.
Quoted 8 lines
Something like this:
evfd =
eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
// register eventfd to receive death notificationpidfd_wait(pid_to_kill, evfd);
// kill the processpidfd_send_signal(pid_to_kill, ...)
// tend to other thingsQuoted 3 lines
Quoted 3 lines
Now you've lost me.
pidfd_waitshould return a *new* FD, not wire up an eventfd.Quoted 7 lines
Ok, I probably misunderstood your post linked by Joel. I though your original proposal was based on being able to poll a file under
/proc/pidand then you changed your mind to have a separate syscall which I assumed would be a blocking one to wait for process exit. Maybe you can describe the new interface you are thinking about in terms of userspace usage like I did above? Several lines of code would explain more than paragraphs of text.Hey, Thanks Suren for the eventfd idea. I agree with Daniel on this. The idea from Daniel here is to wait for process death and exit events by just referring to a stable fd, independent of whatever is going on in /proc.
What is needed is something like this (in highly pseudo-code form):
pidfd =
opendir("/proc/<pid>",..);wait_fd=pidfd_wait(pidfd);
read or pollwait_fd(non-blocking or blocking whichever)Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
wait_fdwill block until the task has either died or reaped. In both these cases, it can return a suitable string such as "dead" or "reaped" although an integer with some predefined meaning is also Ok.I want to return a
siginfo_t: we already use this structure in other contexts to report exit status.Fine with me. I did a prototype (code is below) as a string but I can change that to
siginfo_tin the future.Quoted 19 lines
Quoted 6 lines
Quoted 5 lines
Having
pidfd_wait()return another fd will make the syscall harder to swallow for a lot of people I reckon.
What exactly prevents us from making the pidfd itself readable/pollable for the exit staus? They are "special" fds anyway. I would really like to avoid polluting the api with multiple different types of fds if possible.Quoted 8 lines
If pidfds had been their own file type, I'd agree with you. But pidfds are directories, which means that we're beholden to make them behave like directories normally do. I'd rather introduce another FD than heavily overload the semantics of a directory FD in one particular context. In no other circumstances are directory FDs also weird IO-data sources. Our providing a facility to get a new FD to which we *can* give pipe-like behavior does no harm and *usage* cleaner and easier to reason about.
I have two things I'm currently working on:
- hijackingtranslate_pid()
-pidfd_clone()essentially
Quoted 7 lines
Quoted 5 lines
My first goal is to talk to Eric about taking the
translate_pid()syscall that has been sitting in his tree and expanding it.translate_pid()currently allows you to either get an fd for the pid namespace a pid resides in or the pid number of a given process in another pid namespace relative to a passed in pid namespace fd.That's good to know. More comments below:
Sorry for the delay I'm still traveling. I'll be back on a fully functional schedule starting Monday.
Quoted 18 lines
Quoted 8 lines
I would
like to make it possible for this syscall to also give us back pidfds. One question I'm currently struggling with is exactly what you said
above: what type of file descriptor these are going to give back to us. It seems that a regular file instead of directory would make the most sense and would lead to a nicer API and I'm very much leaning towards that.How about something like the following? We can plumb the new file as a pseudo file that is invisible and linked to the fd. This is extremely rough (does not do error handling, synchronizatoin etc) but just wanted to share the idea of what the "frontend" could look like. It is also missing all the actual pid status messages. It just takes care of the creating new fd from the pidfd part and providing file read ops returning the "status" string. It is also written in
signal.cand should likely go into proc fs files under fs. Appreciate any suggestions (a test program did prove it works).
Quoted 3 lines
Also, I was able to translate a pidfd to a
pid_namespaceby referring to some existing code but perhaps you may be able to suggest something better for such translation..
Yeah, there's better ways but I think there's another issue. See below.
Quoted 115 lines
---8<-----------------------
From: Joel Fernandes <[email protected]>
Subject: [PATCH] Partial skeleton prototype ofpidfd_waitfrontendSigned-off-by Joel Fernandes <[email protected]>--- arch/x86/entry/syscalls/syscall_32.tbl | 1 + arch/x86/entry/syscalls/syscall_64.tbl | 1 + include/linux/syscalls.h | 1 + include/uapi/asm-generic/unistd.h | 4 +- kernel/signal.c | 62 ++++++++++++++++++++++++++ kernel/sys_ni.c | 3 ++ 6 files changed, 71 insertions(+), 1 deletion(-)arch/x86/entry/syscalls/syscall_32.tbl +1
-0@@ -433,3 +433,4 @@ 425 i386 io_uring_setup sys_io_uring_setup __ia32_sys_io_uring_setup 426 i386 io_uring_enter sys_io_uring_enter __ia32_sys_io_uring_enter 427 i386 io_uring_register sys_io_uring_register __ia32_sys_io_uring_register +428 i386 pidfd_wait sys_pidfd_wait __ia32_sys_pidfd_waitarch/x86/entry/syscalls/syscall_64.tbl +1
-0@@ -349,6 +349,7 @@ 425 common io_uring_setup __x64_sys_io_uring_setup 426 common io_uring_enter __x64_sys_io_uring_enter 427 common io_uring_register __x64_sys_io_uring_register +428 common pidfd_wait __x64_sys_pidfd_wait # # x32-specific system call numbers start at 512 to avoid cache impactinclude/linux/syscalls.h +1
-0@@ -988,6 +988,7 @@ asmlinkage long sys_rseq(struct rseq __user *rseq, uint32_t rseq_len,asmlinkage long sys_pidfd_send_signal (int pidfd,int sig,siginfo_t __user *info,unsigned int flags); +asmlinkage long sys_pidfd_wait (int pidfd);/* * Architecture-specific system calls include/uapi/asm-generic/unistd.h +3
-1@@ -832,9 +832,11 @@ __SYSCALL(__NR_io_uring_setup, sys_io_uring_setup) __SYSCALL(__NR_io_uring_enter, sys_io_uring_enter)#define __NR_io_uring_register 427 __SYSCALL(__NR_io_uring_register, sys_io_uring_register) +#define __NR_pidfd_wait 428 +__SYSCALL(__NR_pidfd_wait, sys_pidfd_wait)#undef __NR_syscalls -#define __NR_syscalls 428 +#define __NR_syscalls 429/* * 32 bit systems traditionally used different kernel/signal.c +41
-0@@ -3550,6 +3550,68 @@ static int copy_siginfo_from_user_any(kernel_siginfo_t *kinfo, siginfo_t *info)return copy_siginfo_from_user (kinfo, info); } +static ssize_t pidfd_wait_read_iter (struct kiocb *iocb,struct iov_iter *to) +{ +/* +* This is just a test string, it will contain the actual +* status of the pidfd in the future. +*/ +char buf[] ="status" ; + +return copy_to_iter (buf,strlen (buf)+1 , to); +} + +static const struct file_operations pidfd_wait_file_ops = { + .read_iter = pidfd_wait_read_iter, +}; + +static struct inode *pidfd_wait_get_inode (struct super_block *sb) +{ +struct inode *inode =new_inode (sb); + + inode->i_ino =get_next_ino (); +inode_init_owner (inode,NULL ,S_IFREG ); + + inode->i_op = &simple_dir_inode_operations; + inode->i_fop = &pidfd_wait_file_ops; + +return inode; +} + +SYSCALL_DEFINE1(pidfd_wait, int, pidfd) +{ +struct fd f; +struct inode *inode; +struct file *file; +int new_fd; +struct pid_namespace *pid_ns; +struct super_block *sb; +struct vfsmount *mnt; + + f =fdget_raw (pidfd); +if (!f.file) +return -EBADF ;
kernel/signal.c static int copy_siginfo_from_user_any(kernel_siginfo_t *kinfo, siginfo_t *info) · line 3603
Quoted 10 lines
kernel/signal.c +10
-0+ + sb =file_inode (f.file)->i_sb; + pid_ns = sb->s_fs_info; + + inode =pidfd_wait_get_inode (sb); + + mnt = pid_ns->proc_mnt; + + file =alloc_file_pseudo (inode, mnt,"pidfd_wait" ,O_RDONLY , + &pidfd_wait_file_ops);
So I dislike the idea of allocating new inodes from the procfs super block. I would like to avoid pinning the whole pidfd concept exclusively to proc. The idea is that the pidfd API will be useable through procfs via open("/proc/<pid>") because that is what users expect and really wanted to have for a long time. So it makes sense to have this working. But it should really be useable without it. That's why translate_pid() and pidfd_clone() are on the table. What I'm saying is, once the pidfd api is "complete" you should be able to set CONFIG_PROCFS=N - even though that's crazy - and still be able to use pidfds. This is also a point akpm asked about when I did the pidfd_send_signal work.
So instead of going throught proc we should probably do what David has been doing in the mount API and come to rely on anone_inode. So something like:
fd = anon_inode_getfd("pidfd", &pidfd_fops, file_priv_data, flags);
and stash information such as pid namespace etc. in a pidfd struct or something that we then can stash file->private_data of the new file. This also lets us avoid all this open coding done here. Another advantage is that anon_inodes is its own kernel-internal filesystem.
Christian
Quoted 28 lines
kernel/signal.c +11
-0+ + file->f_mode |=FMODE_PREAD ; + + new_fd =get_unused_fd_flags (0 ); +fd_install (new_fd, file); + +fdput (f); + +return new_fd; +} +/** * sys_pidfd_send_signal - send a signal to a process through a task file * descriptor kernel/sys_ni.c +3
-0@@ -450,3 +450,6 @@ COND_SYSCALL(setuid16);/* restartable sequence */ COND_SYSCALL (rseq); + +/* pidfd */ +COND_SYSCALL (pidfd_wait);Signature · 1 line
2.21.0.225.g810b269d1ac-goog
On Tue, Mar 19, 2019 at 11:14:17PM +0100, Christian Brauner wrote:
[snip]
Quoted 128 lines
Quoted 115 lines
---8<-----------------------
From: Joel Fernandes <[email protected]>
Subject: [PATCH] Partial skeleton prototype ofpidfd_waitfrontendSigned-off-by Joel Fernandes <[email protected]>--- arch/x86/entry/syscalls/syscall_32.tbl | 1 + arch/x86/entry/syscalls/syscall_64.tbl | 1 + include/linux/syscalls.h | 1 + include/uapi/asm-generic/unistd.h | 4 +- kernel/signal.c | 62 ++++++++++++++++++++++++++ kernel/sys_ni.c | 3 ++ 6 files changed, 71 insertions(+), 1 deletion(-)arch/x86/entry/syscalls/syscall_32.tbl +1
-0@@ -433,3 +433,4 @@ 425 i386 io_uring_setup sys_io_uring_setup __ia32_sys_io_uring_setup 426 i386 io_uring_enter sys_io_uring_enter __ia32_sys_io_uring_enter 427 i386 io_uring_register sys_io_uring_register __ia32_sys_io_uring_register +428 i386 pidfd_wait sys_pidfd_wait __ia32_sys_pidfd_waitarch/x86/entry/syscalls/syscall_64.tbl +1
-0@@ -349,6 +349,7 @@ 425 common io_uring_setup __x64_sys_io_uring_setup 426 common io_uring_enter __x64_sys_io_uring_enter 427 common io_uring_register __x64_sys_io_uring_register +428 common pidfd_wait __x64_sys_pidfd_wait # # x32-specific system call numbers start at 512 to avoid cache impactinclude/linux/syscalls.h +1
-0@@ -988,6 +988,7 @@ asmlinkage long sys_rseq(struct rseq __user *rseq, uint32_t rseq_len,asmlinkage long sys_pidfd_send_signal (int pidfd,int sig,siginfo_t __user *info,unsigned int flags); +asmlinkage long sys_pidfd_wait (int pidfd);/* * Architecture-specific system calls include/uapi/asm-generic/unistd.h +3
-1@@ -832,9 +832,11 @@ __SYSCALL(__NR_io_uring_setup, sys_io_uring_setup) __SYSCALL(__NR_io_uring_enter, sys_io_uring_enter)#define __NR_io_uring_register 427 __SYSCALL(__NR_io_uring_register, sys_io_uring_register) +#define __NR_pidfd_wait 428 +__SYSCALL(__NR_pidfd_wait, sys_pidfd_wait)#undef __NR_syscalls -#define __NR_syscalls 428 +#define __NR_syscalls 429/* * 32 bit systems traditionally used different kernel/signal.c +41
-0@@ -3550,6 +3550,68 @@ static int copy_siginfo_from_user_any(kernel_siginfo_t *kinfo, siginfo_t *info)return copy_siginfo_from_user (kinfo, info); } +static ssize_t pidfd_wait_read_iter (struct kiocb *iocb,struct iov_iter *to) +{ +/* +* This is just a test string, it will contain the actual +* status of the pidfd in the future. +*/ +char buf[] ="status" ; + +return copy_to_iter (buf,strlen (buf)+1 , to); +} + +static const struct file_operations pidfd_wait_file_ops = { + .read_iter = pidfd_wait_read_iter, +}; + +static struct inode *pidfd_wait_get_inode (struct super_block *sb) +{ +struct inode *inode =new_inode (sb); + + inode->i_ino =get_next_ino (); +inode_init_owner (inode,NULL ,S_IFREG ); + + inode->i_op = &simple_dir_inode_operations; + inode->i_fop = &pidfd_wait_file_ops; + +return inode; +} + +SYSCALL_DEFINE1(pidfd_wait, int, pidfd) +{ +struct fd f; +struct inode *inode; +struct file *file; +int new_fd; +struct pid_namespace *pid_ns; +struct super_block *sb; +struct vfsmount *mnt; + + f =fdget_raw (pidfd); +if (!f.file) +return -EBADF ;kernel/signal.c static int copy_siginfo_from_user_any(kernel_siginfo_t *kinfo, siginfo_t *info) · line 3603
Quoted 10 lines
kernel/signal.c +10
-0+ + sb =file_inode (f.file)->i_sb; + pid_ns = sb->s_fs_info; + + inode =pidfd_wait_get_inode (sb); + + mnt = pid_ns->proc_mnt; + + file =alloc_file_pseudo (inode, mnt,"pidfd_wait" ,O_RDONLY , + &pidfd_wait_file_ops);So I dislike the idea of allocating new inodes from the procfs super block. I would like to avoid pinning the whole pidfd concept exclusively
Quoted 8 lines
to proc. The idea is that the pidfd API will be useable through procfs via
open("/proc/<pid>")because that is what users expect and really wanted to have for a long time. So it makes sense to have this working. But it should really be useable without it. That's whytranslate_pid()andpidfd_clone()are on the table. What I'm saying is, once the pidfd api is "complete" you should be able to setCONFIG_PROCFS=N- even though that's crazy - and still be able to use pidfds. This is also a point akpm asked about when I did thepidfd_send_signalwork.
Oh, ok. Somehow 'proc' and 'pid' sound very similar in terminology so naturally I felt the proc fs superblock would be a fit, but I see your point.
Quoted 11 lines
So instead of going throught proc we should probably do what David has been doing in the mount API and come to rely on
anone_inode. So something like:fd =
anon_inode_getfd("pidfd", &pidfd_fops, file_priv_data, flags);and stash information such as pid namespace etc. in a pidfd struct or something that we then can stash
file->private_dataof the new file. This also lets us avoid all this open coding done here. Another advantage is thatanon_inodesis its own kernel-internal filesystem.
Thanks for the suggestion! Agreed this is better and will do it this way then.
thanks,
- Joel
Quoted 10 lines · Christian Brauner
So I dislike the idea of allocating new inodes from the procfs super block. I would like to avoid pinning the whole pidfd concept exclusively to proc. The idea is that the pidfd API will be useable through procfs via
open("/proc/<pid>")because that is what users expect and really wanted to have for a long time. So it makes sense to have this working. But it should really be useable without it. That's whytranslate_pid()andpidfd_clone()are on the table. What I'm saying is, once the pidfd api is "complete" you should be able to setCONFIG_PROCFS=N- even though that's crazy - and still be able to use pidfds. This is also a point akpm asked about when I did thepidfd_send_signalwork.
I agree that you shouldn't need CONFIG_PROCFS=Y to use pidfds. One crazy idea that I was discussing with Joel the other day is to just make CONFIG_PROCFS=Y mandatory and provide a new get_procfs_root() system call that returned, out of thin air and independent of the mount table, a procfs root directory file descriptor for the caller's PID namspace and suitable for use with openat(2).
C'mon: /proc is used by everyone today and almost every program breaks if it's not around. The string "/proc" is already de facto kernel ABI. Let's just drop the pretense of /proc being optional and bake it into the kernel proper, then give programs a way to get to /proc that isn't tied to any particular mount configuration. This way, we don't need a translate_pid(), since callers can just use procfs to do the same thing. (That is, if I understand correctly what translate_pid does.)
We still need a pidfd_clone() for atomicity reasons, but that's a separate story. My goal is to be able to write a library that transparently creates and manages a helper child process even in a "hostile" process environment in which some other uncoordinated thread is constantly doing a waitpid(-1) (e.g., the JVM).
Quoted 11 lines
So instead of going throught proc we should probably do what David has been doing in the mount API and come to rely on
anone_inode. So something like:fd =
anon_inode_getfd("pidfd", &pidfd_fops, file_priv_data, flags);and stash information such as pid namespace etc. in a pidfd struct or something that we then can stash
file->private_dataof the new file. This also lets us avoid all this open coding done here. Another advantage is thatanon_inodesis its own kernel-internal filesystem.
Sure. That works too.
Christian Brauner
maintainerQuoted 12 lines · Daniel Colascione
Quoted 10 lines · Christian Brauner
So I dislike the idea of allocating new inodes from the procfs super block. I would like to avoid pinning the whole pidfd concept exclusively to proc. The idea is that the pidfd API will be useable through procfs via
open("/proc/<pid>")because that is what users expect and really wanted to have for a long time. So it makes sense to have this working. But it should really be useable without it. That's whytranslate_pid()andpidfd_clone()are on the table. What I'm saying is, once the pidfd api is "complete" you should be able to setCONFIG_PROCFS=N- even though that's crazy - and still be able to use pidfds. This is also a point akpm asked about when I did thepidfd_send_signalwork.
Quoted 6 lines
I agree that you shouldn't need
CONFIG_PROCFS=Yto use pidfds. One crazy idea that I was discussing with Joel the other day is to just makeCONFIG_PROCFS=Ymandatory and provide a newget_procfs_root()system call that returned, out of thin air and independent of the mount table, a procfs root directory file descriptor for the caller's PID namspace and suitable for use with openat(2).
Even if this works I'm pretty sure that Al and a lot of others will not be happy about this. A syscall to get an fd to /proc? That's not going to happen and I don't see the need for a separate syscall just for that. (I do see the point of making CONFIG_PROCFS=y the default btw.)
Inode allocation from the procfs mount for the file descriptors Joel wants is not correct. Their not really procfs file descriptors so this is a nack. We can't just hook into proc that way.
Quoted 8 lines
C'mon: /proc is used by everyone today and almost every program breaks if it's not around. The string "/proc" is already de facto kernel ABI. Let's just drop the pretense of /proc being optional and bake it into the kernel proper, then give programs a way to get to /proc that isn't tied to any particular mount configuration. This way, we don't need a
translate_pid(), since callers can just use procfs to do the same thing. (That is, if I understand correctly whattranslate_piddoes.)
I'm not sure what you think translate_pid() is doing since you're not saying what you think it does.
Examples from the old patchset:
translate_pid(pid, ns, -1) - get pid in our pid namespace translate_pid(pid, -1, ns) - get pid in other pid namespace translate_pid(1, ns, -1) - get pid of init task for namespace translate_pid(pid, -1, ns) > 0 - is pid is reachable from ns? translate_pid(1, ns1, ns2) > 0 - is ns1 inside ns2?
translate_pid(1, ns1, ns2) == 0 - is ns1 outside ns2? translate_pid(1, ns1, ns2) == 1 - is ns1 equal ns2?
Allowing this syscall to yield pidfds as proper regular file fds and also taking pidfds as argument is an excellent way to kill a few problems at once:
- cheap pid namespace introspection
- creates a bridge between the "old" pid-based api and the "new" pidfd api
- allows us to get proper non-directory file descriptors for any pids we like
The additional advantage is that people are already happy to add this syscall so simply extending it and routing it through the pidfd tree or Eric's tree is reasonable. (It should probably grow a flag argument. I need to start prototyping this.)
Quoted 3 lines
We still need a
pidfd_clone()for atomicity reasons, but that's a separate story. My goal is to be able to write a library that
Yes, on my todo list and I have a ported patch based on prior working rotting somehwere on my git server.
Quoted 10 lines
transparently creates and manages a helper child process even in a "hostile" process environment in which some other uncoordinated thread is constantly doing a waitpid(-1) (e.g., the JVM).
Quoted 6 lines
So instead of going throught proc we should probably do what David has been doing in the mount API and come to rely on
anone_inode. So something like:fd =
anon_inode_getfd("pidfd", &pidfd_fops, file_priv_data, flags);
Quoted 7 lines
Quoted 5 lines
and stash information such as pid namespace etc. in a pidfd struct or something that we then can stash
file->private_dataof the new file. This also lets us avoid all this open coding done here. Another advantage is thatanon_inodesis its own kernel-internal filesystem.Sure. That works too.
Great.
Quoted 20 lines · Christian Brauner
Quoted 12 lines · Daniel Colascione
Quoted 10 lines · Christian Brauner
So I dislike the idea of allocating new inodes from the procfs super block. I would like to avoid pinning the whole pidfd concept exclusively to proc. The idea is that the pidfd API will be useable through procfs via
open("/proc/<pid>")because that is what users expect and really wanted to have for a long time. So it makes sense to have this working. But it should really be useable without it. That's whytranslate_pid()andpidfd_clone()are on the table. What I'm saying is, once the pidfd api is "complete" you should be able to setCONFIG_PROCFS=N- even though that's crazy - and still be able to use pidfds. This is also a point akpm asked about when I did thepidfd_send_signalwork.Quoted 6 lines
I agree that you shouldn't need
CONFIG_PROCFS=Yto use pidfds. One crazy idea that I was discussing with Joel the other day is to just makeCONFIG_PROCFS=Ymandatory and provide a newget_procfs_root()system call that returned, out of thin air and independent of the mount table, a procfs root directory file descriptor for the caller's PID namspace and suitable for use with openat(2).
Quoted 4 lines
Even if this works I'm pretty sure that Al and a lot of others will not be happy about this. A syscall to get an fd to /proc? That's not going to happen and I don't see the need for a separate syscall just for that. (I do see the point of making
CONFIG_PROCFS=ythe default btw.)
I think his point here was that he wanted a handle to procfs no matter where it was mounted and then can later use openat on that. Agreed that it may be unnecessary unless there is a usecase for it, and especially if the /proc directory being the defacto mountpoint for procfs is a universal convention.
Quoted 3 lines
Inode allocation from the procfs mount for the file descriptors Joel wants is not correct. Their not really procfs file descriptors so this is a nack. We can't just hook into proc that way.
I was not particular about using procfs mount for the FDs but that's the only way I knew how to do it until you pointed out anon_inode (my grep skills missed that), so thank you!
Quoted 16 lines
Quoted 7 lines
C'mon: /proc is used by everyone today and almost every program breaks if it's not around. The string "/proc" is already de facto kernel ABI. Let's just drop the pretense of /proc being optional and bake it into the kernel proper, then give programs a way to get to /proc that isn't tied to any particular mount configuration. This way, we don't need a
translate_pid(), since callers can just use procfs to do the same thing. (That is, if I understand correctly whattranslate_piddoes.)I'm not sure what you think
translate_pid()is doing since you're not saying what you think it does.
Examples from the old patchset:translate_pid(pid, ns, -1) - get pid in our pid namespace translate_pid(pid, -1, ns) - get pid in other pid namespace translate_pid(1, ns, -1) - get pid of init task for namespace translate_pid(pid, -1, ns) > 0 - is pid is reachable from ns? translate_pid(1, ns1, ns2) > 0 - is ns1 inside ns2?
Quoted 8 lines
translate_pid(1, ns1, ns2)== 0 - is ns1 outside ns2?translate_pid(1, ns1, ns2)== 1 - is ns1 equal ns2?Allowing this syscall to yield pidfds as proper regular file fds and also taking pidfds as argument is an excellent way to kill a few problems at once:
- cheap pid namespace introspection
- creates a bridge between the "old" pid-based api and the "new" pidfd api
This second point would solve the problem of getting a new pidfd given a pid indeed, without depending on /proc/<pid> at all. So kudos for that and I am glad you are making it return pidfds (but correct me if I misunderstood what you're planning to do with translate_fd). It also obviates any need for dealing with procfs mount points.
Quoted 2 lines
- allows us to get proper non-directory file descriptors for any pids we like
Here I got a bit lost. AIUI pidfd is a directory fd. Why would we want it to not be a directory fd? That would be ambigiuous with what pidfd_send_signal expects.
Also would it be a bad idea to extend translate_pid to also do what we want for the pidfd_wait syscall? So translate_fd in this case would return an fd that is just used for the pid's death status.
All of these extensions seem to mean translate_pid should probably take a fourth parameter that tells it the target translation type?
They way I am hypothesizing, translate_pid, it should probably be
- translation to a pid in some ns
- translation of a pid to a pidfd
- translation of a pid to a "wait" fd which returns the death/reap process status.
If that makes sense, that would also avoid the need for a new syscall we are adding.
Quoted 4 lines
The additional advantage is that people are already happy to add this syscall so simply extending it and routing it through the pidfd tree or Eric's tree is reasonable. (It should probably grow a flag argument. I need to start prototyping this.)
Great!
Quoted 6 lines
Quoted 3 lines
We still need a
pidfd_clone()for atomicity reasons, but that's a separate story. My goal is to be able to write a library thatYes, on my todo list and I have a ported patch based on prior working rotting somehwere on my git server.
Is that different from using dup2 on a pidfd? Sorry I don't follow what is pidfd_clone / why it is needed.
thanks,
- Joel
pidfd design
Quoted 22 lines · Joel Fernandes
Quoted 20 lines · Christian Brauner
Quoted 18 lines · Daniel Colascione
Quoted 10 lines · Christian Brauner
So I dislike the idea of allocating new inodes from the procfs super block. I would like to avoid pinning the whole pidfd concept exclusively to proc. The idea is that the pidfd API will be useable through procfs via
open("/proc/<pid>")because that is what users expect and really wanted to have for a long time. So it makes sense to have this working. But it should really be useable without it. That's whytranslate_pid()andpidfd_clone()are on the table. What I'm saying is, once the pidfd api is "complete" you should be able to setCONFIG_PROCFS=N- even though that's crazy - and still be able to use pidfds. This is also a point akpm asked about when I did thepidfd_send_signalwork.I agree that you shouldn't need
CONFIG_PROCFS=Yto use pidfds. One crazy idea that I was discussing with Joel the other day is to just makeCONFIG_PROCFS=Ymandatory and provide a newget_procfs_root()system call that returned, out of thin air and independent of the mount table, a procfs root directory file descriptor for the caller's PID namspace and suitable for use with openat(2).
Quoted 2 lines
Quoted 2 lines
Even if this works I'm pretty sure that Al and a lot of others will not be happy about this. A syscall to get an fd to /proc?
Why not? procfs provides access to a lot of core kernel functionality. Why should you need a mountpoint to get to it?
Quoted 2 lines
That's not going
Quoted 1 line
to happen and I don't see the need for a separate syscall just for that.
We need a system call for the same reason we need a getrandom(2): you have to bootstrap somehow when you're in a minimal environment.
Quoted 1 line
Quoted 1 line
(I do see the point of making
CONFIG_PROCFS=ythe default btw.)
I'm not proposing that we make CONFIG_PROCFS=y the default. I'm proposing that we *hardwire* it as the default and just declare that it's not possible to build a Linux kernel that doesn't include procfs. Why do we even have that button?
Quoted 4 lines
I think his point here was that he wanted a handle to procfs no matter where it was mounted and then can later use openat on that. Agreed that it may be unnecessary unless there is a usecase for it, and especially if the /proc directory being the defacto mountpoint for procfs is a universal convention.
If it's a universal convention and, in practice, everyone needs proc mounted anyway, so what's the harm in hardwiring CONFIG_PROCFS=y? If we advertise /proc as not merely some kind of optional debug interface but *the* way certain kernel features are exposed --- and there's nothing wrong with that --- then we should give programs access to these core kernel features in a way that doesn't depend on userspace kernel configuration, and you do that by either providing a procfs-root-getting system call or just hardwiring the "/proc/" prefix into VFS.
Quoted 4 lines
Quoted 3 lines
Inode allocation from the procfs mount for the file descriptors Joel wants is not correct. Their not really procfs file descriptors so this is a nack. We can't just hook into proc that way.
Quoted 16 lines
I was not particular about using procfs mount for the FDs but that's the only way I knew how to do it until you pointed out
anon_inode(my grep skills missed that), so thank you!Quoted 12 lines
Quoted 7 lines
C'mon: /proc is used by everyone today and almost every program breaks if it's not around. The string "/proc" is already de facto kernel ABI. Let's just drop the pretense of /proc being optional and bake it into the kernel proper, then give programs a way to get to /proc that isn't tied to any particular mount configuration. This way, we don't need a
translate_pid(), since callers can just use procfs to do the same thing. (That is, if I understand correctly whattranslate_piddoes.)I'm not sure what you think
translate_pid()is doing since you're not saying what you think it does.
Examples from the old patchset:translate_pid(pid, ns, -1)- get pid in our pid namespace
Ah, it's a bit different from what I had in mind. It's fair to want to translate PIDs between namespaces, but the only way to make the translate_pid under discussion robust is to have it accept and produce pidfds. (At that point, you might as well call it translate_pidfd.) We should not be adding new APIs to the kernel that accept numeric PIDs:
it's not possible to use these APIs correctly except under very limited circumstances --- mostly, talking about init or a parent talking about its child.
Really, we need a few related operations, and we shouldn't necessarily mingle them.
1) Given a numeric PID, give me a pidfd: that works today: you just open /proc/<pid>
2) Given a pidfd, give me a numeric PID: that works today: you just openat(pidfd, "stat", O_RDONLY) and read the first token (which is always the numeric PID).
3) Given a pidfd, send a signal: that's what pidfd_send_signal does, and it's a good start on the rest of these operations.
4) Given a pidfd, wait for the named process to exit: that's what my original exithand thing did, and that's what Joel's helpfully agreed to start hacking on.
5) Given a pidfd in NS1, get a pidfd in NS2. That's what translate_pid is for. My preferred signature for this routine is translate_pid(int pidfd, int nsfd) -> pidfd. We don't need two namespace arguments. Why not? Because the pidfd *already* names a single process, uniquely!
6) Make a new process and atomically give me a pidfd for it. We need a new kind of clone(2) for that. People have been proposing some kind of FD-based fork/spawn/etc. thing for ages, and we can finally provide it. Yay.
7) Retrieve miscellaneous information about a process identified by a
pidfd: openat(2) handles this case today.
This is a decent framework for a good general-purpose process API that builds on the one the kernel already provides. With this API, people should never have to touch the old unix process API except to talk to humans and other legacy systems. It's a big project, but worthwhile, and we can do it piecemeal.
Christian, what worries me is that you want to make this project 10x harder, both in technical and lkml-political terms, by making it work without CONFIG_PROCFS=y. Without procfs, all the operations above that involve the word "openat" or "/proc" break, which means that our general-purpose process API needs to provide its own equivalents to these operations, and on top of these, its own non-procfs pidfd FD type --- let's call it pidfd_2. (Let's call a directory FD on /proc/<pid> a pidfd_1.) Under this scheme, we have to have all operations that accept a pidfd_1 (like pidfd_send_signal) and have them accept pidfd_2 file descriptors as well in general fashion. (The difference between pidfd_1 and pidfd_2 is visible to users who can call fstat and look at st_dev.) We'd also need an API to translate a pidfd_2 to a pidfd_1 so you could call openat on it to look at /proc/<pid> data files, to support operation #7 above. The alternative to provide #7 is some kind of new general-purpose process-information-retrieval interface that mirrors the functionality /proc/<pid> already provides --- e.g., getting the smaps list for a process.
To sum it up, we can
A) declare that pidfds don't work without CONFIG_PROCFS=y,
B) hardwire CONFIG_PROCFS=y in all configurations, or
C) support both procfs-based pidfd_1 FDs and non-procfs pidfd_2 FDs.
Option C seems like pointless complexity to me, as I described above. Option C means that we have to duplicate a lot of existing and perfectly good functionality.
Option A is fine by me, since I think CONFIG_PROCFS=n is just a bonkers and broken configuration that's barely even Linux.
From a design perspective, I prefer option B: it turns a de-facto guaranteed /proc ABI into a de-jure guaranteed ABI, and that's just more straightforward for everyone --- especially since it reduces the complexity of the Linux core by deleting all the !CONFIG_PROCFS code paths. My point about the procfs system call is that *if* we go with option B and make procfs mandatory, we're essentially stating that certain kernel functionality is always available, and because (as a general rule) kernel functionality that's always available should be available to every process, we should provide a way to *use* this always-present kernel functionality that doesn't depend on the mount table --- thus my proposed get_procfs_root(2).
We don't have to decide between A and B right now. We can continue develop pidfd development under the assumption we're going with option A, and when option B seems like a good idea, we can just switch with minimal hassle. On the other hand, if we did implement option C and, later, it became apparently that option B was right after all, all the work needed for option C would have been a waste.
Christian Brauner
maintainerRe: pidfd design
Quoted 78 lines · Daniel Colascione
Quoted 22 lines · Joel Fernandes
Quoted 20 lines · Christian Brauner
Quoted 18 lines · Daniel Colascione
Quoted 10 lines · Christian Brauner
So I dislike the idea of allocating new inodes from the procfs super block. I would like to avoid pinning the whole pidfd concept exclusively to proc. The idea is that the pidfd API will be useable through procfs via
open("/proc/<pid>")because that is what users expect and really wanted to have for a long time. So it makes sense to have this working. But it should really be useable without it. That's whytranslate_pid()andpidfd_clone()are on the table. What I'm saying is, once the pidfd api is "complete" you should be able to setCONFIG_PROCFS=N- even though that's crazy - and still be able to use pidfds. This is also a point akpm asked about when I did thepidfd_send_signalwork.I agree that you shouldn't need
CONFIG_PROCFS=Yto use pidfds. One crazy idea that I was discussing with Joel the other day is to just makeCONFIG_PROCFS=Ymandatory and provide a newget_procfs_root()system call that returned, out of thin air and independent of the mount table, a procfs root directory file descriptor for the caller's PID namspace and suitable for use with openat(2).Quoted 2 lines
Quoted 2 lines
Even if this works I'm pretty sure that Al and a lot of others will not be happy about this. A syscall to get an fd to /proc?
Why not? procfs provides access to a lot of core kernel functionality. Why should you need a mountpoint to get to it?
Quoted 2 lines
That's not going
Quoted 1 line
to happen and I don't see the need for a separate syscall just for that.
We need a system call for the same reason we need a getrandom(2): you have to bootstrap somehow when you're in a minimal environment.
Quoted 1 line
Quoted 1 line
(I do see the point of making
CONFIG_PROCFS=ythe default btw.)I'm not proposing that we make
CONFIG_PROCFS=ythe default. I'm proposing that we *hardwire* it as the default and just declare that it's not possible to build a Linux kernel that doesn't include procfs. Why do we even have that button?Quoted 4 lines
I think his point here was that he wanted a handle to procfs no matter where it was mounted and then can later use openat on that. Agreed that it may be unnecessary unless there is a usecase for it, and especially if the /proc directory being the defacto mountpoint for procfs is a universal convention.
If it's a universal convention and, in practice, everyone needs proc mounted anyway, so what's the harm in hardwiring
CONFIG_PROCFS=y? If we advertise /proc as not merely some kind of optional debug interface but *the* way certain kernel features are exposed --- and there's nothing wrong with that --- then we should give programs access to these core kernel features in a way that doesn't depend on userspace kernel configuration, and you do that by either providing a procfs-root-getting system call or just hardwiring the "/proc/" prefix into VFS.Quoted 4 lines
Quoted 3 lines
Inode allocation from the procfs mount for the file descriptors Joel wants is not correct. Their not really procfs file descriptors so this is a nack. We can't just hook into proc that way.
Quoted 16 lines
I was not particular about using procfs mount for the FDs but that's the only way I knew how to do it until you pointed out
anon_inode(my grep skills missed that), so thank you!Quoted 12 lines
Quoted 7 lines
C'mon: /proc is used by everyone today and almost every program breaks if it's not around. The string "/proc" is already de facto kernel ABI. Let's just drop the pretense of /proc being optional and bake it into the kernel proper, then give programs a way to get to /proc that isn't tied to any particular mount configuration. This way, we don't need a
translate_pid(), since callers can just use procfs to do the same thing. (That is, if I understand correctly whattranslate_piddoes.)I'm not sure what you think
translate_pid()is doing since you're not saying what you think it does.
Examples from the old patchset:translate_pid(pid, ns, -1)- get pid in our pid namespace
Quoted 5 lines
Ah, it's a bit different from what I had in mind. It's fair to want to translate PIDs between namespaces, but the only way to make the
translate_pidunder discussion robust is to have it accept and produce pidfds. (At that point, you might as well call ittranslate_pidfd.) We should not be adding new APIs to the kernel that accept numeric PIDs:
The traditional pid-based api is not going away. There are users that have the requirement to translate pids between namespaces and also doing introspection on these namespaces independent of pidfds. We will not restrict the usefulness of this syscall by making it only work with pidfds.
Quoted 2 lines
it's not possible to use these APIs correctly except under very limited circumstances --- mostly, talking about init or a parent
The pid-based api is one of the most widely used apis of the kernel and people have been using it quite successfully for a long time. Yes, it's rac, but it's here to stay.
Quoted 4 lines
talking about its child.
Really, we need a few related operations, and we shouldn't necessarily mingle them.
Yes, we've established that previously.
Quoted 3 lines
1) Given a numeric PID, give me a pidfd: that works today: you just open /proc/<pid>
Agreed.
Quoted 4 lines
2) Given a pidfd, give me a numeric PID: that works today: you just
openat(pidfd, "stat", O_RDONLY)and read the first token (which is always the numeric PID).
Agreed.
Quoted 3 lines
3) Given a pidfd, send a signal: that's what
pidfd_send_signaldoes, and it's a good start on the rest of these operations.
Agreed.
Quoted 4 lines
5) Given a pidfd in NS1, get a pidfd in NS2. That's what
translate_pidis for. My preferred signature for this routine istranslate_pid(int pidfd, int nsfd)-> pidfd. We don't need two namespace arguments. Why not? Because the pidfd *already* names a single process, uniquely!
Given that people are interested in pids we can't just always return a pidfd. That would mean a user would need to do get the pidfd read from <pidfd>/stat and then close the pidfd. If you do that for a 100 pids or more you end up allocating and closing file descriptors constantly for no reason. We can't just debate pids away. So it will also need to be able to yield pids e.g. through a flag argument.
Quoted 5 lines
6) Make a new process and atomically give me a pidfd for it. We need a new kind of clone(2) for that. People have been proposing some kind of FD-based fork/spawn/etc. thing for ages, and we can finally provide it. Yay.
Agreed.
Re: pidfd design
Quoted 130 lines · Christian Brauner
Quoted 78 lines · Daniel Colascione
Quoted 22 lines · Joel Fernandes
Quoted 20 lines · Christian Brauner
Quoted 18 lines · Daniel Colascione
Quoted 10 lines · Christian Brauner
So I dislike the idea of allocating new inodes from the procfs super block. I would like to avoid pinning the whole pidfd concept exclusively to proc. The idea is that the pidfd API will be useable through procfs via
open("/proc/<pid>")because that is what users expect and really wanted to have for a long time. So it makes sense to have this working. But it should really be useable without it. That's whytranslate_pid()andpidfd_clone()are on the table. What I'm saying is, once the pidfd api is "complete" you should be able to setCONFIG_PROCFS=N- even though that's crazy - and still be able to use pidfds. This is also a point akpm asked about when I did thepidfd_send_signalwork.I agree that you shouldn't need
CONFIG_PROCFS=Yto use pidfds. One crazy idea that I was discussing with Joel the other day is to just makeCONFIG_PROCFS=Ymandatory and provide a newget_procfs_root()system call that returned, out of thin air and independent of the mount table, a procfs root directory file descriptor for the caller's PID namspace and suitable for use with openat(2).Quoted 2 lines
Quoted 2 lines
Even if this works I'm pretty sure that Al and a lot of others will not be happy about this. A syscall to get an fd to /proc?
Why not? procfs provides access to a lot of core kernel functionality. Why should you need a mountpoint to get to it?
Quoted 2 lines
That's not going
Quoted 1 line
to happen and I don't see the need for a separate syscall just for that.
We need a system call for the same reason we need a getrandom(2): you have to bootstrap somehow when you're in a minimal environment.
Quoted 1 line
Quoted 1 line
(I do see the point of making
CONFIG_PROCFS=ythe default btw.)I'm not proposing that we make
CONFIG_PROCFS=ythe default. I'm proposing that we *hardwire* it as the default and just declare that it's not possible to build a Linux kernel that doesn't include procfs. Why do we even have that button?Quoted 4 lines
I think his point here was that he wanted a handle to procfs no matter where it was mounted and then can later use openat on that. Agreed that it may be unnecessary unless there is a usecase for it, and especially if the /proc directory being the defacto mountpoint for procfs is a universal convention.
If it's a universal convention and, in practice, everyone needs proc mounted anyway, so what's the harm in hardwiring
CONFIG_PROCFS=y? If we advertise /proc as not merely some kind of optional debug interface but *the* way certain kernel features are exposed --- and there's nothing wrong with that --- then we should give programs access to these core kernel features in a way that doesn't depend on userspace kernel configuration, and you do that by either providing a procfs-root-getting system call or just hardwiring the "/proc/" prefix into VFS.Quoted 4 lines
Quoted 3 lines
Inode allocation from the procfs mount for the file descriptors Joel wants is not correct. Their not really procfs file descriptors so this is a nack. We can't just hook into proc that way.
Quoted 16 lines
I was not particular about using procfs mount for the FDs but that's the only way I knew how to do it until you pointed out
anon_inode(my grep skills missed that), so thank you!Quoted 12 lines
Quoted 7 lines
C'mon: /proc is used by everyone today and almost every program breaks if it's not around. The string "/proc" is already de facto kernel ABI. Let's just drop the pretense of /proc being optional and bake it into the kernel proper, then give programs a way to get to /proc that isn't tied to any particular mount configuration. This way, we don't need a
translate_pid(), since callers can just use procfs to do the same thing. (That is, if I understand correctly whattranslate_piddoes.)I'm not sure what you think
translate_pid()is doing since you're not saying what you think it does.
Examples from the old patchset:translate_pid(pid, ns, -1)- get pid in our pid namespaceQuoted 5 lines
Ah, it's a bit different from what I had in mind. It's fair to want to translate PIDs between namespaces, but the only way to make the
translate_pidunder discussion robust is to have it accept and produce pidfds. (At that point, you might as well call ittranslate_pidfd.) We should not be adding new APIs to the kernel that accept numeric PIDs:The traditional pid-based api is not going away. There are users that have the requirement to translate pids between namespaces and also doing introspection on these namespaces independent of pidfds. We will not restrict the usefulness of this syscall by making it only work with pidfds.
Quoted 2 lines
it's not possible to use these APIs correctly except under very limited circumstances --- mostly, talking about init or a parent
The pid-based api is one of the most widely used apis of the kernel and people have been using it quite successfully for a long time. Yes, it's rac, but it's here to stay.
Quoted 4 lines
talking about its child.
Really, we need a few related operations, and we shouldn't necessarily mingle them.
Yes, we've established that previously.
Quoted 3 lines
1) Given a numeric PID, give me a pidfd: that works today: you just open /proc/<pid>
Agreed.
Quoted 4 lines
2) Given a pidfd, give me a numeric PID: that works today: you just
openat(pidfd, "stat", O_RDONLY)and read the first token (which is always the numeric PID).Agreed.
Quoted 3 lines
3) Given a pidfd, send a signal: that's what
pidfd_send_signaldoes, and it's a good start on the rest of these operations.Agreed.
Quoted 4 lines
5) Given a pidfd in NS1, get a pidfd in NS2. That's what
translate_pidis for. My preferred signature for this routine istranslate_pid(int pidfd, int nsfd)-> pidfd. We don't need two namespace arguments. Why not? Because the pidfd *already* names a single process, uniquely!
Quoted 6 lines
Given that people are interested in pids we can't just always return a pidfd. That would mean a user would need to do get the pidfd read from <pidfd>/stat and then close the pidfd. If you do that for a 100 pids or more you end up allocating and closing file descriptors constantly for no reason. We can't just debate pids away. So it will also need to be able to yield pids e.g. through a flag argument.
Sure, but that's still not a reason that we should care about pidfds working separately from procfs.
Re: pidfd design
Quoted 195 lines · Daniel Colascione
Quoted 3 lines · Christian Brauner
Quoted 1 line · Daniel Colascione
On Tue, Mar 19, 2019 at 6:52 PM Joel Fernandes
Quoted 2 lines · <[email protected]> wrote:
Quoted 2 lines
Quoted 2 lines
On Wed, Mar 20, 2019 at 12:10:23AM +0100, Christian Brauner
Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
On Tue, Mar 19, 2019 at 03:48:32PM -0700, Daniel Colascione
Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
On Tue, Mar 19, 2019 at 3:14 PM Christian Brauner
Quoted 1 line · <[email protected]> wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
So I dislike the idea of allocating new inodes from the
procfs super
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
block. I would like to avoid pinning the whole pidfd
concept exclusively
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to proc. The idea is that the pidfd API will be useable
through procfs
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
via
open("/proc/<pid>")because that is what users expectand really
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
wanted to have for a long time. So it makes sense to have
this working.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
But it should really be useable without it. That's why
translate_pid()Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
and
pidfd_clone()are on the table. What I'm saying is,once the pidfd
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
api is "complete" you should be able to set
CONFIG_PROCFS=N- even
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
though that's crazy - and still be able to use pidfds. This
is also a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
point akpm asked about when I did the
pidfd_send_signalwork.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I agree that you shouldn't need
CONFIG_PROCFS=Yto usepidfds. One
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
crazy idea that I was discussing with Joel the other day is
to just
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
make
CONFIG_PROCFS=Ymandatory and provide a new
get_procfs_root()Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
system call that returned, out of thin air and independent of
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
mount table, a procfs root directory file descriptor for the
caller's
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
PID namspace and suitable for use with openat(2).
Even if this works I'm pretty sure that Al and a lot of others
will not
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
be happy about this. A syscall to get an fd to /proc?
Why not? procfs provides access to a lot of core kernel
functionality.
Quoted 4 lines
Quoted 4 lines
Why should you need a mountpoint to get to it?
Quoted 2 lines
That's not going
Quoted 1 line
to happen and I don't see the need for a separate syscall just
for that.
Quoted 2 lines
Quoted 2 lines
We need a system call for the same reason we need a getrandom(2):
you
Quoted 6 lines
Quoted 6 lines
have to bootstrap somehow when you're in a minimal environment.
Quoted 1 line
Quoted 1 line
(I do see the point of making
CONFIG_PROCFS=ythe default btw.)I'm not proposing that we make
CONFIG_PROCFS=ythe default. I'm proposing that we *hardwire* it as the default and just declarethat
Quoted 1 line
Quoted 1 line
it's not possible to build a Linux kernel that doesn't include
procfs.
Quoted 3 lines
Quoted 3 lines
Why do we even have that button?
Quoted 1 line
I think his point here was that he wanted a handle to procfs no
matter where
Quoted 1 line
Quoted 1 line
Quoted 1 line
it was mounted and then can later use openat on that. Agreed that
it may be
Quoted 1 line
Quoted 1 line
Quoted 1 line
unnecessary unless there is a usecase for it, and especially if
the /proc
Quoted 1 line
Quoted 1 line
Quoted 1 line
directory being the defacto mountpoint for procfs is a universal
convention.
Quoted 2 lines
Quoted 2 lines
If it's a universal convention and, in practice, everyone needs
proc
Quoted 1 line
Quoted 1 line
mounted anyway, so what's the harm in hardwiring
CONFIG_PROCFS=y?If
Quoted 1 line
Quoted 1 line
we advertise /proc as not merely some kind of optional debug
interface
Quoted 3 lines
Quoted 3 lines
but *the* way certain kernel features are exposed --- and there's nothing wrong with that --- then we should give programs access to these core kernel features in a way that doesn't depend on
userspace
Quoted 2 lines
Quoted 2 lines
kernel configuration, and you do that by either providing a procfs-root-getting system call or just hardwiring the "
/proc/"prefix
Quoted 3 lines
Quoted 3 lines
into VFS.
Quoted 1 line
Quoted 1 line
Inode allocation from the procfs mount for the file descriptors
Joel
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
wants is not correct. Their not really procfs file descriptors
so this
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
is a nack. We can't just hook into proc that way.
I was not particular about using procfs mount for the FDs but
that's the only
Quoted 1 line
Quoted 1 line
Quoted 1 line
way I knew how to do it until you pointed out
anon_inode(my grepskills
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
missed that), so thank you!
Quoted 1 line
Quoted 1 line
C'mon: /proc is used by everyone today and almost every
program breaks
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
if it's not around. The string "/proc" is already de facto
kernel ABI.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Let's just drop the pretense of /proc being optional and bake
it into
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the kernel proper, then give programs a way to get to /proc
that isn't
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
tied to any particular mount configuration. This way, we
don't need a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
translate_pid(), since callers can just use procfs to do thesame
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
thing. (That is, if I understand correctly what
translate_piddoes.)
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I'm not sure what you think
translate_pid()is doing sinceyou're not
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
saying what you think it does.
Examples from the old patchset:translate_pid(pid, ns, -1)- get pid in our pid namespaceAh, it's a bit different from what I had in mind. It's fair to want
to
Quoted 2 lines
Quoted 2 lines
translate PIDs between namespaces, but the only way to make the
translate_pidunder discussion robust is to have it accept andproduce
Quoted 1 line
Quoted 1 line
pidfds. (At that point, you might as well call it
translate_pidfd.)We
Quoted 1 line
Quoted 1 line
should not be adding new APIs to the kernel that accept numeric
PIDs:
Quoted 3 lines
The traditional pid-based api is not going away. There are users that have the requirement to translate pids between namespaces and also
doing
Quoted 8 lines
introspection on these namespaces independent of pidfds. We will not restrict the usefulness of this syscall by making it only work with pidfds.
Quoted 2 lines
it's not possible to use these APIs correctly except under very limited circumstances --- mostly, talking about init or a parent
The pid-based api is one of the most widely used apis of the kernel
and
Quoted 1 line
people have been using it quite successfully for a long time. Yes,
it's
Quoted 5 lines
rac, but it's here to stay.
Quoted 3 lines
talking about its child.
Really, we need a few related operations, and we shouldn't
necessarily
Quoted 8 lines
Quoted 1 line
mingle them.
Yes, we've established that previously.
Quoted 3 lines
1) Given a numeric PID, give me a pidfd: that works today: you just open /proc/<pid>
Quoted 11 lines
Agreed.
Quoted 4 lines
2) Given a pidfd, give me a numeric PID: that works today: you just
openat(pidfd, "stat", O_RDONLY)and read the first token (which is always the numeric PID).Agreed.
Quoted 2 lines
3) Given a pidfd, send a signal: that's what
pidfd_send_signaldoes,
Quoted 5 lines
Quoted 1 line
and it's a good start on the rest of these operations.
Agreed.
Quoted 1 line
5) Given a pidfd in NS1, get a pidfd in NS2. That's what
translate_pidQuoted 1 line
Quoted 1 line
is for. My preferred signature for this routine is
translate_pid(int
Quoted 1 line
Quoted 1 line
pidfd, int nsfd) -> pidfd. We don't need two namespace arguments.
Why
Quoted 3 lines
Quoted 1 line
not? Because the pidfd *already* names a single process, uniquely!
Given that people are interested in pids we can't just always return
a
Quoted 1 line
pidfd. That would mean a user would need to do get the pidfd read
from
Quoted 1 line
<pidfd>/stat and then close the pidfd. If you do that for a 100 pids
Quoted 8 lines
or
Quoted 1 line
more you end up allocating and closing file descriptors constantly
for
Quoted 2 lines
no reason. We can't just debate pids away. So it will also need to be able to yield pids e.g. through a flag argument.
Sure, but that's still not a reason that we should care about pidfds working separately from procfs..
Agreed. I can't imagine pidfd being anything but a proc pid directory handle. So I am confused what Christian meant. Pidfd *is* a procfs directory fid always. That's what I gathered from his pidfd_send_signal patch but let me know if I'm way off in the woods.
For my next revision, I am thinking of adding the flag argument Christian mentioned to make translate_pid return an anon_inode FD which can be used for death status, given a <pid>. Since it is thought that translate_pid can be made to return a pid FD, I think it is ok to have it return a pid status FD for the purposes of the death status as well.
Joel Fernandes, Android kernel team
Sent from k9-mail on Android
Christian Brauner
maintainerRe: pidfd design
Quoted 198 lines · Joel Fernandes
Quoted 195 lines · Daniel Colascione
Quoted 3 lines · Christian Brauner
Quoted 1 line · Daniel Colascione
On Tue, Mar 19, 2019 at 6:52 PM Joel Fernandes
Quoted 2 lines · <[email protected]> wrote:
Quoted 2 lines
Quoted 2 lines
On Wed, Mar 20, 2019 at 12:10:23AM +0100, Christian Brauner
Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
On Tue, Mar 19, 2019 at 03:48:32PM -0700, Daniel Colascione
Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
On Tue, Mar 19, 2019 at 3:14 PM Christian Brauner
Quoted 1 line · <[email protected]> wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
So I dislike the idea of allocating new inodes from the
procfs super
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
block. I would like to avoid pinning the whole pidfd
concept exclusively
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to proc. The idea is that the pidfd API will be useable
through procfs
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
via
open("/proc/<pid>")because that is what users expectand really
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
wanted to have for a long time. So it makes sense to have
this working.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
But it should really be useable without it. That's why
translate_pid()Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
and
pidfd_clone()are on the table. What I'm saying is,once the pidfd
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
api is "complete" you should be able to set
CONFIG_PROCFS=N- even
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
though that's crazy - and still be able to use pidfds. This
is also a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
point akpm asked about when I did the
pidfd_send_signalwork.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I agree that you shouldn't need
CONFIG_PROCFS=Yto usepidfds. One
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
crazy idea that I was discussing with Joel the other day is
to just
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
make
CONFIG_PROCFS=Ymandatory and provide a new
get_procfs_root()Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
system call that returned, out of thin air and independent of
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
mount table, a procfs root directory file descriptor for the
caller's
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
PID namspace and suitable for use with openat(2).
Even if this works I'm pretty sure that Al and a lot of others
will not
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
be happy about this. A syscall to get an fd to /proc?
Why not? procfs provides access to a lot of core kernel
functionality.
Quoted 4 lines
Quoted 4 lines
Why should you need a mountpoint to get to it?
Quoted 2 lines
That's not going
Quoted 1 line
to happen and I don't see the need for a separate syscall just
for that.
Quoted 2 lines
Quoted 2 lines
We need a system call for the same reason we need a getrandom(2):
you
Quoted 6 lines
Quoted 6 lines
have to bootstrap somehow when you're in a minimal environment.
Quoted 1 line
Quoted 1 line
(I do see the point of making
CONFIG_PROCFS=ythe default btw.)I'm not proposing that we make
CONFIG_PROCFS=ythe default. I'm proposing that we *hardwire* it as the default and just declarethat
Quoted 1 line
Quoted 1 line
it's not possible to build a Linux kernel that doesn't include
procfs.
Quoted 3 lines
Quoted 3 lines
Why do we even have that button?
Quoted 1 line
I think his point here was that he wanted a handle to procfs no
matter where
Quoted 1 line
Quoted 1 line
Quoted 1 line
it was mounted and then can later use openat on that. Agreed that
it may be
Quoted 1 line
Quoted 1 line
Quoted 1 line
unnecessary unless there is a usecase for it, and especially if
the /proc
Quoted 1 line
Quoted 1 line
Quoted 1 line
directory being the defacto mountpoint for procfs is a universal
convention.
Quoted 2 lines
Quoted 2 lines
If it's a universal convention and, in practice, everyone needs
proc
Quoted 1 line
Quoted 1 line
mounted anyway, so what's the harm in hardwiring
CONFIG_PROCFS=y?If
Quoted 1 line
Quoted 1 line
we advertise /proc as not merely some kind of optional debug
interface
Quoted 3 lines
Quoted 3 lines
but *the* way certain kernel features are exposed --- and there's nothing wrong with that --- then we should give programs access to these core kernel features in a way that doesn't depend on
userspace
Quoted 2 lines
Quoted 2 lines
kernel configuration, and you do that by either providing a procfs-root-getting system call or just hardwiring the "
/proc/"prefix
Quoted 3 lines
Quoted 3 lines
into VFS.
Quoted 1 line
Quoted 1 line
Inode allocation from the procfs mount for the file descriptors
Joel
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
wants is not correct. Their not really procfs file descriptors
so this
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
is a nack. We can't just hook into proc that way.
I was not particular about using procfs mount for the FDs but
that's the only
Quoted 1 line
Quoted 1 line
Quoted 1 line
way I knew how to do it until you pointed out
anon_inode(my grepskills
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
missed that), so thank you!
Quoted 1 line
Quoted 1 line
C'mon: /proc is used by everyone today and almost every
program breaks
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
if it's not around. The string "/proc" is already de facto
kernel ABI.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Let's just drop the pretense of /proc being optional and bake
it into
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the kernel proper, then give programs a way to get to /proc
that isn't
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
tied to any particular mount configuration. This way, we
don't need a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
translate_pid(), since callers can just use procfs to do thesame
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
thing. (That is, if I understand correctly what
translate_piddoes.)
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I'm not sure what you think
translate_pid()is doing sinceyou're not
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
saying what you think it does.
Examples from the old patchset:translate_pid(pid, ns, -1)- get pid in our pid namespaceAh, it's a bit different from what I had in mind. It's fair to want
to
Quoted 2 lines
Quoted 2 lines
translate PIDs between namespaces, but the only way to make the
translate_pidunder discussion robust is to have it accept andproduce
Quoted 1 line
Quoted 1 line
pidfds. (At that point, you might as well call it
translate_pidfd.)We
Quoted 1 line
Quoted 1 line
should not be adding new APIs to the kernel that accept numeric
PIDs:
Quoted 3 lines
The traditional pid-based api is not going away. There are users that have the requirement to translate pids between namespaces and also
doing
Quoted 8 lines
introspection on these namespaces independent of pidfds. We will not restrict the usefulness of this syscall by making it only work with pidfds.
Quoted 2 lines
it's not possible to use these APIs correctly except under very limited circumstances --- mostly, talking about init or a parent
The pid-based api is one of the most widely used apis of the kernel
and
Quoted 1 line
people have been using it quite successfully for a long time. Yes,
it's
Quoted 5 lines
rac, but it's here to stay.
Quoted 3 lines
talking about its child.
Really, we need a few related operations, and we shouldn't
necessarily
Quoted 8 lines
Quoted 1 line
mingle them.
Yes, we've established that previously.
Quoted 3 lines
1) Given a numeric PID, give me a pidfd: that works today: you just open /proc/<pid>
Quoted 11 lines
Agreed.
Quoted 4 lines
2) Given a pidfd, give me a numeric PID: that works today: you just
openat(pidfd, "stat", O_RDONLY)and read the first token (which is always the numeric PID).Agreed.
Quoted 2 lines
3) Given a pidfd, send a signal: that's what
pidfd_send_signaldoes,
Quoted 5 lines
Quoted 1 line
and it's a good start on the rest of these operations.
Agreed.
Quoted 1 line
5) Given a pidfd in NS1, get a pidfd in NS2. That's what
translate_pidQuoted 1 line
Quoted 1 line
is for. My preferred signature for this routine is
translate_pid(int
Quoted 1 line
Quoted 1 line
pidfd, int nsfd) -> pidfd. We don't need two namespace arguments.
Why
Quoted 3 lines
Quoted 1 line
not? Because the pidfd *already* names a single process, uniquely!
Given that people are interested in pids we can't just always return
a
Quoted 1 line
pidfd. That would mean a user would need to do get the pidfd read
from
Quoted 1 line
<pidfd>/stat and then close the pidfd. If you do that for a 100 pids
Quoted 8 lines
Quoted 8 lines
or
Quoted 1 line
more you end up allocating and closing file descriptors constantly
for
Quoted 2 lines
no reason. We can't just debate pids away. So it will also need to be able to yield pids e.g. through a flag argument.
Sure, but that's still not a reason that we should care about pidfds working separately from procfs..
That's unrelated to the point made in the above paragraph. Please note, I said that the pidfd api should work when proc is not available not that they can't be dirfds.
Quoted 2 lines
Agreed. I can't imagine pidfd being anything but a proc pid directory handle. So I am confused what Christian meant. Pidfd *is* a procfs directory fid always. That's what I gathered from his
pidfd_send_signalpatch but let me know if I'm way off in the woods.
(K9 Mail still hasn't learned to wrap lines at 80 it seems. :))
Again, I never said that pidfds should be a directory handle. (Though I would like to point out that one of the original ideas I discussed at LPC was to have something like this to get regular file descriptors instead of dirfds:
https://gist.github.com/brauner/59eec91550c5624c9999eaebd95a70df)
Quoted 2 lines
For my next revision, I am thinking of adding the flag argument Christian mentioned to make
translate_pidreturn ananon_inodeFD which can be used for death status, given a <pid>. Since it is thought thattranslate_pidcan be made to return a pid FD, I think it is ok to have it return a pid status FD for the purposes of the death status as well.
translate_pid() should just return you a pidfd. Having it return a pidfd and a status fd feels like stuffing too much functionality in there. If you're fine with it I'll finish prototyping what I had in mind. As I said in previous mails I'm already working on this.
Would you be ok with prototyping the pidfd_wait() syscall you had in mind? Especially the wait_fd part that you want to have I would like to see how that is supposed to work, e.g. who is allowed to wait on the process and how notifications will work for non-parent processes and so on. I feel we won't get anywhere by talking in the abstrace and other people are far more likely to review/comment once there's actual code.
Christian
Re: pidfd design
Quoted 205 lines · Christian Brauner
Quoted 204 lines · Joel Fernandes
Quoted 201 lines · Daniel Colascione
Quoted 3 lines · Christian Brauner
Quoted 1 line · Daniel Colascione
On Tue, Mar 19, 2019 at 6:52 PM Joel Fernandes
Quoted 2 lines · <[email protected]> wrote:
Quoted 2 lines
Quoted 2 lines
On Wed, Mar 20, 2019 at 12:10:23AM +0100, Christian Brauner
Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
On Tue, Mar 19, 2019 at 03:48:32PM -0700, Daniel Colascione
Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
On Tue, Mar 19, 2019 at 3:14 PM Christian Brauner
Quoted 1 line · <[email protected]> wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
So I dislike the idea of allocating new inodes from the
procfs super
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
block. I would like to avoid pinning the whole pidfd
concept exclusively
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to proc. The idea is that the pidfd API will be useable
through procfs
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
via
open("/proc/<pid>")because that is what users expectand really
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
wanted to have for a long time. So it makes sense to have
this working.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
But it should really be useable without it. That's why
translate_pid()Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
and
pidfd_clone()are on the table. What I'm saying is,once the pidfd
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
api is "complete" you should be able to set
CONFIG_PROCFS=N- even
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
though that's crazy - and still be able to use pidfds. This
is also a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
point akpm asked about when I did the
pidfd_send_signalwork.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I agree that you shouldn't need
CONFIG_PROCFS=Yto usepidfds. One
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
crazy idea that I was discussing with Joel the other day is
to just
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
make
CONFIG_PROCFS=Ymandatory and provide a new
get_procfs_root()Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
system call that returned, out of thin air and independent of
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
mount table, a procfs root directory file descriptor for the
caller's
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
PID namspace and suitable for use with openat(2).
Even if this works I'm pretty sure that Al and a lot of others
will not
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
be happy about this. A syscall to get an fd to /proc?
Why not? procfs provides access to a lot of core kernel
functionality.
Quoted 4 lines
Quoted 4 lines
Why should you need a mountpoint to get to it?
Quoted 2 lines
That's not going
Quoted 1 line
to happen and I don't see the need for a separate syscall just
for that.
Quoted 2 lines
Quoted 2 lines
We need a system call for the same reason we need a getrandom(2):
you
Quoted 6 lines
Quoted 6 lines
have to bootstrap somehow when you're in a minimal environment.
Quoted 1 line
Quoted 1 line
(I do see the point of making
CONFIG_PROCFS=ythe default btw.)I'm not proposing that we make
CONFIG_PROCFS=ythe default. I'm proposing that we *hardwire* it as the default and just declarethat
Quoted 1 line
Quoted 1 line
it's not possible to build a Linux kernel that doesn't include
procfs.
Quoted 3 lines
Quoted 3 lines
Why do we even have that button?
Quoted 1 line
I think his point here was that he wanted a handle to procfs no
matter where
Quoted 1 line
Quoted 1 line
Quoted 1 line
it was mounted and then can later use openat on that. Agreed that
it may be
Quoted 1 line
Quoted 1 line
Quoted 1 line
unnecessary unless there is a usecase for it, and especially if
the /proc
Quoted 1 line
Quoted 1 line
Quoted 1 line
directory being the defacto mountpoint for procfs is a universal
convention.
Quoted 2 lines
Quoted 2 lines
If it's a universal convention and, in practice, everyone needs
proc
Quoted 1 line
Quoted 1 line
mounted anyway, so what's the harm in hardwiring
CONFIG_PROCFS=y?If
Quoted 1 line
Quoted 1 line
we advertise /proc as not merely some kind of optional debug
interface
Quoted 3 lines
Quoted 3 lines
but *the* way certain kernel features are exposed --- and there's nothing wrong with that --- then we should give programs access to these core kernel features in a way that doesn't depend on
userspace
Quoted 2 lines
Quoted 2 lines
kernel configuration, and you do that by either providing a procfs-root-getting system call or just hardwiring the "
/proc/"prefix
Quoted 3 lines
Quoted 3 lines
into VFS.
Quoted 1 line
Quoted 1 line
Inode allocation from the procfs mount for the file descriptors
Joel
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
wants is not correct. Their not really procfs file descriptors
so this
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
is a nack. We can't just hook into proc that way.
I was not particular about using procfs mount for the FDs but
that's the only
Quoted 1 line
Quoted 1 line
Quoted 1 line
way I knew how to do it until you pointed out
anon_inode(my grepskills
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
missed that), so thank you!
Quoted 1 line
Quoted 1 line
C'mon: /proc is used by everyone today and almost every
program breaks
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
if it's not around. The string "/proc" is already de facto
kernel ABI.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Let's just drop the pretense of /proc being optional and bake
it into
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the kernel proper, then give programs a way to get to /proc
that isn't
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
tied to any particular mount configuration. This way, we
don't need a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
translate_pid(), since callers can just use procfs to do thesame
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
thing. (That is, if I understand correctly what
translate_piddoes.)
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I'm not sure what you think
translate_pid()is doing sinceyou're not
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
saying what you think it does.
Examples from the old patchset:translate_pid(pid, ns, -1)- get pid in our pid namespaceAh, it's a bit different from what I had in mind. It's fair to want
to
Quoted 2 lines
Quoted 2 lines
translate PIDs between namespaces, but the only way to make the
translate_pidunder discussion robust is to have it accept andproduce
Quoted 1 line
Quoted 1 line
pidfds. (At that point, you might as well call it
translate_pidfd.)We
Quoted 1 line
Quoted 1 line
should not be adding new APIs to the kernel that accept numeric
PIDs:
Quoted 3 lines
The traditional pid-based api is not going away. There are users that have the requirement to translate pids between namespaces and also
doing
Quoted 8 lines
introspection on these namespaces independent of pidfds. We will not restrict the usefulness of this syscall by making it only work with pidfds.
Quoted 2 lines
it's not possible to use these APIs correctly except under very limited circumstances --- mostly, talking about init or a parent
The pid-based api is one of the most widely used apis of the kernel
and
Quoted 1 line
people have been using it quite successfully for a long time. Yes,
it's
Quoted 5 lines
rac, but it's here to stay.
Quoted 3 lines
talking about its child.
Really, we need a few related operations, and we shouldn't
necessarily
Quoted 8 lines
Quoted 1 line
mingle them.
Yes, we've established that previously.
Quoted 3 lines
1) Given a numeric PID, give me a pidfd: that works today: you just open /proc/<pid>
Quoted 11 lines
Agreed.
Quoted 4 lines
2) Given a pidfd, give me a numeric PID: that works today: you just
openat(pidfd, "stat", O_RDONLY)and read the first token (which is always the numeric PID).Agreed.
Quoted 2 lines
3) Given a pidfd, send a signal: that's what
pidfd_send_signaldoes,
Quoted 5 lines
Quoted 1 line
and it's a good start on the rest of these operations.
Agreed.
Quoted 1 line
5) Given a pidfd in NS1, get a pidfd in NS2. That's what
translate_pidQuoted 1 line
Quoted 1 line
is for. My preferred signature for this routine is
translate_pid(int
Quoted 1 line
Quoted 1 line
pidfd, int nsfd) -> pidfd. We don't need two namespace arguments.
Why
Quoted 3 lines
Quoted 1 line
not? Because the pidfd *already* names a single process, uniquely!
Given that people are interested in pids we can't just always return
a
Quoted 1 line
pidfd. That would mean a user would need to do get the pidfd read
from
Quoted 1 line
<pidfd>/stat and then close the pidfd. If you do that for a 100 pids
or
Quoted 1 line
more you end up allocating and closing file descriptors constantly
for
Quoted 2 lines
no reason. We can't just debate pids away. So it will also need to be able to yield pids e.g. through a flag argument.
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Sure, but that's still not a reason that we should care about pidfds working separately from procfs..
That's unrelated to the point made in the above paragraph. Please note, I said that the pidfd api should work when proc is not available not that they can't be dirfds.
What do you mean by "not available"? CONFIG_PROCFS=n? If pidfds supposed to work when proc is unavailable yet also be directory FDs, to what directory should the FD refer? As I mentioned in my previous message, trying to make pidfd work without CONFIG_PROCFS is a very bad idea.
Quoted 11 lines
Quoted 3 lines
Agreed. I can't imagine pidfd being anything but a proc pid directory handle. So I am confused what Christian meant. Pidfd *is* a procfs directory fid always. That's what I gathered from his
pidfd_send_signalpatch but let me know if I'm way off in the woods.(K9 Mail still hasn't learned to wrap lines at 80 it seems. :))
Again, I never said that pidfds should be a directory handle. (Though I would like to point out that one of the original ideas I discussed at LPC was to have something like this to get regular file descriptors instead of dirfds:
https://gist.github.com/brauner/59eec91550c5624c9999eaebd95a70df)
As I mentioned in my original email on this thread, if you have regular file descriptors instead of directory FDs, you have to use some special new API instead of openat to get metadata about a process. That's pointless duplication of functionality considering that a directory FD gives you that information automatically.
Quoted 6 lines
Quoted 1 line
For my next revision, I am thinking of adding the flag argument Christian mentioned to make
translate_pidreturn ananon_inodeFD which can be used for death status, given a <pid>. Since it is thought thattranslate_pidcan be made to return a pid FD, I think it is ok to have it return a pid status FD for the purposes of the death status as well.
translate_pid()should just return you a pidfd. Having it return a pidfd and a status fd feels like stuffing too much functionality in there. If you're fine with it I'll finish prototyping what I had in mind. As I said in previous mails I'm already working on this.
translate_pid also needs to *accept* pidfds, at least optionally. Unless you have a function from pidfd to pidfd, you race.
Christian Brauner
maintainerRe: pidfd design
Quoted 206 lines · Daniel Colascione
Quoted 205 lines · Christian Brauner
Quoted 204 lines · Joel Fernandes
Quoted 201 lines · Daniel Colascione
Quoted 3 lines · Christian Brauner
Quoted 1 line · Daniel Colascione
On Tue, Mar 19, 2019 at 6:52 PM Joel Fernandes
Quoted 2 lines · <[email protected]> wrote:
Quoted 2 lines
Quoted 2 lines
On Wed, Mar 20, 2019 at 12:10:23AM +0100, Christian Brauner
Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
On Tue, Mar 19, 2019 at 03:48:32PM -0700, Daniel Colascione
Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
On Tue, Mar 19, 2019 at 3:14 PM Christian Brauner
Quoted 1 line · <[email protected]> wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
So I dislike the idea of allocating new inodes from the
procfs super
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
block. I would like to avoid pinning the whole pidfd
concept exclusively
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to proc. The idea is that the pidfd API will be useable
through procfs
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
via
open("/proc/<pid>")because that is what users expectand really
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
wanted to have for a long time. So it makes sense to have
this working.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
But it should really be useable without it. That's why
translate_pid()Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
and
pidfd_clone()are on the table. What I'm saying is,once the pidfd
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
api is "complete" you should be able to set
CONFIG_PROCFS=N- even
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
though that's crazy - and still be able to use pidfds. This
is also a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
point akpm asked about when I did the
pidfd_send_signalwork.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I agree that you shouldn't need
CONFIG_PROCFS=Yto usepidfds. One
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
crazy idea that I was discussing with Joel the other day is
to just
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
make
CONFIG_PROCFS=Ymandatory and provide a new
get_procfs_root()Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
system call that returned, out of thin air and independent of
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
mount table, a procfs root directory file descriptor for the
caller's
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
PID namspace and suitable for use with openat(2).
Even if this works I'm pretty sure that Al and a lot of others
will not
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
be happy about this. A syscall to get an fd to /proc?
Why not? procfs provides access to a lot of core kernel
functionality.
Quoted 4 lines
Quoted 4 lines
Why should you need a mountpoint to get to it?
Quoted 2 lines
That's not going
Quoted 1 line
to happen and I don't see the need for a separate syscall just
for that.
Quoted 2 lines
Quoted 2 lines
We need a system call for the same reason we need a getrandom(2):
you
Quoted 6 lines
Quoted 6 lines
have to bootstrap somehow when you're in a minimal environment.
Quoted 1 line
Quoted 1 line
(I do see the point of making
CONFIG_PROCFS=ythe default btw.)I'm not proposing that we make
CONFIG_PROCFS=ythe default. I'm proposing that we *hardwire* it as the default and just declarethat
Quoted 1 line
Quoted 1 line
it's not possible to build a Linux kernel that doesn't include
procfs.
Quoted 3 lines
Quoted 3 lines
Why do we even have that button?
Quoted 1 line
I think his point here was that he wanted a handle to procfs no
matter where
Quoted 1 line
Quoted 1 line
Quoted 1 line
it was mounted and then can later use openat on that. Agreed that
it may be
Quoted 1 line
Quoted 1 line
Quoted 1 line
unnecessary unless there is a usecase for it, and especially if
the /proc
Quoted 1 line
Quoted 1 line
Quoted 1 line
directory being the defacto mountpoint for procfs is a universal
convention.
Quoted 2 lines
Quoted 2 lines
If it's a universal convention and, in practice, everyone needs
proc
Quoted 1 line
Quoted 1 line
mounted anyway, so what's the harm in hardwiring
CONFIG_PROCFS=y?If
Quoted 1 line
Quoted 1 line
we advertise /proc as not merely some kind of optional debug
interface
Quoted 3 lines
Quoted 3 lines
but *the* way certain kernel features are exposed --- and there's nothing wrong with that --- then we should give programs access to these core kernel features in a way that doesn't depend on
userspace
Quoted 2 lines
Quoted 2 lines
kernel configuration, and you do that by either providing a procfs-root-getting system call or just hardwiring the "
/proc/"prefix
Quoted 3 lines
Quoted 3 lines
into VFS.
Quoted 1 line
Quoted 1 line
Inode allocation from the procfs mount for the file descriptors
Joel
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
wants is not correct. Their not really procfs file descriptors
so this
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
is a nack. We can't just hook into proc that way.
I was not particular about using procfs mount for the FDs but
that's the only
Quoted 1 line
Quoted 1 line
Quoted 1 line
way I knew how to do it until you pointed out
anon_inode(my grepskills
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
missed that), so thank you!
Quoted 1 line
Quoted 1 line
C'mon: /proc is used by everyone today and almost every
program breaks
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
if it's not around. The string "/proc" is already de facto
kernel ABI.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Let's just drop the pretense of /proc being optional and bake
it into
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the kernel proper, then give programs a way to get to /proc
that isn't
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
tied to any particular mount configuration. This way, we
don't need a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
translate_pid(), since callers can just use procfs to do thesame
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
thing. (That is, if I understand correctly what
translate_piddoes.)
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I'm not sure what you think
translate_pid()is doing sinceyou're not
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
saying what you think it does.
Examples from the old patchset:translate_pid(pid, ns, -1)- get pid in our pid namespaceAh, it's a bit different from what I had in mind. It's fair to want
to
Quoted 2 lines
Quoted 2 lines
translate PIDs between namespaces, but the only way to make the
translate_pidunder discussion robust is to have it accept andproduce
Quoted 1 line
Quoted 1 line
pidfds. (At that point, you might as well call it
translate_pidfd.)We
Quoted 1 line
Quoted 1 line
should not be adding new APIs to the kernel that accept numeric
PIDs:
Quoted 3 lines
The traditional pid-based api is not going away. There are users that have the requirement to translate pids between namespaces and also
doing
Quoted 8 lines
introspection on these namespaces independent of pidfds. We will not restrict the usefulness of this syscall by making it only work with pidfds.
Quoted 2 lines
it's not possible to use these APIs correctly except under very limited circumstances --- mostly, talking about init or a parent
The pid-based api is one of the most widely used apis of the kernel
and
Quoted 1 line
people have been using it quite successfully for a long time. Yes,
it's
Quoted 5 lines
rac, but it's here to stay.
Quoted 3 lines
talking about its child.
Really, we need a few related operations, and we shouldn't
necessarily
Quoted 8 lines
Quoted 1 line
mingle them.
Yes, we've established that previously.
Quoted 3 lines
1) Given a numeric PID, give me a pidfd: that works today: you just open /proc/<pid>
Quoted 11 lines
Agreed.
Quoted 4 lines
2) Given a pidfd, give me a numeric PID: that works today: you just
openat(pidfd, "stat", O_RDONLY)and read the first token (which is always the numeric PID).Agreed.
Quoted 2 lines
3) Given a pidfd, send a signal: that's what
pidfd_send_signaldoes,
Quoted 5 lines
Quoted 1 line
and it's a good start on the rest of these operations.
Agreed.
Quoted 1 line
5) Given a pidfd in NS1, get a pidfd in NS2. That's what
translate_pidQuoted 1 line
Quoted 1 line
is for. My preferred signature for this routine is
translate_pid(int
Quoted 1 line
Quoted 1 line
pidfd, int nsfd) -> pidfd. We don't need two namespace arguments.
Why
Quoted 3 lines
Quoted 1 line
not? Because the pidfd *already* names a single process, uniquely!
Given that people are interested in pids we can't just always return
a
Quoted 1 line
pidfd. That would mean a user would need to do get the pidfd read
from
Quoted 1 line
<pidfd>/stat and then close the pidfd. If you do that for a 100 pids
or
Quoted 1 line
more you end up allocating and closing file descriptors constantly
for
Quoted 2 lines
no reason. We can't just debate pids away. So it will also need to be able to yield pids e.g. through a flag argument.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Sure, but that's still not a reason that we should care about pidfds working separately from procfs..
That's unrelated to the point made in the above paragraph. Please note, I said that the pidfd api should work when proc is not available not that they can't be dirfds.
What do you mean by "not available"?
CONFIG_PROCFS=n? If pidfds
I'm talking about the ability to clone processes and get fd handles on them via pidfd_clone() or CLONE_NEWFD.
Quoted 8 lines
Quoted 5 lines
translate_pid()should just return you a pidfd. Having it return a pidfd and a status fd feels like stuffing too much functionality in there. If you're fine with it I'll finish prototyping what I had in mind. As I said in previous mails I'm already working on this.
translate_pidalso needs to *accept* pidfds, at least optionally. Unless you have a function from pidfd to pidfd, you race.
You're misunderstanding. Again, I said in my previous mails it should accept pidfds optionally as arguments, yes. But I don't want it to return the status fds that you previously wanted pidfd_wait() to return. I really want to see Joel's pidfd_wait() patchset and have more people review the actual code.
Andy Lutomirski
maintainerRe: pidfd design
Quoted 6 lines · Christian Brauner
You're misunderstanding. Again, I said in my previous mails it should accept pidfds optionally as arguments, yes. But I don't want it to return the status fds that you previously wanted
pidfd_wait()to return. I really want to see Joel'spidfd_wait()patchset and have more people review the actual code.
Just to make sure that no one is forgetting a material security consideration:
$ ls /proc/self attr exe mountinfo projid_map status autogroup fd mounts root syscall auxv fdinfo mountstats sched task cgroup gid_map net schedstat timers clear_refs io ns sessionid timerslack_ns cmdline latency numa_maps setgroups uid_map comm limits oom_adj smaps wchan coredump_filter loginuid oom_score smaps_rollup cpuset map_files oom_score_adj stack cwd maps pagemap stat environ mem personality statm
A bunch of this stuff makes sense to make accessible through a syscall interface that we expect to be used even in sandboxes. But a bunch of it does not. For example, *_map, mounts, mountstats, and net are all namespace-wide things that certain policies expect to be unavailable. stack, for example, is a potential attack surface. Etc.
As it stands, if you create a fresh userns and mountns and try to mount /proc, there are some really awful and hideous rules that are checked for security reasons. All these new APIs either need to return something more restrictive than a proc dirfd or they need to follow the same rules. And I'm afraid that the latter may be a nonstarter if you expect these APIs to be used in libraries.
Yes, this is unfortunate, but it is indeed the current situation. I suppose that we could return magic restricted dirfds, or we could return things that aren't dirfds and all and have some API that gives you the dirfd associated with a procfd but only if you can see /proc/PID.
--Andy
Christian Brauner
maintainerRe: pidfd design
Quoted 9 lines · Andy Lutomirski
Quoted 6 lines · Christian Brauner
You're misunderstanding. Again, I said in my previous mails it should accept pidfds optionally as arguments, yes. But I don't want it to return the status fds that you previously wanted
pidfd_wait()to return. I really want to see Joel'spidfd_wait()patchset and have more people review the actual code.Just to make sure that no one is forgetting a material security consideration:
Andy, thanks for commenting!
Quoted 27 lines
$ ls /proc/self attr exe mountinfo projid_map status autogroup fd mounts root syscall auxv fdinfo mountstats sched task cgroup gid_map net schedstat timers clear_refs io ns sessionid timerslack_ns cmdline latency numa_maps setgroups uid_map comm limits oom_adj smaps wchan coredump_filter loginuid oom_score smaps_rollup cpuset map_files oom_score_adj stack cwd maps pagemap stat environ mem personality statmA bunch of this stuff makes sense to make accessible through a syscall interface that we expect to be used even in sandboxes. But a bunch of it does not. For example,
*_map, mounts, mountstats, and net are all namespace-wide things that certain policies expect to be unavailable. stack, for example, is a potential attack surface. Etc.As it stands, if you create a fresh userns and mountns and try to mount /proc, there are some really awful and hideous rules that are checked for security reasons. All these new APIs either need to return something more restrictive than a proc dirfd or they need to follow the same rules. And I'm afraid that the latter may be a nonstarter if you expect these APIs to be used in libraries.
Quoted 5 lines
Yes, this is unfortunate, but it is indeed the current situation. I suppose that we could return magic restricted dirfds, or we could return things that aren't dirfds and all and have some API that gives you the dirfd associated with a procfd but only if you can see
/proc/PID.
What would be your opinion to having a
/proc/<pid>/handle
file instead of having a dirfd. Essentially, what I initially proposed at LPC. The change on what we currently have in master would be:
https://gist.github.com/brauner/59eec91550c5624c9999eaebd95a70df
Re: pidfd design
Quoted 10 lines · Christian Brauner
Quoted 8 lines · Andy Lutomirski
Quoted 6 lines · Christian Brauner
You're misunderstanding. Again, I said in my previous mails it should accept pidfds optionally as arguments, yes. But I don't want it to return the status fds that you previously wanted
pidfd_wait()to return. I really want to see Joel'spidfd_wait()patchset and have more people review the actual code.
Quoted 23 lines
Quoted 1 line
Just to make sure that no one is forgetting a material security consideration:
Andy, thanks for commenting!
Quoted 19 lines
$ ls /proc/self attr exe mountinfo projid_map status autogroup fd mounts root syscall auxv fdinfo mountstats sched task cgroup gid_map net schedstat timers clear_refs io ns sessionid timerslack_ns cmdline latency numa_maps setgroups uid_map comm limits oom_adj smaps wchan coredump_filter loginuid oom_score smaps_rollup cpuset map_files oom_score_adj stack cwd maps pagemap stat environ mem personality statmA bunch of this stuff makes sense to make accessible through a syscall interface that we expect to be used even in sandboxes. But a bunch of it does not. For example,
*_map, mounts, mountstats, and net are all namespace-wide things that certain policies expect to be unavailable. stack, for example, is a potential attack surface. Etc.
If you can access these files sources via open(2) on /proc/<pid>, you should be able to access them via a pidfd. If you can't, you shouldn't. Which /proc? The one you'd get by mounting procfs. I don't see how pidfd makes any material changes to anyone's security. As far as I'm concerned, if a sandbox can't mount /proc at all, it's just a broken and unsupported configuration.
An actual threat model and real thought paid to access capabilities would help. Almost everything around the interaction of Linux kernel namespaces and security feels like a jumble of ad-hoc patches added as afterthoughts in response to random objections.
Quoted 3 lines
Quoted 3 lines
All these new APIs either need to
return something more restrictive than a proc dirfd or they need to follow the same rules.
What's wrong with the latter?
Quoted 2 lines
Quoted 2 lines
And I'm afraid that the latter may be a
nonstarter if you expect these APIs to be used in libraries.
What's special about libraries? How is a library any worse-off using openat(2) on a pidfd than it would be just opening the file called "/proc/$apid"?
Quoted 11 lines
Quoted 5 lines
Yes, this is unfortunate, but it is indeed the current situation. I suppose that we could return magic restricted dirfds, or we could return things that aren't dirfds and all and have some API that gives you the dirfd associated with a procfd but only if you can see
/proc/PID.What would be your opinion to having a
/proc/<pid>/handle
file instead of having a dirfd. Essentially, what I initially proposed at LPC. The change on what we currently have in master would be:
https://gist.github.com/brauner/59eec91550c5624c9999eaebd95a70df
And how do you propose, given one of these handle objects, getting a process's current priority, or its current oom score, or its list of memory maps? As I mentioned in my original email, and which nobody has addressed, if you don't use a dirfd as your process handle or you don't provide an easy way to get one of these proc directory FDs, you need to duplicate a lot of metadata access interfaces.
Andy Lutomirski
maintainerRe: pidfd design
Quoted 36 lines · Daniel Colascione
Quoted 10 lines · Christian Brauner
Quoted 8 lines · Andy Lutomirski
Quoted 6 lines · Christian Brauner
You're misunderstanding. Again, I said in my previous mails it should accept pidfds optionally as arguments, yes. But I don't want it to return the status fds that you previously wanted
pidfd_wait()to return. I really want to see Joel'spidfd_wait()patchset and have more people review the actual code.Quoted 23 lines
Quoted 1 line
Just to make sure that no one is forgetting a material security consideration:
Andy, thanks for commenting!
Quoted 19 lines
$ ls /proc/self attr exe mountinfo projid_map status autogroup fd mounts root syscall auxv fdinfo mountstats sched task cgroup gid_map net schedstat timers clear_refs io ns sessionid timerslack_ns cmdline latency numa_maps setgroups uid_map comm limits oom_adj smaps wchan coredump_filter loginuid oom_score smaps_rollup cpuset map_files oom_score_adj stack cwd maps pagemap stat environ mem personality statmA bunch of this stuff makes sense to make accessible through a syscall interface that we expect to be used even in sandboxes. But a bunch of it does not. For example,
*_map, mounts, mountstats, and net are all namespace-wide things that certain policies expect to be unavailable. stack, for example, is a potential attack surface. Etc.
Quoted 6 lines
If you can access these files sources via open(2) on /proc/<pid>, you should be able to access them via a pidfd. If you can't, you shouldn't. Which /proc? The one you'd get by mounting procfs. I don't see how pidfd makes any material changes to anyone's security. As far as I'm concerned, if a sandbox can't mount /proc at all, it's just a broken and unsupported configuration.
It's not "broken and unsupported". I know of an actual working, deployed container-ish sandbox that does exactly this. I would also guess that quite a few not-at-all-container-like sandboxes work like this. (The obvious seccomp + unshare + pivot_root deny-myself-access-to-lots-of-things trick results in no /proc, which is by dsign.)
Quoted 5 lines
An actual threat model and real thought paid to access capabilities would help. Almost everything around the interaction of Linux kernel namespaces and security feels like a jumble of ad-hoc patches added as afterthoughts in response to random objections.
I fully agree. But if you start thinking for real about access capabilities, there's no way that you're going to conclude that a capability to access some process implies a capability to access the settings of its network namespace.
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
All these new APIs either need to
return something more restrictive than a proc dirfd or they need to follow the same rules.
...
Quoted 3 lines
What's special about libraries? How is a library any worse-off using openat(2) on a pidfd than it would be just opening the file called "
/proc/$apid"?
Because most libraries actually work, right now, without /proc. Even libraries that spawn subprocesses. If we make the new API have the property that it doesn't work if you're in a non-root user namespace and /proc isn't mounted, the result will be an utter mess.
Quoted 13 lines
Quoted 7 lines
Quoted 6 lines
Yes, this is unfortunate, but it is indeed the current situation. I suppose that we could return magic restricted dirfds, or we could return things that aren't dirfds and all and have some API that gives you the dirfd associated with a procfd but only if you can see
/proc/PID.Quoted 5 lines
What would be your opinion to having a
/proc/<pid>/handle
file instead of having a dirfd. Essentially, what I initially proposed at LPC. The change on what we currently have in master would be:
https://gist.github.com/brauner/59eec91550c5624c9999eaebd95a70df
Quoted 6 lines
And how do you propose, given one of these handle objects, getting a process's current priority, or its current oom score, or its list of memory maps? As I mentioned in my original email, and which nobody has addressed, if you don't use a dirfd as your process handle or you don't provide an easy way to get one of these proc directory FDs, you need to duplicate a lot of metadata access interfaces.
An API that takes a process handle object and an fd pointing at /proc (the root of the proc fs) and gives you back a proc dirfd would do the trick. You could do this with no new kernel features at all if you're willing to read the pid, call openat(2), and handle the races in user code.
Re: pidfd design
Quoted 97 lines · Andy Lutomirski
Quoted 34 lines · Daniel Colascione
Quoted 9 lines · Christian Brauner
Quoted 8 lines · Andy Lutomirski
Quoted 6 lines · Christian Brauner
You're misunderstanding. Again, I said in my previous mails it should accept pidfds optionally as arguments, yes. But I don't want it to return the status fds that you previously wanted
pidfd_wait()to return. I really want to see Joel'spidfd_wait()patchset and have more people review the actual code.Quoted 23 lines
Quoted 1 line
Just to make sure that no one is forgetting a material security consideration:
Andy, thanks for commenting!
Quoted 19 lines
$ ls /proc/self attr exe mountinfo projid_map status autogroup fd mounts root syscall auxv fdinfo mountstats sched task cgroup gid_map net schedstat timers clear_refs io ns sessionid timerslack_ns cmdline latency numa_maps setgroups uid_map comm limits oom_adj smaps wchan coredump_filter loginuid oom_score smaps_rollup cpuset map_files oom_score_adj stack cwd maps pagemap stat environ mem personality statmA bunch of this stuff makes sense to make accessible through a syscall interface that we expect to be used even in sandboxes. But a bunch of it does not. For example,
*_map, mounts, mountstats, and net are all namespace-wide things that certain policies expect to be unavailable. stack, for example, is a potential attack surface. Etc.Quoted 6 lines
If you can access these files sources via open(2) on /proc/<pid>, you should be able to access them via a pidfd. If you can't, you shouldn't. Which /proc? The one you'd get by mounting procfs. I don't see how pidfd makes any material changes to anyone's security. As far as I'm concerned, if a sandbox can't mount /proc at all, it's just a broken and unsupported configuration.
It's not "broken and unsupported". I know of an actual working, deployed container-ish sandbox that does exactly this. I would also guess that quite a few not-at-all-container-like sandboxes work like this. (The obvious seccomp + unshare +
pivot_rootdeny-myself-access-to-lots-of-things trick results in no /proc, which is by dsign.)Quoted 5 lines
An actual threat model and real thought paid to access capabilities would help. Almost everything around the interaction of Linux kernel namespaces and security feels like a jumble of ad-hoc patches added as afterthoughts in response to random objections.
I fully agree. But if you start thinking for real about access capabilities, there's no way that you're going to conclude that a capability to access some process implies a capability to access the settings of its network namespace.
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
All these new APIs either need to
return something more restrictive than a proc dirfd or they need to follow the same rules....
Quoted 3 lines
What's special about libraries? How is a library any worse-off using openat(2) on a pidfd than it would be just opening the file called "
/proc/$apid"?Because most libraries actually work, right now, without /proc. Even libraries that spawn subprocesses. If we make the new API have the property that it doesn't work if you're in a non-root user namespace and /proc isn't mounted, the result will be an utter mess.
Quoted 13 lines
Quoted 7 lines
Quoted 6 lines
Yes, this is unfortunate, but it is indeed the current situation. I suppose that we could return magic restricted dirfds, or we could return things that aren't dirfds and all and have some API that gives you the dirfd associated with a procfd but only if you can see
/proc/PID.Quoted 5 lines
What would be your opinion to having a
/proc/<pid>/handle
file instead of having a dirfd. Essentially, what I initially proposed at LPC. The change on what we currently have in master would be:
https://gist.github.com/brauner/59eec91550c5624c9999eaebd95a70dfQuoted 6 lines
And how do you propose, given one of these handle objects, getting a process's current priority, or its current oom score, or its list of memory maps? As I mentioned in my original email, and which nobody has addressed, if you don't use a dirfd as your process handle or you don't provide an easy way to get one of these proc directory FDs, you need to duplicate a lot of metadata access interfaces.
Quoted 5 lines
An API that takes a process handle object and an fd pointing at /proc (the root of the proc fs) and gives you back a proc dirfd would do the trick. You could do this with no new kernel features at all if you're willing to read the pid, call openat(2), and handle the races in user code.
This seems like something that might be a good fit for two ioctls?
One ioctl on procfs roots to translate pidfds into that procfs, subject to both the normal lookup permission checks and only working if the pidfd has a translation into the procfs:
int proc_root_fd = open("/proc", O_RDONLY);
int proc_dir_fd = ioctl(proc_root_fd, PROC_PIDFD_TO_PROCFSFD, pidfd);
And one ioctl on procfs directories to translate from PGIDs and PIDs to pidfds:
int proc_pgid_fd = open("/proc/self", O_RDONLY);
int self_pg_pidfd = ioctl(proc_pgid_fd, PROC_PROCFSFD_TO_PIDFD, 0); int proc_pid_fd = open("/proc/thread-self", O_RDONLY); int self_p_pidfd = ioctl(proc_pid_fd, PROC_PROCFSFD_TO_PIDFD, 0);
And then, as you proposed, the new sys_clone() can just return a pidfd, and you can convert it into a procfs fd yourself if you want.
Re: pidfd design
Quoted 99 lines · Jann Horn
Quoted 97 lines · Andy Lutomirski
Quoted 34 lines · Daniel Colascione
Quoted 9 lines · Christian Brauner
Quoted 8 lines · Andy Lutomirski
Quoted 6 lines · Christian Brauner
You're misunderstanding. Again, I said in my previous mails it should accept pidfds optionally as arguments, yes. But I don't want it to return the status fds that you previously wanted
pidfd_wait()to return. I really want to see Joel'spidfd_wait()patchset and have more people review the actual code.Quoted 23 lines
Quoted 1 line
Just to make sure that no one is forgetting a material security consideration:
Andy, thanks for commenting!
Quoted 19 lines
$ ls /proc/self attr exe mountinfo projid_map status autogroup fd mounts root syscall auxv fdinfo mountstats sched task cgroup gid_map net schedstat timers clear_refs io ns sessionid timerslack_ns cmdline latency numa_maps setgroups uid_map comm limits oom_adj smaps wchan coredump_filter loginuid oom_score smaps_rollup cpuset map_files oom_score_adj stack cwd maps pagemap stat environ mem personality statmA bunch of this stuff makes sense to make accessible through a syscall interface that we expect to be used even in sandboxes. But a bunch of it does not. For example,
*_map, mounts, mountstats, and net are all namespace-wide things that certain policies expect to be unavailable. stack, for example, is a potential attack surface. Etc.Quoted 6 lines
If you can access these files sources via open(2) on /proc/<pid>, you should be able to access them via a pidfd. If you can't, you shouldn't. Which /proc? The one you'd get by mounting procfs. I don't see how pidfd makes any material changes to anyone's security. As far as I'm concerned, if a sandbox can't mount /proc at all, it's just a broken and unsupported configuration.
It's not "broken and unsupported". I know of an actual working, deployed container-ish sandbox that does exactly this. I would also guess that quite a few not-at-all-container-like sandboxes work like this. (The obvious seccomp + unshare +
pivot_rootdeny-myself-access-to-lots-of-things trick results in no /proc, which is by dsign.)Quoted 5 lines
An actual threat model and real thought paid to access capabilities would help. Almost everything around the interaction of Linux kernel namespaces and security feels like a jumble of ad-hoc patches added as afterthoughts in response to random objections.
I fully agree. But if you start thinking for real about access capabilities, there's no way that you're going to conclude that a capability to access some process implies a capability to access the settings of its network namespace.
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
All these new APIs either need to
return something more restrictive than a proc dirfd or they need to follow the same rules....
Quoted 3 lines
What's special about libraries? How is a library any worse-off using openat(2) on a pidfd than it would be just opening the file called "
/proc/$apid"?Because most libraries actually work, right now, without /proc. Even libraries that spawn subprocesses. If we make the new API have the property that it doesn't work if you're in a non-root user namespace and /proc isn't mounted, the result will be an utter mess.
Quoted 19 lines
Quoted 7 lines
Quoted 6 lines
Yes, this is unfortunate, but it is indeed the current situation. I suppose that we could return magic restricted dirfds, or we could return things that aren't dirfds and all and have some API that gives you the dirfd associated with a procfd but only if you can see
/proc/PID.Quoted 5 lines
What would be your opinion to having a
/proc/<pid>/handle
file instead of having a dirfd. Essentially, what I initially proposed at LPC. The change on what we currently have in master would be:
https://gist.github.com/brauner/59eec91550c5624c9999eaebd95a70dfAnd how do you propose, given one of these handle objects, getting a process's current priority, or its current oom score, or its list of memory maps? As I mentioned in my original email, and which nobody has addressed, if you don't use a dirfd as your process handle or you don't provide an easy way to get one of these proc directory FDs, you need to duplicate a lot of metadata access interfaces.
Quoted 7 lines
Quoted 5 lines
An API that takes a process handle object and an fd pointing at /proc (the root of the proc fs) and gives you back a proc dirfd would do the trick. You could do this with no new kernel features at all if you're willing to read the pid, call openat(2), and handle the races in user code.
This seems like something that might be a good fit for two ioctls?
As an aside, we had a long discussion about why fundamental facilities like this should be system calls, not ioctls. I think the arguments still apply.
Quoted 9 lines
One ioctl on procfs roots to translate pidfds into that procfs, subject to both the normal lookup permission checks and only working if the pidfd has a translation into the procfs:
int
proc_root_fd=open("/proc", O_RDONLY);
intproc_dir_fd=ioctl(proc_root_fd, PROC_PIDFD_TO_PROCFSFD, pidfd);And one ioctl on procfs directories to translate from PGIDs and PIDs to pidfds:
Quoted 8 lines
int
proc_pgid_fd=open("/proc/self", O_RDONLY);
intself_pg_pidfd=ioctl(proc_pgid_fd, PROC_PROCFSFD_TO_PIDFD, 0); intproc_pid_fd=open("/proc/thread-self", O_RDONLY); intself_p_pidfd=ioctl(proc_pid_fd, PROC_PROCFSFD_TO_PIDFD, 0);And then, as you proposed, the new
sys_clone()can just return a pidfd, and you can convert it into a procfs fd yourself if you want.
I think that's the consensus we reached on the other thread. The O_DIRECTORY open on /proc/self/fd/mypidfd seems like it'd work well enough.
Andy Lutomirski
maintainerRe: pidfd design
Quoted 12 lines · Daniel Colascione
Quoted 10 lines · Jann Horn
On Mon, Mar 25, 2019 at 8:44 PM Andy Lutomirski <[email protected]> wrote:
One ioctl on procfs roots to translate pidfds into that procfs, subject to both the normal lookup permission checks and only working if the pidfd has a translation into the procfs:
int
proc_root_fd=open("/proc", O_RDONLY);
intproc_dir_fd=ioctl(proc_root_fd, PROC_PIDFD_TO_PROCFSFD, pidfd);
Quoted 7 lines
Quoted 7 lines
And one ioctl on procfs directories to translate from PGIDs and PIDs to pidfds:
int
proc_pgid_fd=open("/proc/self", O_RDONLY);
intself_pg_pidfd=ioctl(proc_pgid_fd, PROC_PROCFSFD_TO_PIDFD, 0); intproc_pid_fd=open("/proc/thread-self", O_RDONLY); intself_p_pidfd=ioctl(proc_pid_fd, PROC_PROCFSFD_TO_PIDFD, 0);
This sounds okay to me. Or we could make it so that a procfs directory fd also works as a pidfd, but that seems more likely to be problematic than just allowing two-way translation like this
Quoted 7 lines
Quoted 3 lines
And then, as you proposed, the new
sys_clone()can just return a pidfd, and you can convert it into a procfs fd yourself if you want.I think that's the consensus we reached on the other thread. The
O_DIRECTORYopen on/proc/self/fd/mypidfdseems like it'd work well enough.
I must have missed this particular email.
IMO, if /proc/self/fd/mypidfd allows O_DIRECTORY open to work, then it really ought to do function just like /proc/self/fd/mypidfd/. and /proc/self/fd/mypidfd/status should work. And these latter two options seem nutty.
Also, this O_DIRECTORY thing is missing the entire point of the ioctl interface -- it doesn't require procfs access.
--Andy
Christian Brauner
maintainerRe: pidfd design
Quoted 35 lines · Andy Lutomirski
Quoted 12 lines · Daniel Colascione
Quoted 10 lines · Jann Horn
On Mon, Mar 25, 2019 at 8:44 PM Andy Lutomirski <[email protected]> wrote:
One ioctl on procfs roots to translate pidfds into that procfs, subject to both the normal lookup permission checks and only working if the pidfd has a translation into the procfs:
int
proc_root_fd=open("/proc", O_RDONLY);
intproc_dir_fd=ioctl(proc_root_fd, PROC_PIDFD_TO_PROCFSFD, pidfd);Quoted 7 lines
Quoted 7 lines
And one ioctl on procfs directories to translate from PGIDs and PIDs to pidfds:
int
proc_pgid_fd=open("/proc/self", O_RDONLY);
intself_pg_pidfd=ioctl(proc_pgid_fd, PROC_PROCFSFD_TO_PIDFD, 0); intproc_pid_fd=open("/proc/thread-self", O_RDONLY); intself_p_pidfd=ioctl(proc_pid_fd, PROC_PROCFSFD_TO_PIDFD, 0);This sounds okay to me. Or we could make it so that a procfs directory fd also works as a pidfd, but that seems more likely to be problematic than just allowing two-way translation like this
Quoted 7 lines
Quoted 3 lines
And then, as you proposed, the new
sys_clone()can just return a pidfd, and you can convert it into a procfs fd yourself if you want.I think that's the consensus we reached on the other thread. The
O_DIRECTORYopen on/proc/self/fd/mypidfdseems like it'd work well enough.I must have missed this particular email.
Quoted 7 lines
IMO, if
/proc/self/fd/mypidfdallowsO_DIRECTORYopen to work, then it really ought to do function just like/proc/self/fd/mypidfd/. and/proc/self/fd/mypidfd/statusshould work. And these latter two options seem nutty.Also, this
O_DIRECTORYthing is missing the entire point of the ioctl interface -- it doesn't require procfs access.
The other option was to encode the pid in the callers pid namespace into the pidfd's fdinfo so that you can parse it out and open /proc/<pid>. You'd just need an event on the pidfd to tell you when the process has died. Jonathan and I just discussed this.
Andy Lutomirski
maintainerRe: pidfd design
Quoted 45 lines · Christian Brauner
Quoted 35 lines · Andy Lutomirski
Quoted 12 lines · Daniel Colascione
Quoted 10 lines · Jann Horn
On Mon, Mar 25, 2019 at 8:44 PM Andy Lutomirski <[email protected]> wrote:
One ioctl on procfs roots to translate pidfds into that procfs, subject to both the normal lookup permission checks and only working if the pidfd has a translation into the procfs:
int
proc_root_fd=open("/proc", O_RDONLY);
intproc_dir_fd=ioctl(proc_root_fd, PROC_PIDFD_TO_PROCFSFD, pidfd);Quoted 7 lines
Quoted 7 lines
And one ioctl on procfs directories to translate from PGIDs and PIDs to pidfds:
int
proc_pgid_fd=open("/proc/self", O_RDONLY);
intself_pg_pidfd=ioctl(proc_pgid_fd, PROC_PROCFSFD_TO_PIDFD, 0); intproc_pid_fd=open("/proc/thread-self", O_RDONLY); intself_p_pidfd=ioctl(proc_pid_fd, PROC_PROCFSFD_TO_PIDFD, 0);This sounds okay to me. Or we could make it so that a procfs directory fd also works as a pidfd, but that seems more likely to be problematic than just allowing two-way translation like this
Quoted 7 lines
Quoted 3 lines
And then, as you proposed, the new
sys_clone()can just return a pidfd, and you can convert it into a procfs fd yourself if you want.I think that's the consensus we reached on the other thread. The
O_DIRECTORYopen on/proc/self/fd/mypidfdseems like it'd work well enough.I must have missed this particular email.
Quoted 7 lines
IMO, if
/proc/self/fd/mypidfdallowsO_DIRECTORYopen to work, then it really ought to do function just like/proc/self/fd/mypidfd/. and/proc/self/fd/mypidfd/statusshould work. And these latter two options seem nutty.Also, this
O_DIRECTORYthing is missing the entire point of the ioctl interface -- it doesn't require procfs access.
Quoted 6 lines
The other option was to encode the pid in the callers pid namespace into the pidfd's fdinfo so that you can parse it out and open /proc/<pid>. You'd just need an event on the pidfd to tell you when the process has died. Jonathan and I just discussed this.
From an application developer's POV, the ioctl interface sounds much,
much nicer.
Christian Brauner
maintainerRe: pidfd design
Quoted 46 lines · Andy Lutomirski
Quoted 45 lines · Christian Brauner
Quoted 42 lines · Andy Lutomirski
Quoted 12 lines · Daniel Colascione
Quoted 10 lines · Jann Horn
On Mon, Mar 25, 2019 at 8:44 PM Andy Lutomirski <[email protected]> wrote:
One ioctl on procfs roots to translate pidfds into that procfs, subject to both the normal lookup permission checks and only working if the pidfd has a translation into the procfs:
int
proc_root_fd=open("/proc", O_RDONLY);
intproc_dir_fd=ioctl(proc_root_fd, PROC_PIDFD_TO_PROCFSFD, pidfd);Quoted 7 lines
Quoted 7 lines
And one ioctl on procfs directories to translate from PGIDs and PIDs to pidfds:
int
proc_pgid_fd=open("/proc/self", O_RDONLY);
intself_pg_pidfd=ioctl(proc_pgid_fd, PROC_PROCFSFD_TO_PIDFD, 0); intproc_pid_fd=open("/proc/thread-self", O_RDONLY); intself_p_pidfd=ioctl(proc_pid_fd, PROC_PROCFSFD_TO_PIDFD, 0);This sounds okay to me. Or we could make it so that a procfs directory fd also works as a pidfd, but that seems more likely to be problematic than just allowing two-way translation like this
Quoted 7 lines
Quoted 3 lines
And then, as you proposed, the new
sys_clone()can just return a pidfd, and you can convert it into a procfs fd yourself if you want.I think that's the consensus we reached on the other thread. The
O_DIRECTORYopen on/proc/self/fd/mypidfdseems like it'd work well enough.I must have missed this particular email.
IMO, if
/proc/self/fd/mypidfdallowsO_DIRECTORYopen to work, then it really ought to do function just like/proc/self/fd/mypidfd/. and/proc/self/fd/mypidfd/statusshould work. And these latter two options seem nutty.Also, this
O_DIRECTORYthing is missing the entire point of the ioctl interface -- it doesn't require procfs access.
Quoted 7 lines
Quoted 4 lines
The other option was to encode the pid in the callers pid namespace into the pidfd's fdinfo so that you can parse it out and open /proc/<pid>. You'd just need an event on the pidfd to tell you when the process has died. Jonathan and I just discussed this.
From an application developer's POV, the ioctl interface sounds much, much nicer.
Some people are strongly against ioctl()s some don't. I'm not against them so both options are fine with me if people can agree.
Christian
Andy Lutomirski
maintainerRe: pidfd design
Quoted 53 lines · Christian Brauner
Quoted 51 lines · Andy Lutomirski
Quoted 49 lines · Christian Brauner
Quoted 35 lines · Andy Lutomirski
Quoted 12 lines · Daniel Colascione
Quoted 10 lines · Jann Horn
On Mon, Mar 25, 2019 at 8:44 PM Andy Lutomirski <[email protected]> wrote:
One ioctl on procfs roots to translate pidfds into that procfs, subject to both the normal lookup permission checks and only working if the pidfd has a translation into the procfs:
int
proc_root_fd=open("/proc", O_RDONLY);
intproc_dir_fd=ioctl(proc_root_fd, PROC_PIDFD_TO_PROCFSFD, pidfd);Quoted 7 lines
Quoted 7 lines
And one ioctl on procfs directories to translate from PGIDs and PIDs to pidfds:
int
proc_pgid_fd=open("/proc/self", O_RDONLY);
intself_pg_pidfd=ioctl(proc_pgid_fd, PROC_PROCFSFD_TO_PIDFD, 0); intproc_pid_fd=open("/proc/thread-self", O_RDONLY); intself_p_pidfd=ioctl(proc_pid_fd, PROC_PROCFSFD_TO_PIDFD, 0);This sounds okay to me. Or we could make it so that a procfs directory fd also works as a pidfd, but that seems more likely to be problematic than just allowing two-way translation like this
Quoted 7 lines
Quoted 3 lines
And then, as you proposed, the new
sys_clone()can just return a pidfd, and you can convert it into a procfs fd yourself if you want.I think that's the consensus we reached on the other thread. The
O_DIRECTORYopen on/proc/self/fd/mypidfdseems like it'd work well enough.I must have missed this particular email.
Quoted 7 lines
IMO, if
/proc/self/fd/mypidfdallowsO_DIRECTORYopen to work, then it really ought to do function just like/proc/self/fd/mypidfd/. and/proc/self/fd/mypidfd/statusshould work. And these latter two options seem nutty.Also, this
O_DIRECTORYthing is missing the entire point of the ioctl interface -- it doesn't require procfs access.The other option was to encode the pid in the callers pid namespace into the pidfd's fdinfo so that you can parse it out and open /proc/<pid>. You'd just need an event on the pidfd to tell you when the process has died. Jonathan and I just discussed this.
Quoted 6 lines
Quoted 2 lines
From an application developer's POV, the ioctl interface sounds much, much nicer.
Some people are strongly against ioctl()s some don't. I'm not against them so both options are fine with me if people can agree.
There are certainly non-ioctl equivalents that are functionally equivalent. For example, there could be a syscall procfs_open_pidfd(procfs_fd, pid_fd). I personally don't really mind ioctl() when it's really an operation on an fd.
Christian Brauner
maintainerRe: pidfd design
Quoted 57 lines · Andy Lutomirski
Quoted 56 lines · Christian Brauner
Quoted 53 lines · Andy Lutomirski
Quoted 42 lines · Christian Brauner
Quoted 40 lines · Andy Lutomirski
Quoted 12 lines · Daniel Colascione
Quoted 10 lines · Jann Horn
On Mon, Mar 25, 2019 at 8:44 PM Andy Lutomirski <[email protected]> wrote:
One ioctl on procfs roots to translate pidfds into that procfs, subject to both the normal lookup permission checks and only working if the pidfd has a translation into the procfs:
int
proc_root_fd=open("/proc", O_RDONLY);
intproc_dir_fd=ioctl(proc_root_fd, PROC_PIDFD_TO_PROCFSFD, pidfd);Quoted 7 lines
Quoted 7 lines
And one ioctl on procfs directories to translate from PGIDs and PIDs to pidfds:
int
proc_pgid_fd=open("/proc/self", O_RDONLY);
intself_pg_pidfd=ioctl(proc_pgid_fd, PROC_PROCFSFD_TO_PIDFD, 0); intproc_pid_fd=open("/proc/thread-self", O_RDONLY); intself_p_pidfd=ioctl(proc_pid_fd, PROC_PROCFSFD_TO_PIDFD, 0);This sounds okay to me. Or we could make it so that a procfs directory fd also works as a pidfd, but that seems more likely to be problematic than just allowing two-way translation like this
Quoted 7 lines
Quoted 3 lines
And then, as you proposed, the new
sys_clone()can just return a pidfd, and you can convert it into a procfs fd yourself if you want.I think that's the consensus we reached on the other thread. The
O_DIRECTORYopen on/proc/self/fd/mypidfdseems like it'd work well enough.I must have missed this particular email.
IMO, if
/proc/self/fd/mypidfdallowsO_DIRECTORYopen to work, then it really ought to do function just like/proc/self/fd/mypidfd/. and/proc/self/fd/mypidfd/statusshould work. And these latter two options seem nutty.Quoted 7 lines
Quoted 2 lines
Also, this
O_DIRECTORYthing is missing the entire point of the ioctl interface -- it doesn't require procfs access.The other option was to encode the pid in the callers pid namespace into the pidfd's fdinfo so that you can parse it out and open /proc/<pid>. You'd just need an event on the pidfd to tell you when the process has died. Jonathan and I just discussed this.
From an application developer's POV, the ioctl interface sounds much, much nicer.
Quoted 8 lines
Quoted 3 lines
Some people are strongly against ioctl()s some don't. I'm not against them so both options are fine with me if people can agree.
There are certainly non-ioctl equivalents that are functionally equivalent. For example, there could be a syscall
procfs_open_pidfd(procfs_fd, pid_fd). I personally don't really mindioctl()when it's really an operation on an fd.
I totally missed that mail somehow.
Yes, I agree that an ioctl() makes sense for that.
Re: pidfd design
In reply to Christian BraunerOn Wed, Mar 20, 2019 at 07:51:57PM +0100, Christian Brauner wrote:
[snip]
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
translate_pid()should just return you a pidfd. Having it return a pidfd and a status fd feels like stuffing too much functionality in there. If you're fine with it I'll finish prototyping what I had in mind. As I said in previous mails I'm already working on this.
Quoted 8 lines
Quoted 2 lines
translate_pidalso needs to *accept* pidfds, at least optionally. Unless you have a function from pidfd to pidfd, you race.You're misunderstanding. Again, I said in my previous mails it should accept pidfds optionally as arguments, yes. But I don't want it to return the status fds that you previously wanted
pidfd_wait()to return. I really want to see Joel'spidfd_wait()patchset and have more people review the actual code.
No problem, pidfd_wait is also fine with me and we can change it later to translate_pid or something else if needed.
Agreed that lets get to some code writing now that (I hope) we are all on the same page and discuss on actual code.
- Joel
Re: pidfd design
In reply to Christian BraunerQuoted 211 lines · Christian Brauner
Quoted 209 lines · Daniel Colascione
Quoted 208 lines · Christian Brauner
Quoted 206 lines · Joel Fernandes
Quoted 203 lines · Daniel Colascione
Quoted 3 lines · Christian Brauner
Quoted 1 line · Daniel Colascione
On Tue, Mar 19, 2019 at 6:52 PM Joel Fernandes
Quoted 2 lines · <[email protected]> wrote:
Quoted 2 lines
Quoted 2 lines
On Wed, Mar 20, 2019 at 12:10:23AM +0100, Christian Brauner
Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
On Tue, Mar 19, 2019 at 03:48:32PM -0700, Daniel Colascione
Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
On Tue, Mar 19, 2019 at 3:14 PM Christian Brauner
Quoted 1 line · <[email protected]> wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
So I dislike the idea of allocating new inodes from the
procfs super
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
block. I would like to avoid pinning the whole pidfd
concept exclusively
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to proc. The idea is that the pidfd API will be useable
through procfs
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
via
open("/proc/<pid>")because that is what users expectand really
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
wanted to have for a long time. So it makes sense to have
this working.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
But it should really be useable without it. That's why
translate_pid()Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
and
pidfd_clone()are on the table. What I'm saying is,once the pidfd
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
api is "complete" you should be able to set
CONFIG_PROCFS=N- even
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
though that's crazy - and still be able to use pidfds. This
is also a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
point akpm asked about when I did the
pidfd_send_signalwork.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I agree that you shouldn't need
CONFIG_PROCFS=Yto usepidfds. One
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
crazy idea that I was discussing with Joel the other day is
to just
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
make
CONFIG_PROCFS=Ymandatory and provide a new
get_procfs_root()Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
system call that returned, out of thin air and independent of
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
mount table, a procfs root directory file descriptor for the
caller's
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
PID namspace and suitable for use with openat(2).
Even if this works I'm pretty sure that Al and a lot of others
will not
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
be happy about this. A syscall to get an fd to /proc?
Why not? procfs provides access to a lot of core kernel
functionality.
Quoted 4 lines
Quoted 4 lines
Why should you need a mountpoint to get to it?
Quoted 2 lines
That's not going
Quoted 1 line
to happen and I don't see the need for a separate syscall just
for that.
Quoted 2 lines
Quoted 2 lines
We need a system call for the same reason we need a getrandom(2):
you
Quoted 6 lines
Quoted 6 lines
have to bootstrap somehow when you're in a minimal environment.
Quoted 1 line
Quoted 1 line
(I do see the point of making
CONFIG_PROCFS=ythe default btw.)I'm not proposing that we make
CONFIG_PROCFS=ythe default. I'm proposing that we *hardwire* it as the default and just declarethat
Quoted 1 line
Quoted 1 line
it's not possible to build a Linux kernel that doesn't include
procfs.
Quoted 3 lines
Quoted 3 lines
Why do we even have that button?
Quoted 1 line
I think his point here was that he wanted a handle to procfs no
matter where
Quoted 1 line
Quoted 1 line
Quoted 1 line
it was mounted and then can later use openat on that. Agreed that
it may be
Quoted 1 line
Quoted 1 line
Quoted 1 line
unnecessary unless there is a usecase for it, and especially if
the /proc
Quoted 1 line
Quoted 1 line
Quoted 1 line
directory being the defacto mountpoint for procfs is a universal
convention.
Quoted 2 lines
Quoted 2 lines
If it's a universal convention and, in practice, everyone needs
proc
Quoted 1 line
Quoted 1 line
mounted anyway, so what's the harm in hardwiring
CONFIG_PROCFS=y?If
Quoted 1 line
Quoted 1 line
we advertise /proc as not merely some kind of optional debug
interface
Quoted 3 lines
Quoted 3 lines
but *the* way certain kernel features are exposed --- and there's nothing wrong with that --- then we should give programs access to these core kernel features in a way that doesn't depend on
userspace
Quoted 2 lines
Quoted 2 lines
kernel configuration, and you do that by either providing a procfs-root-getting system call or just hardwiring the "
/proc/"prefix
Quoted 3 lines
Quoted 3 lines
into VFS.
Quoted 1 line
Quoted 1 line
Inode allocation from the procfs mount for the file descriptors
Joel
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
wants is not correct. Their not really procfs file descriptors
so this
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
is a nack. We can't just hook into proc that way.
I was not particular about using procfs mount for the FDs but
that's the only
Quoted 1 line
Quoted 1 line
Quoted 1 line
way I knew how to do it until you pointed out
anon_inode(my grepskills
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
missed that), so thank you!
Quoted 1 line
Quoted 1 line
C'mon: /proc is used by everyone today and almost every
program breaks
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
if it's not around. The string "/proc" is already de facto
kernel ABI.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Let's just drop the pretense of /proc being optional and bake
it into
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the kernel proper, then give programs a way to get to /proc
that isn't
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
tied to any particular mount configuration. This way, we
don't need a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
translate_pid(), since callers can just use procfs to do thesame
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
thing. (That is, if I understand correctly what
translate_piddoes.)
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I'm not sure what you think
translate_pid()is doing sinceyou're not
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
saying what you think it does.
Examples from the old patchset:translate_pid(pid, ns, -1)- get pid in our pid namespaceAh, it's a bit different from what I had in mind. It's fair to want
to
Quoted 2 lines
Quoted 2 lines
translate PIDs between namespaces, but the only way to make the
translate_pidunder discussion robust is to have it accept andproduce
Quoted 1 line
Quoted 1 line
pidfds. (At that point, you might as well call it
translate_pidfd.)We
Quoted 1 line
Quoted 1 line
should not be adding new APIs to the kernel that accept numeric
PIDs:
Quoted 3 lines
The traditional pid-based api is not going away. There are users that have the requirement to translate pids between namespaces and also
doing
Quoted 8 lines
introspection on these namespaces independent of pidfds. We will not restrict the usefulness of this syscall by making it only work with pidfds.
Quoted 2 lines
it's not possible to use these APIs correctly except under very limited circumstances --- mostly, talking about init or a parent
The pid-based api is one of the most widely used apis of the kernel
and
Quoted 1 line
people have been using it quite successfully for a long time. Yes,
it's
Quoted 5 lines
rac, but it's here to stay.
Quoted 3 lines
talking about its child.
Really, we need a few related operations, and we shouldn't
necessarily
Quoted 8 lines
Quoted 1 line
mingle them.
Yes, we've established that previously.
Quoted 3 lines
1) Given a numeric PID, give me a pidfd: that works today: you just open /proc/<pid>
Quoted 11 lines
Agreed.
Quoted 4 lines
2) Given a pidfd, give me a numeric PID: that works today: you just
openat(pidfd, "stat", O_RDONLY)and read the first token (which is always the numeric PID).Agreed.
Quoted 2 lines
3) Given a pidfd, send a signal: that's what
pidfd_send_signaldoes,
Quoted 5 lines
Quoted 1 line
and it's a good start on the rest of these operations.
Agreed.
Quoted 1 line
5) Given a pidfd in NS1, get a pidfd in NS2. That's what
translate_pidQuoted 1 line
Quoted 1 line
is for. My preferred signature for this routine is
translate_pid(int
Quoted 1 line
Quoted 1 line
pidfd, int nsfd) -> pidfd. We don't need two namespace arguments.
Why
Quoted 3 lines
Quoted 1 line
not? Because the pidfd *already* names a single process, uniquely!
Given that people are interested in pids we can't just always return
a
Quoted 1 line
pidfd. That would mean a user would need to do get the pidfd read
from
Quoted 1 line
<pidfd>/stat and then close the pidfd. If you do that for a 100 pids
or
Quoted 1 line
more you end up allocating and closing file descriptors constantly
for
Quoted 2 lines
no reason. We can't just debate pids away. So it will also need to be able to yield pids e.g. through a flag argument.
Sure, but that's still not a reason that we should care about pidfds working separately from procfs..
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
That's unrelated to the point made in the above paragraph. Please note, I said that the pidfd api should work when proc is not available not that they can't be dirfds.
What do you mean by "not available"?
CONFIG_PROCFS=n? If pidfdsI'm talking about the ability to clone processes and get fd handles on them via
pidfd_clone()orCLONE_NEWFD.
I wouldn't call that situation "proc [not being] available". We need pidfd_clone to return a pidfd for atomicity reasons, not /proc availability reasons. Again, it doesn't make any sense to support this stuff when CONFIG_PROCFS=n, and CONFIG_PROCFS=n shouldn't even be a supported configuration.
Quoted 11 lines
Quoted 7 lines
Quoted 4 lines
translate_pid()should just return you a pidfd. Having it return a pidfd and a status fd feels like stuffing too much functionality in there. If you're fine with it I'll finish prototyping what I had in mind. As I said in previous mails I'm already working on this.
translate_pidalso needs to *accept* pidfds, at least optionally. Unless you have a function from pidfd to pidfd, you race.You're misunderstanding. Again, I said in my previous mails it should accept pidfds optionally as arguments, yes. But I don't want it to return the status fds that you previously wanted
pidfd_wait()to return.
Agreed. There should be a different way to get these wait handle FDs.
Quoted 2 lines
I really want to see Joel's
pidfd_wait()patchset and have more people review the actual code.
Sure. But it's also unpleasant to have people write code and then have to throw it away due to guessing incorrectly about unclear requirements.
Re: pidfd design
Quoted 7 lines · Daniel Colascione
Quoted 2 lines · Christian Brauner
I really want to see Joel's
pidfd_wait()patchset and have more people review the actual code.Sure. But it's also unpleasant to have people write code and then have to throw it away due to guessing incorrectly about unclear requirements.
No, it is not. It is not unpleasant. And it is useful. It is the best way to identify and resolve those incorrect guesses and unclear requirements.
Re: pidfd design
Quoted 6 lines · Serge E. Hallyn
Quoted 4 lines · Daniel Colascione
Quoted 2 lines · Christian Brauner
I really want to see Joel's
pidfd_wait()patchset and have more people review the actual code.
Quoted 6 lines
Quoted 3 lines
Sure. But it's also unpleasant to have people write code and then have to throw it away due to guessing incorrectly about unclear requirements.
No, it is not. It is not unpleasant. And it is useful. It is the best way to identify and resolve those incorrect guesses and unclear requirements.
No problem, a bit of discussion helped set the direction. Personally it did help clarify lot of things for me. We are hard at work with come up with an implementation and are looking at posting something soon. I agree that the best is to discuss on actual code where possible.
Re: pidfd design
In reply to Christian BraunerQuoted 208 lines · Christian Brauner
Quoted 198 lines · Joel Fernandes
Quoted 195 lines · Daniel Colascione
Quoted 3 lines · Christian Brauner
Quoted 1 line · Daniel Colascione
On Tue, Mar 19, 2019 at 6:52 PM Joel Fernandes
Quoted 2 lines · <[email protected]> wrote:
Quoted 2 lines
Quoted 2 lines
On Wed, Mar 20, 2019 at 12:10:23AM +0100, Christian Brauner
Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
On Tue, Mar 19, 2019 at 03:48:32PM -0700, Daniel Colascione
Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
On Tue, Mar 19, 2019 at 3:14 PM Christian Brauner
Quoted 1 line · <[email protected]> wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
So I dislike the idea of allocating new inodes from the
procfs super
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
block. I would like to avoid pinning the whole pidfd
concept exclusively
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to proc. The idea is that the pidfd API will be useable
through procfs
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
via
open("/proc/<pid>")because that is what users expectand really
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
wanted to have for a long time. So it makes sense to have
this working.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
But it should really be useable without it. That's why
translate_pid()Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
and
pidfd_clone()are on the table. What I'm saying is,once the pidfd
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
api is "complete" you should be able to set
CONFIG_PROCFS=N- even
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
though that's crazy - and still be able to use pidfds. This
is also a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
point akpm asked about when I did the
pidfd_send_signalwork.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I agree that you shouldn't need
CONFIG_PROCFS=Yto usepidfds. One
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
crazy idea that I was discussing with Joel the other day is
to just
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
make
CONFIG_PROCFS=Ymandatory and provide a new
get_procfs_root()Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
system call that returned, out of thin air and independent of
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
mount table, a procfs root directory file descriptor for the
caller's
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
PID namspace and suitable for use with openat(2).
Even if this works I'm pretty sure that Al and a lot of others
will not
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
be happy about this. A syscall to get an fd to /proc?
Why not? procfs provides access to a lot of core kernel
functionality.
Quoted 4 lines
Quoted 4 lines
Why should you need a mountpoint to get to it?
Quoted 2 lines
That's not going
Quoted 1 line
to happen and I don't see the need for a separate syscall just
for that.
Quoted 2 lines
Quoted 2 lines
We need a system call for the same reason we need a getrandom(2):
you
Quoted 6 lines
Quoted 6 lines
have to bootstrap somehow when you're in a minimal environment.
Quoted 1 line
Quoted 1 line
(I do see the point of making
CONFIG_PROCFS=ythe default btw.)I'm not proposing that we make
CONFIG_PROCFS=ythe default. I'm proposing that we *hardwire* it as the default and just declarethat
Quoted 1 line
Quoted 1 line
it's not possible to build a Linux kernel that doesn't include
procfs.
Quoted 3 lines
Quoted 3 lines
Why do we even have that button?
Quoted 1 line
I think his point here was that he wanted a handle to procfs no
matter where
Quoted 1 line
Quoted 1 line
Quoted 1 line
it was mounted and then can later use openat on that. Agreed that
it may be
Quoted 1 line
Quoted 1 line
Quoted 1 line
unnecessary unless there is a usecase for it, and especially if
the /proc
Quoted 1 line
Quoted 1 line
Quoted 1 line
directory being the defacto mountpoint for procfs is a universal
convention.
Quoted 2 lines
Quoted 2 lines
If it's a universal convention and, in practice, everyone needs
proc
Quoted 1 line
Quoted 1 line
mounted anyway, so what's the harm in hardwiring
CONFIG_PROCFS=y?If
Quoted 1 line
Quoted 1 line
we advertise /proc as not merely some kind of optional debug
interface
Quoted 3 lines
Quoted 3 lines
but *the* way certain kernel features are exposed --- and there's nothing wrong with that --- then we should give programs access to these core kernel features in a way that doesn't depend on
userspace
Quoted 2 lines
Quoted 2 lines
kernel configuration, and you do that by either providing a procfs-root-getting system call or just hardwiring the "
/proc/"prefix
Quoted 3 lines
Quoted 3 lines
into VFS.
Quoted 1 line
Quoted 1 line
Inode allocation from the procfs mount for the file descriptors
Joel
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
wants is not correct. Their not really procfs file descriptors
so this
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
is a nack. We can't just hook into proc that way.
I was not particular about using procfs mount for the FDs but
that's the only
Quoted 1 line
Quoted 1 line
Quoted 1 line
way I knew how to do it until you pointed out
anon_inode(my grepskills
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
missed that), so thank you!
Quoted 1 line
Quoted 1 line
C'mon: /proc is used by everyone today and almost every
program breaks
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
if it's not around. The string "/proc" is already de facto
kernel ABI.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Let's just drop the pretense of /proc being optional and bake
it into
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
the kernel proper, then give programs a way to get to /proc
that isn't
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
tied to any particular mount configuration. This way, we
don't need a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
translate_pid(), since callers can just use procfs to do thesame
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
thing. (That is, if I understand correctly what
translate_piddoes.)
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I'm not sure what you think
translate_pid()is doing sinceyou're not
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
saying what you think it does.
Examples from the old patchset:translate_pid(pid, ns, -1)- get pid in our pid namespaceAh, it's a bit different from what I had in mind. It's fair to want
to
Quoted 2 lines
Quoted 2 lines
translate PIDs between namespaces, but the only way to make the
translate_pidunder discussion robust is to have it accept andproduce
Quoted 1 line
Quoted 1 line
pidfds. (At that point, you might as well call it
translate_pidfd.)We
Quoted 1 line
Quoted 1 line
should not be adding new APIs to the kernel that accept numeric
PIDs:
Quoted 3 lines
The traditional pid-based api is not going away. There are users that have the requirement to translate pids between namespaces and also
doing
Quoted 8 lines
introspection on these namespaces independent of pidfds. We will not restrict the usefulness of this syscall by making it only work with pidfds.
Quoted 2 lines
it's not possible to use these APIs correctly except under very limited circumstances --- mostly, talking about init or a parent
The pid-based api is one of the most widely used apis of the kernel
and
Quoted 1 line
people have been using it quite successfully for a long time. Yes,
it's
Quoted 5 lines
rac, but it's here to stay.
Quoted 3 lines
talking about its child.
Really, we need a few related operations, and we shouldn't
necessarily
Quoted 8 lines
Quoted 1 line
mingle them.
Yes, we've established that previously.
Quoted 3 lines
1) Given a numeric PID, give me a pidfd: that works today: you just open /proc/<pid>
Quoted 11 lines
Agreed.
Quoted 4 lines
2) Given a pidfd, give me a numeric PID: that works today: you just
openat(pidfd, "stat", O_RDONLY)and read the first token (which is always the numeric PID).Agreed.
Quoted 2 lines
3) Given a pidfd, send a signal: that's what
pidfd_send_signaldoes,
Quoted 5 lines
Quoted 1 line
and it's a good start on the rest of these operations.
Agreed.
Quoted 1 line
5) Given a pidfd in NS1, get a pidfd in NS2. That's what
translate_pidQuoted 1 line
Quoted 1 line
is for. My preferred signature for this routine is
translate_pid(int
Quoted 1 line
Quoted 1 line
pidfd, int nsfd) -> pidfd. We don't need two namespace arguments.
Why
Quoted 3 lines
Quoted 1 line
not? Because the pidfd *already* names a single process, uniquely!
Given that people are interested in pids we can't just always return
a
Quoted 1 line
pidfd. That would mean a user would need to do get the pidfd read
from
Quoted 1 line
<pidfd>/stat and then close the pidfd. If you do that for a 100 pids
Quoted 8 lines
Quoted 8 lines
or
Quoted 1 line
more you end up allocating and closing file descriptors constantly
for
Quoted 2 lines
no reason. We can't just debate pids away. So it will also need to be able to yield pids e.g. through a flag argument.
Sure, but that's still not a reason that we should care about pidfds working separately from procfs..
Quoted 8 lines
That's unrelated to the point made in the above paragraph. Please note, I said that the pidfd api should work when proc is not available not that they can't be dirfds.
Quoted 2 lines
Agreed. I can't imagine pidfd being anything but a proc pid directory handle. So I am confused what Christian meant. Pidfd *is* a procfs directory fid always. That's what I gathered from his
pidfd_send_signalpatch but let me know if I'm way off in the woods.(K9 Mail still hasn't learned to wrap lines at 80 it seems. :))
Indeed, or I misconfigured it :) Just set it up recently so I'm still messing with it.
The other issue is it does wrapping on quoted lines too, and there's a bug filed somewhere for that.
Quoted 5 lines
Again, I never said that pidfds should be a directory handle. (Though I would like to point out that one of the original ideas I discussed at LPC was to have something like this to get regular file descriptors instead of dirfds:
https://gist.github.com/brauner/59eec91550c5624c9999eaebd95a70df)
Ok. I was just going by this code in your send_signal patch where you error out if the pidfd is not a directory.
+struct pid *tgid_pidfd_to_pid(const struct file *file) +{
+ if (!d_is_dir(file->f_path.dentry) ||
+ (file->f_op != &proc_tgid_base_operations))
+ return ERR_PTR(-EBADF);
Quoted 6 lines
Quoted 1 line
For my next revision, I am thinking of adding the flag argument Christian mentioned to make
translate_pidreturn ananon_inodeFD which can be used for death status, given a <pid>. Since it is thought thattranslate_pidcan be made to return a pid FD, I think it is ok to have it return a pid status FD for the purposes of the death status as well.
translate_pid()should just return you a pidfd. Having it return a pidfd and a status fd feels like stuffing too much functionality in there. If you're fine with it I'll finish prototyping what I had in mind. As I said in previous mails I'm already working on this.
Yes, please continue to work on it. No problem.
Quoted 2 lines
Would you be ok with prototyping the
pidfd_wait()syscall you had in mind?
Yes, Of course, I am working on it. No problem. It is still good to discuss these ideas and to know what my direction should be, so I appreciate the conversation here.
Quoted 5 lines
Especially the
wait_fdpart that you want to have I would like to see how that is supposed to work, e.g. who is allowed to wait on the process and how notifications will work for non-parent processes and so on. I feel we won't get anywhere by talking in the abstrace and other people are far more likely to review/comment once there's actual code.
Got it. Lets chat more once I post something.
thanks,
- Joel
This is a complete low memory killer solution for Android that is small and simple. Processes are killed according to the priorities that Android gives them, so that the least important processes are always killed first. Processes are killed until memory deficits are satisfied, as observed from kswapd struggling to free up pages. Simple LMK stops killing processes when kswapd finally goes back to sleep.
The only tunables are the desired amount of memory to be freed per reclaim event and desired frequency of reclaim events. Simple LMK tries to free at least the desired amount of memory per reclaim and waits until all of its victims' memory is freed before proceeding to kill more processes.
---
Hello everyone,
I've addressed some of the concerns that were brought up with the first version of the Simple LMK patch. I understand that a kernel-based solution like this that contains policy decisions for a specific userspace is not the way to go, but the Android ecosystem still has a pressing need for a low memory killer that works well.
Most Android devices still use the ancient and deprecated lowmemorykiller.c kernel driver; Simple LMK seeks to replace that, at the very least until PSI and a userspace daemon utilizing PSI are ready for *all* Android devices, and not just the privileged Pixel phone line.
The feedback I received last time was quite helpful. This Simple LMK patch works significantly better than the first, improving memory management by a large margin while being immune to transient spikes in memory usage (since the signal to start killing processes is determined by how hard kswapd tries to free up pages, which is something that occurs over a span of time and not a single point in time).
I'd love to hear some feedback on this new patch. I do encourage those who are interested to take it for a spin on an Android device. This patch has been tested successfully on Android 4.4 and 4.9 kernels. For the sake of review here, I have adapted the patch to 5.1.
Thanks,
Sultan
drivers/android/Kconfig | 33 ++++ drivers/android/Makefile | 1 + drivers/android/simple_lmk.c | 315 +++++++++++++++++++++++++++++++++++ include/linux/mm_types.h | 4 + include/linux/simple_lmk.h | 11 ++ kernel/fork.c | 13 ++ mm/vmscan.c | 12 ++ 7 files changed, 389 insertions(+) create mode 100644 drivers/android/simple_lmk.c create mode 100644 include/linux/simple_lmk.h
drivers/android/Kconfig +33 -0
@@ -54,6 +54,39 @@ config ANDROID_BINDER_IPC_SELFTEST exhaustively with combinations of various buffer sizes and alignments. +config ANDROID_SIMPLE_LMK + bool "Simple Android Low Memory Killer" + depends on !ANDROID_LOW_MEMORY_KILLER && !MEMCG + ---help--- + This is a complete low memory killer solution for Android that is + small and simple. Processes are killed according to the priorities + that Android gives them, so that the least important processes are + always killed first. Processes are killed until memory deficits are + satisfied, as observed from kswapd struggling to free up pages. Simple + LMK stops killing processes when kswapd finally goes back to sleep. + +if ANDROID_SIMPLE_LMK + +config ANDROID_SIMPLE_LMK_AGGRESSION + int "Reclaim frequency selection" + range 1 3 + default 1 + help + This value determines how frequently Simple LMK will perform memory + reclaims. A lower value corresponds to less frequent reclaims, which + maximizes memory usage. The range of values has a logarithmic + correlation; 2 is twice as aggressive as 1, and 3 is twice as + aggressive as 2, which makes 3 four times as aggressive as 1. + +config ANDROID_SIMPLE_LMK_MINFREE + int "Minimum MiB of memory to free per reclaim" + range 8 512 + default 64 + help + Simple LMK will try to free at least this much memory per reclaim. + +endif + endif # if ANDROID endmenu
drivers/android/Makefile +1 -0
@@ -3,3 +3,4 @@ ccflags-y += -I$(src) # needed for trace events obj-$(CONFIG_ANDROID_BINDERFS) += binderfs.o obj-$(CONFIG_ANDROID_BINDER_IPC) += binder.o binder_alloc.o obj-$(CONFIG_ANDROID_BINDER_IPC_SELFTEST) += binder_alloc_selftest.o +obj-$(CONFIG_ANDROID_SIMPLE_LMK) += simple_lmk.o
drivers/android/simple_lmk.c new file +315 -0
315 lines not on this page · raw message
include/linux/mm_types.h +4 -0
@@ -501,6 +501,10 @@ struct mm_struct {#if IS_ENABLED (CONFIG_HMM )/* HMM needs to track a few things per mm */ struct hmm *hmm; +#endif +#ifdef CONFIG_ANDROID_SIMPLE_LMK +wait_queue_head_t *slmk_waitq; +atomic_t *slmk_counter;#endif } __randomize_layout;
include/linux/simple_lmk.h new file +11 -0
@@ -0,0 +1,11 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Copyright (C) 2019 Sultan Alsawaf <[email protected]>. +*/ +#ifndef _SIMPLE_LMK_H_ +#define _SIMPLE_LMK_H_ + +void simple_lmk_start_reclaim (void ); +void simple_lmk_stop_reclaim (void ); + +#endif /* _SIMPLE_LMK_H_ */
kernel/fork.c +13 -0
@@ -995,6 +995,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, mm->pmd_huge_pte =NULL ;#endif mm_init_uprobes_state (mm); +#ifdef CONFIG_ANDROID_SIMPLE_LMK + mm->slmk_waitq =NULL ; +#endif if (current->mm) { mm->flags = current->mm->flags &MMF_INIT_MASK ;
@@ -1037,6 +1040,10 @@ struct mm_struct *mm_alloc(void)static inline void __mmput (struct mm_struct *mm) { +#ifdef CONFIG_ANDROID_SIMPLE_LMK +wait_queue_head_t *slmk_waitq = mm->slmk_waitq; +atomic_t *slmk_counter = mm->slmk_counter; +#endif VM_BUG_ON (atomic_read (&mm->mm_users));uprobe_clear_state (mm);
@@ -1054,6 +1061,12 @@ static inline void __mmput(struct mm_struct *mm)if (mm->binfmt)module_put (mm->binfmt->module);mmdrop (mm); +#ifdef CONFIG_ANDROID_SIMPLE_LMK +if (slmk_waitq) { +atomic_dec (slmk_counter); +wake_up (slmk_waitq); + } +#endif }/*
mm/vmscan.c +12 -0
@@ -51,6 +51,7 @@#include <linux/printk.h> #include <linux/dax.h> #include <linux/psi.h> +#include <linux/simple_lmk.h> #include <asm/tlbflush.h> #include <asm/div64.h>
@@ -3541,6 +3542,11 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int classzone_idx)bool balanced;bool ret; +#ifdef CONFIG_ANDROID_SIMPLE_LMK +if (sc.priority ==CONFIG_ANDROID_SIMPLE_LMK_AGGRESSION ) +simple_lmk_start_reclaim (); +#endif + sc.reclaim_idx = classzone_idx;/*
@@ -3737,6 +3743,9 @@ static void kswapd_try_to_sleep(pg_data_t *pgdat, int alloc_order, int reclaim_o* succeed. */ if (prepare_kswapd_sleep (pgdat, reclaim_order, classzone_idx)) { +#ifdef CONFIG_ANDROID_SIMPLE_LMK +simple_lmk_stop_reclaim (); +#endif /* * Compaction records what page blocks it recently failed to * isolate pages from and skips them in the future scanning.
@@ -3773,6 +3782,9 @@ static void kswapd_try_to_sleep(pg_data_t *pgdat, int alloc_order, int reclaim_o*/ if (!remaining &&prepare_kswapd_sleep (pgdat, reclaim_order, classzone_idx)) { +#ifdef CONFIG_ANDROID_SIMPLE_LMK +simple_lmk_stop_reclaim (); +#endif trace_mm_vmscan_kswapd_sleep (pgdat->node_id);/*
Signature · 1 line
2.21.0
Greg Kroah-Hartman
maintainerQuoted 23 lines · Sultan Alsawaf
This is a complete low memory killer solution for Android that is small and simple. Processes are killed according to the priorities that Android gives them, so that the least important processes are always killed first. Processes are killed until memory deficits are satisfied, as observed from kswapd struggling to free up pages. Simple LMK stops killing processes when kswapd finally goes back to sleep.
The only tunables are the desired amount of memory to be freed per reclaim event and desired frequency of reclaim events. Simple LMK tries to free at least the desired amount of memory per reclaim and waits until all of its victims' memory is freed before proceeding to kill more processes.
Signed-off-by Sultan Alsawaf <[email protected]>---
Hello everyone,I've addressed some of the concerns that were brought up with the first version of the Simple LMK patch. I understand that a kernel-based solution like this that contains policy decisions for a specific userspace is not the way to go, but the Android ecosystem still has a pressing need for a low memory killer that works well.
Quoted 4 lines
Most Android devices still use the ancient and deprecated
lowmemorykiller.ckernel driver; Simple LMK seeks to replace that, at the very least until PSI and a userspace daemon utilizing PSI are ready for *all* Android devices, and not just the privileged Pixel phone line.
Um, why can't "all" Android devices take the same patches that the Pixel phones are using today? They should all be in the public android-common kernel repositories that all Android devices should be syncing with on a weekly/monthly basis anyway, right?
thanks,
greg k-h
Quoted 8 lines · Greg Kroah-Hartman
Um, why can't "all" Android devices take the same patches that the Pixel phones are using today? They should all be in the public android-common kernel repositories that all Android devices should be syncing with on a weekly/monthly basis anyway, right?
thanks,
greg k-h
Hi Greg,
I only see PSI present in the android-common kernels for 4.9 and above. The vast majority of Android devices do not run a 4.9+ kernel. It seems unreasonable to expect OEMs to toil with backporting PSI themselves to get decent memory management.
But even if they did backport PSI, it wouldn't help too much because a PSI-enabled LMKD solution is not ready yet. It looks like a PSI-based LMKD is still under heavy development and won't be ready for all Android devices for quite some time.
Additionally, it looks like the supposedly-dead lowmemorykiller.c is still being actively tweaked by Google [1], which does not instill confidence that a definitive LMK solution a la PSI is coming any time soon.
Thanks,
Sultan
[1] https://android.googlesource.com/kernel/common/+/152bacdd85c46f0c76b00c4acc253e414513634c
Greg Kroah-Hartman
maintainerQuoted 8 lines · Sultan Alsawaf
Quoted 7 lines · Greg Kroah-Hartman
Um, why can't "all" Android devices take the same patches that the Pixel phones are using today? They should all be in the public android-common kernel repositories that all Android devices should be syncing with on a weekly/monthly basis anyway, right?
thanks,
Quoted 8 lines
Quoted 1 line
greg k-h
Hi Greg,
I only see PSI present in the android-common kernels for 4.9 and above. The vast majority of Android devices do not run a 4.9+ kernel. It seems unreasonable to expect OEMs to toil with backporting PSI themselves to get decent memory management.
Given that any "new" android device that gets shipped "soon" should be using 4.9.y or newer, is this a real issue?
And if it is, I'm sure that asking for those patches to be backported to 4.4.y would be just fine, have you asked?
Note that I know of Android Go devices, running 3.18.y kernels, do NOT use the in-kernel memory killer, but instead use the userspace solution today. So trying to get another in-kernel memory killer solution added anywhere seems quite odd.
thanks,
greg k-h
Quoted 2 lines · Greg Kroah-Hartman
Given that any "new" android device that gets shipped "soon" should be using 4.9.y or newer, is this a real issue?
It's certainly a real issue for those who can't buy brand new Android devices without software bugs every six months :)
Quoted 7 lines
And if it is, I'm sure that asking for those patches to be backported to 4.4.y would be just fine, have you asked?
Note that I know of Android Go devices, running 3.18.y kernels, do NOT use the in-kernel memory killer, but instead use the userspace solution today. So trying to get another in-kernel memory killer solution added anywhere seems quite odd.
It's even more odd that although a userspace solution is touted as the proper way to go on LKML, almost no Android OEMs are using it, and even in that commit I linked in the previous message, Google made a rather large set of modifications to the supposedly-defunct lowmemorykiller.c not one month ago. What's going on?
Qualcomm still uses lowmemorykiller.c [1] on the Snapdragon 845. If PSI were backported to 4.4, or even 3.18, would it really be used? I don't really understand the aversion to an in-kernel memory killer on LKML despite the rest of the industry's attraction to it. Perhaps there's some inherently great cost in using the userspace solution that I'm unaware of?
Regardless, even if PSI were backported, a full-fledged LMKD using it has yet to be made, so it wouldn't be of much use now.
Thanks,
Sultan
Christian Brauner
maintainerQuoted 10 lines · Sultan Alsawaf
Quoted 2 lines · Greg Kroah-Hartman
Given that any "new" android device that gets shipped "soon" should be using 4.9.y or newer, is this a real issue?
It's certainly a real issue for those who can't buy brand new Android devices without software bugs every six months :)
Quoted 3 lines
And if it is, I'm sure that asking for those patches to be backported to 4.4.y would be just fine, have you asked?
Quoted 7 lines
Quoted 4 lines
Note that I know of Android Go devices, running 3.18.y kernels, do NOT use the in-kernel memory killer, but instead use the userspace solution today. So trying to get another in-kernel memory killer solution added anywhere seems quite odd.
It's even more odd that although a userspace solution is touted as the proper way to go on LKML, almost no Android OEMs are using it, and even in that commit
That's probably because without proper kernel changes this is rather tricky to use safely (see below).
Quoted 4 lines
I linked in the previous message, Google made a rather large set of modifications to the supposedly-defunct
lowmemorykiller.cnot one month ago. What's going on?
Quoted 8 lines
Qualcomm still uses
lowmemorykiller.c[1] on the Snapdragon 845. If PSI were backported to 4.4, or even 3.18, would it really be used? I don't really understand the aversion to an in-kernel memory killer on LKML despite the rest of the industry's attraction to it. Perhaps there's some inherently great cost in using the userspace solution that I'm unaware of?Regardless, even if PSI were backported, a full-fledged LMKD using it has yet to be made, so it wouldn't be of much use now.
This is work that is ongoing and requires kernel changes to make it feasible. One of the things that I have been working on for quite a while is the whole file descriptor for processes thing that is important for LMKD (Even though I never thought about this use-case when I started pitching this.). Joel and Daniel have joined in and are working on making LMKD possible.
What I find odd is that every couple of weeks different solutions to the low memory problem are pitched. There is simple_lkml, there is LMKD, and there was a patchset that wanted to speed up memory reclaim at process kill-time by adding a new flag to the new pidfd_send_signal() syscall. That all seems - though related - rather uncoordinated. Now granted, coordinated is usually not how kernel development necessarily works but it would probably be good to have some sort of direction and from what I have seen LMKD seems to be the most coordinated effort. But that might just be my impression.
Christian
Suren Baghdasaryan
reviewerFrom: Christian Brauner <[email protected]>
Date: Tue, May 7, 2019 at 3:58 AM
To: Sultan Alsawaf
Cc: Greg Kroah-Hartman, open list:ANDROID DRIVERS, Daniel Colascione, Todd Kjos, Kees Cook, Peter Zijlstra, Martijn Coenen, LKML, Tim Murray, Michal Hocko, Suren Baghdasaryan, linux-mm, Arve Hjønnevåg, Ingo Molnar, Steven Rostedt, Oleg Nesterov, Joel Fernandes, Andy Lutomirski, kernel-team
Quoted 8 lines · Sultan Alsawaf
Quoted 7 lines · Greg Kroah-Hartman
Quoted 2 lines
Given that any "new" android device that gets shipped "soon" should be using 4.9.y or newer, is this a real issue?
It's certainly a real issue for those who can't buy brand new Android devices without software bugs every six months :)
Hi Sultan,
Looks like you are posting this patch for devices that do not use userspace LMKD solution due to them using older kernels or due to their vendors sticking to in-kernel solution. If so, I see couple logistical issues with this patch. I don't see it being adopted in upstream kernel 5.x since it re-implements a deprecated mechanism even though vendors still use it. Vendors on the other hand, will not adopt it until you show evidence that it works way better than what lowmemorykilled driver does now. You would have to provide measurable data and explain your tests before they would consider spending time on this.
On the implementation side I'm not convinced at all that this would work better on all devices and in all circumstances. We had cases when a new mechanism would show very good results until one usecase completely broke it. Bulk killing of processes that you are doing in your patch was a very good example of such a decision which later on we had to rethink. That's why baking these policies into kernel is very problematic. Another problem I see with the implementation that it ties process killing with the reclaim scan depth. It's very similar to how vmpressure works and vmpressure in my experience is very unpredictable.
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
And if it is, I'm sure that asking for those patches to be backported to 4.4.y would be just fine, have you asked?
Note that I know of Android Go devices, running 3.18.y kernels, do NOT use the in-kernel memory killer, but instead use the userspace solution today. So trying to get another in-kernel memory killer solution added anywhere seems quite odd.
Quoted 9 lines
Quoted 2 lines
It's even more odd that although a userspace solution is touted as the proper way to go on LKML, almost no Android OEMs are using it, and even in that commit
That's probably because without proper kernel changes this is rather tricky to use safely (see below).
Quoted 3 lines
I linked in the previous message, Google made a rather large set of modifications to the supposedly-defunct
lowmemorykiller.cnot one month ago. What's going on?
If you look into that commit, it adds ability to report kill stats. If that was a change in how that driver works it would be rejected.
Quoted 6 lines
Quoted 6 lines
Qualcomm still uses
lowmemorykiller.c[1] on the Snapdragon 845. If PSI were backported to 4.4, or even 3.18, would it really be used? I don't really understand the aversion to an in-kernel memory killer on LKML despite the rest of the industry's attraction to it. Perhaps there's some inherently great cost in using the userspace solution that I'm unaware of?
Vendors are cautious about adopting userspace solution and it is a process to address all concerns but we are getting there.
Quoted 14 lines
Quoted 2 lines
Regardless, even if PSI were backported, a full-fledged LMKD using it has yet to be made, so it wouldn't be of much use now.
This is work that is ongoing and requires kernel changes to make it feasible. One of the things that I have been working on for quite a while is the whole file descriptor for processes thing that is important for LMKD (Even though I never thought about this use-case when I started pitching this.). Joel and Daniel have joined in and are working on making LMKD possible.
What I find odd is that every couple of weeks different solutions to the low memory problem are pitched. There issimple_lkml, there is LMKD, and there was a patchset that wanted to speed up memory reclaim at process kill-time by adding a new flag to the newpidfd_send_signal()syscall. That all seems - though related - rather uncoordinated.
I'm not sure why pidfd_wait and expedited reclaim is seen as uncoordinated effort. All of them are done to improve userspace LMKD.
Quoted 7 lines
Now granted,
coordinated is usually not how kernel development necessarily works but it would probably be good to have some sort of direction and from what I have seen LMKD seems to be the most coordinated effort. But that might just be my impression.Christian
Thanks,
Suren.
Christian Brauner
maintainerQuoted 86 lines · Suren Baghdasaryan
From: Christian Brauner <[email protected]>
Date: Tue, May 7, 2019 at 3:58 AM
To: Sultan Alsawaf
Cc: Greg Kroah-Hartman, open list:ANDROID DRIVERS, Daniel Colascione, Todd Kjos, Kees Cook, Peter Zijlstra, Martijn Coenen, LKML, Tim Murray, Michal Hocko, Suren Baghdasaryan, linux-mm, Arve Hjønnevåg, Ingo Molnar, Steven Rostedt, Oleg Nesterov, Joel Fernandes, Andy Lutomirski, kernel-teamQuoted 8 lines · Sultan Alsawaf
Quoted 7 lines · Greg Kroah-Hartman
Quoted 2 lines
Given that any "new" android device that gets shipped "soon" should be using 4.9.y or newer, is this a real issue?
It's certainly a real issue for those who can't buy brand new Android devices without software bugs every six months :)
Hi Sultan,
Looks like you are posting this patch for devices that do not use userspace LMKD solution due to them using older kernels or due to their vendors sticking to in-kernel solution. If so, I see couple logistical issues with this patch. I don't see it being adopted in upstream kernel 5.x since it re-implements a deprecated mechanism even though vendors still use it. Vendors on the other hand, will not adopt it until you show evidence that it works way better than what lowmemorykilled driver does now. You would have to provide measurable data and explain your tests before they would consider spending time on this.
On the implementation side I'm not convinced at all that this would work better on all devices and in all circumstances. We had cases when a new mechanism would show very good results until one usecase completely broke it. Bulk killing of processes that you are doing in your patch was a very good example of such a decision which later on we had to rethink. That's why baking these policies into kernel is very problematic. Another problem I see with the implementation that it ties process killing with the reclaim scan depth. It's very similar to how vmpressure works and vmpressure in my experience is very unpredictable.Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
And if it is, I'm sure that asking for those patches to be backported to 4.4.y would be just fine, have you asked?
Note that I know of Android Go devices, running 3.18.y kernels, do NOT use the in-kernel memory killer, but instead use the userspace solution today. So trying to get another in-kernel memory killer solution added anywhere seems quite odd.
Quoted 9 lines
Quoted 2 lines
It's even more odd that although a userspace solution is touted as the proper way to go on LKML, almost no Android OEMs are using it, and even in that commit
That's probably because without proper kernel changes this is rather tricky to use safely (see below).
Quoted 3 lines
I linked in the previous message, Google made a rather large set of modifications to the supposedly-defunct
lowmemorykiller.cnot one month ago. What's going on?If you look into that commit, it adds ability to report kill stats. If that was a change in how that driver works it would be rejected.
Quoted 6 lines
Quoted 6 lines
Qualcomm still uses
lowmemorykiller.c[1] on the Snapdragon 845. If PSI were backported to 4.4, or even 3.18, would it really be used? I don't really understand the aversion to an in-kernel memory killer on LKML despite the rest of the industry's attraction to it. Perhaps there's some inherently great cost in using the userspace solution that I'm unaware of?Vendors are cautious about adopting userspace solution and it is a process to address all concerns but we are getting there.
Quoted 14 lines
Quoted 2 lines
Regardless, even if PSI were backported, a full-fledged LMKD using it has yet to be made, so it wouldn't be of much use now.
This is work that is ongoing and requires kernel changes to make it feasible. One of the things that I have been working on for quite a while is the whole file descriptor for processes thing that is important for LMKD (Even though I never thought about this use-case when I started pitching this.). Joel and Daniel have joined in and are working on making LMKD possible.
What I find odd is that every couple of weeks different solutions to the low memory problem are pitched. There issimple_lkml, there is LMKD, and there was a patchset that wanted to speed up memory reclaim at process kill-time by adding a new flag to the newpidfd_send_signal()syscall. That all seems - though related - rather uncoordinated.
Quoted 2 lines
I'm not sure why
pidfd_waitand expedited reclaim is seen as uncoordinated effort. All of them are done to improve userspace LMKD.
If so that wasn't very obvious and there was some disagreement there as well whether this would be the right solution.
In any case, the point is that LMKD seems to be the way forward and with all of the arguments brought forward here this patchset seems like it's going in the wrong direction.
Christian
Quoted 11 lines
Quoted 8 lines
Now granted,
coordinated is usually not how kernel development necessarily works but it would probably be good to have some sort of direction and from what I have seen LMKD seems to be the most coordinated effort. But that might just be my impression.Christian
Thanks,
Suren.
Quoted 11 lines · Suren Baghdasaryan
Hi Sultan,
Looks like you are posting this patch for devices that do not use userspace LMKD solution due to them using older kernels or due to their vendors sticking to in-kernel solution. If so, I see couple logistical issues with this patch. I don't see it being adopted in upstream kernel 5.x since it re-implements a deprecated mechanism even though vendors still use it. Vendors on the other hand, will not adopt it until you show evidence that it works way better than what lowmemorykilled driver does now. You would have to provide measurable data and explain your tests before they would consider spending time on this.
Yes, this is mostly for the devices already produced that are forced to suffer with poor memory management. I can't even convince vendors to fix kernel memory leaks, so there's no way I'd be able to convince them of trying this patch, especially since it seems like you're having trouble convincing vendors to stop using lowmemorykiller in the first place. And thankfully, convincing vendors isn't my job :)
Quoted 10 lines
On the implementation side I'm not convinced at all that this would work better on all devices and in all circumstances. We had cases when a new mechanism would show very good results until one usecase completely broke it. Bulk killing of processes that you are doing in your patch was a very good example of such a decision which later on we had to rethink. That's why baking these policies into kernel is very problematic. Another problem I see with the implementation that it ties process killing with the reclaim scan depth. It's very similar to how vmpressure works and vmpressure in my experience is very unpredictable.
Could you elaborate a bit on why bulk killing isn't good?
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
I linked in the previous message, Google made a rather large set of modifications to the supposedly-defunct
lowmemorykiller.cnot one month ago. What's going on?If you look into that commit, it adds ability to report kill stats. If that was a change in how that driver works it would be rejected.
Fair, though it was quite strange seeing something that was supposedly totally abandoned receiving a large chunk of code for reporting stats.
Thanks,
Sultan
Suren Baghdasaryan
reviewerFrom: Sultan Alsawaf <[email protected]>
Date: Tue, May 7, 2019 at 9:53 AM
To: Suren Baghdasaryan
Cc: Christian Brauner, Greg Kroah-Hartman, open list:ANDROID DRIVERS, Daniel Colascione, Todd Kjos, Kees Cook, Peter Zijlstra, Martijn Coenen, LKML, Tim Murray, Michal Hocko, linux-mm, Arve Hjønnevåg, Ingo Molnar, Steven Rostedt, Oleg Nesterov, Joel Fernandes, Andy Lutomirski, kernel-team
Quoted 31 lines · Suren Baghdasaryan
Quoted 11 lines
Hi Sultan,
Looks like you are posting this patch for devices that do not use userspace LMKD solution due to them using older kernels or due to their vendors sticking to in-kernel solution. If so, I see couple logistical issues with this patch. I don't see it being adopted in upstream kernel 5.x since it re-implements a deprecated mechanism even though vendors still use it. Vendors on the other hand, will not adopt it until you show evidence that it works way better than what lowmemorykilled driver does now. You would have to provide measurable data and explain your tests before they would consider spending time on this.Yes, this is mostly for the devices already produced that are forced to suffer with poor memory management. I can't even convince vendors to fix kernel memory leaks, so there's no way I'd be able to convince them of trying this patch, especially since it seems like you're having trouble convincing vendors to stop using lowmemorykiller in the first place. And thankfully, convincing vendors isn't my job :)
Quoted 10 lines
On the implementation side I'm not convinced at all that this would work better on all devices and in all circumstances. We had cases when a new mechanism would show very good results until one usecase completely broke it. Bulk killing of processes that you are doing in your patch was a very good example of such a decision which later on we had to rethink. That's why baking these policies into kernel is very problematic. Another problem I see with the implementation that it ties process killing with the reclaim scan depth. It's very similar to how vmpressure works and vmpressure in my experience is very unpredictable.
Quoted 1 line
Could you elaborate a bit on why bulk killing isn't good?
Yes. Several months ago we got reports that LMKD got very aggressive killing more processes in situations which did not require that many kills to recover from memory pressure. After investigation we tracked it to the bulk killing which would kill too many processes during a memory usage spike. When killing gradually the kills would be avoided, so we had to get back to a more balanced approach. The conceptual issue with bulk killing is that you can't cancel it when device recovers quicker than you expected.
Quoted 12 lines
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
I linked in the previous message, Google made a rather large set of modifications to the supposedly-defunct
lowmemorykiller.cnot one month ago. What's going on?If you look into that commit, it adds ability to report kill stats. If that was a change in how that driver works it would be rejected.
Fair, though it was quite strange seeing something that was supposedly totally abandoned receiving a large chunk of code for reporting stats.
Thanks,
Sultan
Quoted 31 lines · Suren Baghdasaryan
From: Christian Brauner <[email protected]>
Date: Tue, May 7, 2019 at 3:58 AM
To: Sultan Alsawaf
Cc: Greg Kroah-Hartman, open list:ANDROID DRIVERS, Daniel Colascione, Todd Kjos, Kees Cook, Peter Zijlstra, Martijn Coenen, LKML, Tim Murray, Michal Hocko, Suren Baghdasaryan, linux-mm, Arve Hjønnevåg, Ingo Molnar, Steven Rostedt, Oleg Nesterov, Joel Fernandes, Andy Lutomirski, kernel-teamQuoted 8 lines · Sultan Alsawaf
Quoted 7 lines · Greg Kroah-Hartman
Quoted 2 lines
Given that any "new" android device that gets shipped "soon" should be using 4.9.y or newer, is this a real issue?
It's certainly a real issue for those who can't buy brand new Android devices without software bugs every six months :)
Hi Sultan,
Looks like you are posting this patch for devices that do not use userspace LMKD solution due to them using older kernels or due to their vendors sticking to in-kernel solution. If so, I see couple logistical issues with this patch. I don't see it being adopted in upstream kernel 5.x since it re-implements a deprecated mechanism even though vendors still use it. Vendors on the other hand, will not adopt it until you show evidence that it works way better than what lowmemorykilled driver does now. You would have to provide measurable data and explain your tests before they would consider spending time on this.
On the implementation side I'm not convinced at all that this would work better on all devices and in all circumstances. We had cases when
Quoted 8 lines
a new mechanism would show very good results until one usecase completely broke it. Bulk killing of processes that you are doing in your patch was a very good example of such a decision which later on we had to rethink. That's why baking these policies into kernel is very problematic. Another problem I see with the implementation that it ties process killing with the reclaim scan depth. It's very similar to how vmpressure works and vmpressure in my experience is very unpredictable.
Yeah it does seem conceptually similar, good point.
Quoted 15 lines
Quoted 14 lines
Quoted 2 lines
Regardless, even if PSI were backported, a full-fledged LMKD using it has yet to be made, so it wouldn't be of much use now.
This is work that is ongoing and requires kernel changes to make it feasible. One of the things that I have been working on for quite a while is the whole file descriptor for processes thing that is important for LMKD (Even though I never thought about this use-case when I started pitching this.). Joel and Daniel have joined in and are working on making LMKD possible.
What I find odd is that every couple of weeks different solutions to the low memory problem are pitched. There issimple_lkml, there is LMKD, and there was a patchset that wanted to speed up memory reclaim at process kill-time by adding a new flag to the newpidfd_send_signal()syscall. That all seems - though related - rather uncoordinated.
Quoted 2 lines
I'm not sure why
pidfd_waitand expedited reclaim is seen as uncoordinated effort. All of them are done to improve userspace LMKD.
Christian, pidfd_wait and expedited reclaim are both coordinated efforts and solve different problems related to LMK. simple_lmk is entirely different effort that we already hesitated about when it was first posted, now we hesitate again due to the issues Suren and others mentioned.
I think it is a better idea for Sultan to spend his time on using/improving PSI/LMKd than spending it on the simple_lmk. It could also be a good topic to discuss in the Android track of the Linux plumbers conference.
thanks,
- Joel
Quoted 7 lines · Christian Brauner
This is work that is ongoing and requires kernel changes to make it feasible. One of the things that I have been working on for quite a while is the whole file descriptor for processes thing that is important for LMKD (Even though I never thought about this use-case when I started pitching this.). Joel and Daniel have joined in and are working on making LMKD possible.
What I find odd is that every couple of weeks different solutions to the
Quoted 8 lines
low memory problem are pitched. There is
simple_lkml, there is LMKD, and there was a patchset that wanted to speed up memory reclaim at process kill-time by adding a new flag to the newpidfd_send_signal()syscall. That all seems - though related - rather uncoordinated. Now granted, coordinated is usually not how kernel development necessarily works but it would probably be good to have some sort of direction and from what I have seen LMKD seems to be the most coordinated effort. But that might just be my impression.
LMKD is just Android's userspace low-memory-killer daemon. It's been around for a while.
This patch (simple_lmk) is meant to serve as an immediate solution for the devices that'll never see a single line of PSI code running on them, which amounts to... well, most Android devices currently in existence. I'm more of a cowboy who made this patch after waiting a few years for memory management improvements on Android that never happened. Though it looks like it's going to happen soon(ish?) for super new devices that'll have the privilege of shipping with PSI in use.
Quoted 7 lines · Greg Kroah-Hartman
Quoted 5 lines
It's even more odd that although a userspace solution is touted as the proper way to go on LKML, almost no Android OEMs are using it, and even in that commit I linked in the previous message, Google made a rather large set of modifications to the supposedly-defunct
lowmemorykiller.cnot one month ago. What's going on?"almost no"? Again, Android Go is doing that, right?
I'd check for myself, but I can't seem to find kernel source for an Android Go device...
This seems more confusing though. Why would the ultra-low-end devices use LMKD while other devices use the broken lowmemorykiller driver?
Quoted 4 lines
Quoted 1 line
Qualcomm still uses
lowmemorykiller.c[1] on the Snapdragon 845.Qualcomm should never be used as an example of a company that has any idea of what to do in their kernel :)
Agreed, but nearly all OEMs that use Qualcomm chipsets roll with Qualcomm's kernel decisions, so Qualcomm has a bit of influence here.
Quoted 3 lines
Quoted 1 line
If PSI were backported to 4.4, or even 3.18, would it really be used?
Why wouldn't it, if it worked properly?
For the same mysterious reason that Qualcomm and others cling to lowmemorykiller, I presume. This is part of what's been confusing me for quite some time...
Quoted 10 lines
Please see the work that went into PSI and the patches around it. There's also a lwn.net article last week about the further work ongoing in this area. With all of that, you should see how in-kernel memory killers are NOT the way to go.
Quoted 2 lines
Regardless, even if PSI were backported, a full-fledged LMKD using it has yet to be made, so it wouldn't be of much use now.
"LMKD"? Again, PSI is in the 4.9.y android-common tree, so the userspace side should be in AOSP, right?
LMKD as in Android's low-memory-killer daemon. It is in AOSP, but it looks like it's still a work in progress.
Thanks,
Sultan
Greg Kroah-Hartman
maintainerQuoted 7 lines · Sultan Alsawaf
Quoted 6 lines · Greg Kroah-Hartman
Quoted 5 lines
It's even more odd that although a userspace solution is touted as the proper way to go on LKML, almost no Android OEMs are using it, and even in that commit I linked in the previous message, Google made a rather large set of modifications to the supposedly-defunct
lowmemorykiller.cnot one month ago. What's going on?
Quoted 7 lines
Quoted 1 line
"almost no"? Again, Android Go is doing that, right?
I'd check for myself, but I can't seem to find kernel source for an Android Go device...
This seems more confusing though. Why would the ultra-low-end devices use LMKD while other devices use the broken lowmemorykiller driver?
It's probably because the Android Go devices got a lot more "help" from people at Google than did the other devices you are looking at. Also, despite the older kernel version, they are probably running a newer version of Android userspace, specially tuned just for lower memory devices.
So those 3.18.y based Android Go devices are newer than the 4.4.y based "full Android" devices on the market, and even some 4.9.y based devices.
Yes, it is strange :)
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Qualcomm still uses
lowmemorykiller.c[1] on the Snapdragon 845.Qualcomm should never be used as an example of a company that has any idea of what to do in their kernel :)
Agreed, but nearly all OEMs that use Qualcomm chipsets roll with Qualcomm's kernel decisions, so Qualcomm has a bit of influence here.
Yes, because almost no OEM wants to mess with their kernel, they just take QCOM's kernel and run with it. But don't take that for some sort of "best design practice" summary at all.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
If PSI were backported to 4.4, or even 3.18, would it really be used?
Why wouldn't it, if it worked properly?
For the same mysterious reason that Qualcomm and others cling to lowmemorykiller, I presume. This is part of what's been confusing me for quite some time...
QCOM's 4.4.y based kernel work was done 3-4 years ago, if not older. They didn't know that this was not the "right way" to do things. The Google developers have been working for the past few years to do it correct, but they can not go back in time to change old repos, sorry.
Now that I understand you just want to work on your local device, that makes more sense. But I think you will have a better result trying to do a 4.4 backport of PSI combined with the userspace stuff, than to try to worry about your driver in 5.2 or newer.
Or you can forward-port your kernel to 4.9, or better yet, 4.14. That would probably be a much better thing to do overall as 4.4 is really old now.
Good luck!
greg k-h
Greg Kroah-Hartman
maintainerQuoted 6 lines · Sultan Alsawaf
Quoted 2 lines · Greg Kroah-Hartman
Given that any "new" android device that gets shipped "soon" should be using 4.9.y or newer, is this a real issue?
It's certainly a real issue for those who can't buy brand new Android devices without software bugs every six months :)
Heh.
But, your "new code" isn't going to be going into any existing device, or any device that will come out this year. The soonest it would be would be next year, and by then, 4.9.y is fine.
Quoted 8 lines
Quoted 7 lines
And if it is, I'm sure that asking for those patches to be backported to 4.4.y would be just fine, have you asked?
Note that I know of Android Go devices, running 3.18.y kernels, do NOT use the in-kernel memory killer, but instead use the userspace solution today. So trying to get another in-kernel memory killer solution added anywhere seems quite odd.
Quoted 5 lines
It's even more odd that although a userspace solution is touted as the proper way to go on LKML, almost no Android OEMs are using it, and even in that commit I linked in the previous message, Google made a rather large set of modifications to the supposedly-defunct
lowmemorykiller.cnot one month ago. What's going on?
"almost no"? Again, Android Go is doing that, right?
And yes, there is still some 4.4 android-common work happening in this area, see this patch that just got merged:
https://android-review.googlesource.com/c/kernel/common/+/953354
So, for 4.4.y based devices, that should be enough, right?
Quoted 1 line
Qualcomm still uses
lowmemorykiller.c[1] on the Snapdragon 845.
Qualcomm should never be used as an example of a company that has any idea of what to do in their kernel :)
Quoted 1 line
If PSI were backported to 4.4, or even 3.18, would it really be used?
Why wouldn't it, if it worked properly?
Quoted 4 lines
I don't really understand the aversion to an in-kernel memory killer on LKML despite the rest of the industry's attraction to it. Perhaps there's some inherently great cost in using the userspace solution that I'm unaware of?
Please see the work that went into PSI and the patches around it. There's also a lwn.net article last week about the further work ongoing in this area. With all of that, you should see how in-kernel memory killers are NOT the way to go.
Quoted 2 lines
Regardless, even if PSI were backported, a full-fledged LMKD using it has yet to be made, so it wouldn't be of much use now.
"LMKD"? Again, PSI is in the 4.9.y android-common tree, so the userspace side should be in AOSP, right?
thanks,
greg k-h
Michal Hocko
reviewerQuoted 7 lines · Sultan Alsawaf
This is a complete low memory killer solution for Android that is small and simple. Processes are killed according to the priorities that Android gives them, so that the least important processes are always killed first. Processes are killed until memory deficits are satisfied, as observed from kswapd struggling to free up pages. Simple LMK stops killing processes when kswapd finally goes back to sleep.
Quoted 5 lines
The only tunables are the desired amount of memory to be freed per reclaim event and desired frequency of reclaim events. Simple LMK tries to free at least the desired amount of memory per reclaim and waits until all of its victims' memory is freed before proceeding to kill more processes.
Why do we need something like that in the kernel? I really do not like an idea of having two OOM killer implementations in the kernel. As already pointed out newer kernels can do PSI and older kernels can live with an out of tree code to achieve what they need. I do not see why we really need this code in the upstream kernel.
Signature · 2 lines
Michal Hocko SUSE Labs
I am not going to comment the intent, but to be honest I am skeptical too.
Quoted 15 lines · Sultan Alsawaf
drivers/android/simple_lmk.c new file +14
-0+static unsigned long find_victims (struct victim_info *varr,int *vindex, +int vmaxlen,int min_adj,int max_adj) +{ +unsigned long pages_found =0 ; +int old_vindex = *vindex; +struct task_struct *tsk; + +for_each_process (tsk) { +struct task_struct *vtsk; +unsigned long tasksize; +short oom_score_adj; + +/* Make sure there's space left in the victim array */ +if (*vindex == vmaxlen)
drivers/android/simple_lmk.c line 119
Quoted 10 lines
drivers/android/simple_lmk.c new file +10
-0+break ; + +/* Don't kill current, kthreads, init, or duplicates */ +if (same_thread_group (tsk, current) || + tsk->flags &PF_KTHREAD || +is_global_init (tsk) || +vtsk_is_duplicate (varr, *vindex, tsk)) +continue ; + + vtsk =find_lock_task_mm (tsk);
Did you test this patch with lockdep enabled?
If I read the patch correctly, lockdep should complain. vtsk_is_duplicate() ensures that we do not take the same ->alloc_lock twice or more, but lockdep can't know this.
Quoted 6 lines
drivers/android/simple_lmk.c new file +6
-0+static void scan_and_kill (unsigned long pages_needed) +{ +static DECLARE_WAIT_QUEUE_HEAD (victim_waitq); +struct victim_info victims[MAX_VICTIMS ]; +int i, nr_to_kill =0 , nr_victims =0 ; +unsigned long pages_found =0 ;
drivers/android/simple_lmk.c line 177
Quoted 10 lines
drivers/android/simple_lmk.c new file +10
-0+atomic_t victim_count; + +/* +* Hold the tasklist lock so tasks don't disappear while scanning. This +* is preferred to holding an RCU read lock so that the list of tasks +* is guaranteed to be up to date. Keep preemption disabled until the +* SIGKILLs are sent so the victim kill process isn't interrupted. +*/ +read_lock (&tasklist_lock); +preempt_disable ();
read_lock() disables preemption, every task_lock() too, so this looks unnecessary.
Quoted 16 lines
drivers/android/simple_lmk.c new file +16
-0+for (i =1 ; i <ARRAY_SIZE (adj_prio); i++) { + pages_found +=find_victims (victims, &nr_victims,MAX_VICTIMS , + adj_prio[i], adj_prio[i -1 ]); +if (pages_found >= pages_needed || nr_victims ==MAX_VICTIMS ) +break ; + } + +/* +* Calculate the number of tasks that need to be killed and quickly +* release the references to those that'll live. +*/ +for (i =0 , pages_found =0 ; i < nr_victims; i++) { +struct victim_info *victim = &victims[i]; +struct task_struct *vtsk = victim->tsk; + +/* The victims' mm lock is taken in find_victims; release it */
drivers/android/simple_lmk.c line 203
Quoted 10 lines
drivers/android/simple_lmk.c new file +10
-0+if (pages_found >= pages_needed) { +task_unlock (vtsk); +continue ; + } + +/* +* Grab a reference to the victim so it doesn't disappear after +* the tasklist lock is released. +*/ +get_task_struct (vtsk);
The comment doesn't look correct. the victim can't dissapear until task_unlock() below, it can't pass exit_mm().
Quoted 11 lines
drivers/android/simple_lmk.c new file +11
-0+ pages_found += victim->size; + nr_to_kill++; + } +read_unlock (&tasklist_lock); + +/* Kill the victims */ + victim_count = (atomic_t )ATOMIC_INIT (nr_to_kill); +for (i =0 ; i < nr_to_kill; i++) { +struct victim_info *victim = &victims[i]; +struct task_struct *vtsk = victim->tsk; +
drivers/android/simple_lmk.c line 224
Quoted 10 lines
drivers/android/simple_lmk.c new file +10
-0+pr_info ("Killing %s with adj %d to free %lu kiB\n" , vtsk->comm, + vtsk->signal->oom_score_adj, + victim->size << (PAGE_SHIFT -10 )); + +/* Configure the victim's mm to notify us when it's freed */ + vtsk->mm->slmk_waitq = &victim_waitq; + vtsk->mm->slmk_counter = &victim_count; + +/* Accelerate the victim's death by forcing the kill signal */ +do_send_sig_info (SIGKILL ,SIG_INFO_TYPE , vtsk, true);
^^^^
this should be PIDTYPE_TGID
drivers/android/simple_lmk.c line 229
Quoted 5 lines
drivers/android/simple_lmk.c new file +5
-0+ +/* Finally release the victim's mm lock */ +task_unlock (vtsk); + } +preempt_enable_no_resched ();
See above. And I don't understand how can _no_resched() really help...
Quoted 7 lines
drivers/android/simple_lmk.c new file +7
-0+ +/* Try to speed up the death process now that we can schedule again */ +for (i =0 ; i < nr_to_kill; i++) { +struct task_struct *vtsk = victims[i].tsk; + +/* Increase the victim's priority to make it die faster */ +set_user_nice (vtsk,MIN_NICE );
drivers/android/simple_lmk.c line 246
Quoted 10 lines
drivers/android/simple_lmk.c new file +10
-0+ +/* Allow the victim to run on any CPU */ +set_cpus_allowed_ptr (vtsk, cpu_all_mask); + +/* Finally release the victim reference acquired earlier */ +put_task_struct (vtsk); + } + +/* Wait until all the victims die */ +wait_event (victim_waitq, !atomic_read (&victim_count));
Can't we avoid the new slmk_waitq/slmk_counter members in mm_struct?
I mean, can't we export victim_waitq and victim_count and, say, set/test MMF_OOM_VICTIM. In fact I think you should try to re-use mark_oom_victim() at least.
Oleg.
Quoted 1 line · Oleg Nesterov
I am not going to comment the intent, but to be honest I am skeptical too.
The general sentiment has been that this is a really bad idea, but I'm just a frustrated Android user who wants his phone to not require mountains of zRAM only to still manage memory poorly. Until I can go out and buy a non-Pixel phone that uses PSI to make these decisions (and does a good job of it), I'm going to stick to my hacky little driver for my personal devices. Many others who like to hack their Android devices to make them last longer will probably find value in this as well, since there are millions of people who use devices that'll never seen any of PSI ported to their ancient 3.x kernels.
And yes, I know this would never be accepted to upstream in a million years. I mostly wanted some code review and guidance, since mm code is pretty tricky :)
Quoted 29 lines · Sultan Alsawaf
Quoted 15 lines
drivers/android/simple_lmk.c new file +14
-0+static unsigned long find_victims (struct victim_info *varr,int *vindex, +int vmaxlen,int min_adj,int max_adj) +{ +unsigned long pages_found =0 ; +int old_vindex = *vindex; +struct task_struct *tsk; + +for_each_process (tsk) { +struct task_struct *vtsk; +unsigned long tasksize; +short oom_score_adj; + +/* Make sure there's space left in the victim array */ +if (*vindex == vmaxlen)drivers/android/simple_lmk.c line 119
Quoted 10 lines
drivers/android/simple_lmk.c new file +10
-0+break ; + +/* Don't kill current, kthreads, init, or duplicates */ +if (same_thread_group (tsk, current) || + tsk->flags &PF_KTHREAD || +is_global_init (tsk) || +vtsk_is_duplicate (varr, *vindex, tsk)) +continue ; + + vtsk =find_lock_task_mm (tsk);Did you test this patch with lockdep enabled?
Quoted 3 lines
If I read the patch correctly, lockdep should complain.
vtsk_is_duplicate()ensures that we do not take the same ->alloc_lock twice or more, but lockdep can't know this.
Yeah, lockdep is fine with this, at least on 4.4.
Quoted 11 lines
Quoted 11 lines
drivers/android/simple_lmk.c new file +11
-0+static void scan_and_kill (unsigned long pages_needed) +{ +static DECLARE_WAIT_QUEUE_HEAD (victim_waitq); +struct victim_info victims[MAX_VICTIMS ]; +int i, nr_to_kill =0 , nr_victims =0 ; +unsigned long pages_found =0 ; +atomic_t victim_count; + +/* +* Hold the tasklist lock so tasks don't disappear while scanning. This +* is preferred to holding an RCU read lock so that the list of tasks
Quoted 8 lines
drivers/android/simple_lmk.c line 177
Quoted 5 lines
drivers/android/simple_lmk.c new file +5
-0+* is guaranteed to be up to date. Keep preemption disabled until the +* SIGKILLs are sent so the victim kill process isn't interrupted. +*/ +read_lock (&tasklist_lock); +preempt_disable ();
read_lock()disables preemption, everytask_lock()too, so this looks unnecessary.
Good point.
Quoted 21 lines
Quoted 21 lines
drivers/android/simple_lmk.c new file +21
-0+for (i =1 ; i <ARRAY_SIZE (adj_prio); i++) { + pages_found +=find_victims (victims, &nr_victims,MAX_VICTIMS , + adj_prio[i], adj_prio[i -1 ]); +if (pages_found >= pages_needed || nr_victims ==MAX_VICTIMS ) +break ; + } + +/* +* Calculate the number of tasks that need to be killed and quickly +* release the references to those that'll live. +*/ +for (i =0 , pages_found =0 ; i < nr_victims; i++) { +struct victim_info *victim = &victims[i]; +struct task_struct *vtsk = victim->tsk; + +/* The victims' mm lock is taken in find_victims; release it */ +if (pages_found >= pages_needed) { +task_unlock (vtsk); +continue ; + } +
Quoted 8 lines
drivers/android/simple_lmk.c line 203
Quoted 5 lines
drivers/android/simple_lmk.c new file +5
-0+/* +* Grab a reference to the victim so it doesn't disappear after +* the tasklist lock is released. +*/ +get_task_struct (vtsk);The comment doesn't look correct. the victim can't dissapear until
task_unlock()below, it can't passexit_mm().
I was always unsure about this and decided to hold a reference to the task_struct to be safe. Thanks for clearing that up.
Quoted 22 lines
Quoted 11 lines
drivers/android/simple_lmk.c new file +11
-0+ pages_found += victim->size; + nr_to_kill++; + } +read_unlock (&tasklist_lock); + +/* Kill the victims */ + victim_count = (atomic_t )ATOMIC_INIT (nr_to_kill); +for (i =0 ; i < nr_to_kill; i++) { +struct victim_info *victim = &victims[i]; +struct task_struct *vtsk = victim->tsk; +drivers/android/simple_lmk.c line 224
Quoted 10 lines
drivers/android/simple_lmk.c new file +10
-0+pr_info ("Killing %s with adj %d to free %lu kiB\n" , vtsk->comm, + vtsk->signal->oom_score_adj, + victim->size << (PAGE_SHIFT -10 )); + +/* Configure the victim's mm to notify us when it's freed */ + vtsk->mm->slmk_waitq = &victim_waitq; + vtsk->mm->slmk_counter = &victim_count; + +/* Accelerate the victim's death by forcing the kill signal */ +do_send_sig_info (SIGKILL ,SIG_INFO_TYPE , vtsk, true);^^^^
Quoted 1 line
this should be
PIDTYPE_TGID
Thanks, I didn't realize the last argument to do_send_sig_info changed in newer kernels. The compiler didn't complain, so it went over my head.
Quoted 7 lines
drivers/android/simple_lmk.c line 229
Quoted 5 lines
drivers/android/simple_lmk.c new file +5
-0+ +/* Finally release the victim's mm lock */ +task_unlock (vtsk); + } +preempt_enable_no_resched ();See above. And I don't understand how can
_no_resched()really help...
Yeah, good point.
Quoted 20 lines
Quoted 7 lines
drivers/android/simple_lmk.c new file +7
-0+ +/* Try to speed up the death process now that we can schedule again */ +for (i =0 ; i < nr_to_kill; i++) { +struct task_struct *vtsk = victims[i].tsk; + +/* Increase the victim's priority to make it die faster */ +set_user_nice (vtsk,MIN_NICE );drivers/android/simple_lmk.c line 246
Quoted 10 lines
drivers/android/simple_lmk.c new file +10
-0+ +/* Allow the victim to run on any CPU */ +set_cpus_allowed_ptr (vtsk, cpu_all_mask); + +/* Finally release the victim reference acquired earlier */ +put_task_struct (vtsk); + } + +/* Wait until all the victims die */ +wait_event (victim_waitq, !atomic_read (&victim_count));Can't we avoid the new slmk_waitq/slmk_counter members in
mm_struct?
Quoted 3 lines
I mean, can't we export
victim_waitqandvictim_countand, say, set/testMMF_OOM_VICTIM. In fact I think you should try to re-usemark_oom_victim()at least.
This makes the patch less portable across different kernel versions, which is kind of one of its major goals.
Thanks for the code review, Oleg.
Sultan
Quoted 10 lines · Sultan Alsawaf
Quoted 5 lines · Oleg Nesterov
Did you test this patch with lockdep enabled?
If I read the patch correctly, lockdep should complain.
vtsk_is_duplicate()ensures that we do not take the same ->alloc_lock twice or more, but lockdep can't know this.Yeah, lockdep is fine with this, at least on 4.4.
Impossible ;) I bet lockdep should report the deadlock as soon as find_victims() calls find_lock_task_mm() when you already have a locked victim.
Nevermind, I guess this code won't run with lockdep enabled...
As for https://github.com/kerneltoast/android_kernel_google_wahoo/commit/afc8c9bf2dbde95941253c168d1adb64cfa2e3ad
Well,
mmdrop(mm); simple_lmk_mm_freed(mm);
looks racy because mmdrop(mm) can free this mm_struct. Yes, simple_lmk_mm_freed() does not dereference this pointer, but the same memory can be re-allocated as another ->mm for the new task which can be found by find_victims(), and _in theory_ this all can happen in between, so the "victims[i].mm == mm" can be false positive.
And this also means that simple_lmk_mm_freed() should clear victims[i].mm when it detects "victims[i].mm == mm", otherwise we have the same theoretical race, victims_to_kill is only cleared when the last victim goes away.
Another nit... you can drop tasklist_lock right after the 1st "find_victims" loop.
And it seems that you do not really need to walk the "victims" array twice after that, you can do everything in a single loop, but this is cosmetic.
Oleg.
Quoted 2 lines · Oleg Nesterov
Impossible ;) I bet lockdep should report the deadlock as soon as
find_victims()callsfind_lock_task_mm()when you already have a locked victim.
I hope you're not a betting man ;)
With the following configured:CONFIG_DEBUG_RT_MUTEXES=yCONFIG_DEBUG_SPINLOCK=y
# CONFIG_DEBUG_SPINLOCK_BITE_ON_BUG is not setCONFIG_DEBUG_SPINLOCK_PANIC_ON_BUG=yCONFIG_DEBUG_MUTEXES=yCONFIG_DEBUG_WW_MUTEX_SLOWPATH=yCONFIG_DEBUG_LOCK_ALLOC=yCONFIG_PROVE_LOCKING=yCONFIG_LOCKDEP=yCONFIG_LOCK_STAT=yCONFIG_DEBUG_LOCKDEP=yCONFIG_DEBUG_ATOMIC_SLEEP=y
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
# CONFIG_LOCK_TORTURE_TEST is not set
And a printk added in vtsk_is_duplicate() to print when a duplicate is detected, and my phone's memory cut in half to make simple_lmk do something, this is what I observed:
taimen:/ # dmesg | grep lockdep
[ 0.000000] \x09RCU lockdep checking is enabled. taimen:/ # dmesg | grep simple_lmk [ 23.211091] simple_lmk: Killing android.carrier with adj 906 to free 37420 kiB [ 23.211160] simple_lmk: Killing oadcastreceiver with adj 906 to free 36784 kiB [ 23.248457] simple_lmk: Killing .apps.translate with adj 904 to free 45884 kiB [ 23.248720] simple_lmk: Killing ndroid.settings with adj 904 to free 42868 kiB [ 23.313417] simple_lmk: DUPLICATE VTSK! [ 23.313440] simple_lmk: Killing ndroid.keychain with adj 906 to free 33680 kiB [ 23.313513] simple_lmk: Killing com.whatsapp with adj 904 to free 51436 kiB [ 34.646695] simple_lmk: DUPLICATE VTSK! [ 34.646717] simple_lmk: Killing ndroid.apps.gcs with adj 906 to free 37956 kiB [ 34.646792] simple_lmk: Killing droid.apps.maps with adj 904 to free 63600 kiB taimen:/ # dmesg | grep lockdep [ 0.000000] \x09RCU lockdep checking is enabled.
taimen:/ #
Quoted 6 lines
As for https://github.com/kerneltoast/android_kernel_google_wahoo/commit/afc8c9bf2dbde95941253c168d1adb64cfa2e3ad
Well,mmdrop(mm); simple_lmk_mm_freed(mm);
Quoted 8 lines
looks racy because mmdrop(mm) can free this
mm_struct. Yes,simple_lmk_mm_freed()does not dereference this pointer, but the same memory can be re-allocated as another ->mm for the new task which can be found byfind_victims(), and_intheory_this all can happen in between, so the "victims[i].mm == mm" can be false positive.And this also means that
simple_lmk_mm_freed()should clear victims[i].mm when it detects "victims[i].mm == mm", otherwise we have the same theoretical race,victims_to_killis only cleared when the last victim goes away.
Fair point. Putting simple_lmk_mm_freed(mm) right before mmdrop(mm), and sprinkling in a cmpxchg in simple_lmk_mm_freed() should fix that up.
Quoted 1 line
Another nit... you can drop
tasklist_lockright after the 1st "find_victims" loop.
True!
Quoted 2 lines
And it seems that you do not really need to walk the "victims" array twice after that, you can do everything in a single loop, but this is cosmetic.
Won't this result in potentially holding the task lock way longer than necessary for multiple tasks that aren't going to be killed?
Thanks,
Sultan
Quoted 6 lines · Sultan Alsawaf
Quoted 2 lines · Oleg Nesterov
Impossible ;) I bet lockdep should report the deadlock as soon as
find_victims()callsfind_lock_task_mm()when you already have a locked victim.I hope you're not a betting man ;)
I am starting to think I am ;)
If you have task1 != task2 this code
task_lock(task1); task_lock(task2);
should trigger print_deadlock_bug(), task1->alloc_lock and task2->alloc_lock are the "same" lock from lockdep pov, held_lock's will have the same hlock_class().
Quoted 1 line
CONFIG_PROVE_LOCKING=y
OK,
Quoted 1 line
And a printk added in
vtsk_is_duplicate()to print when a duplicate is detected,
in this case find_lock_task_mm() won't be called, and this is what saves us from the actual deadlock.
Quoted 4 lines
and my phone's memory cut in half to make
simple_lmkdo something, this is what I observed:
taimen:/ # dmesg | grep lockdep[ 0.000000] \x09RCU lockdep checking is enabled.
this reports that CONFIG_PROVE_RCU is enabled ;)
Quoted 3 lines
taimen:/ # dmesg | grep
simple_lmk[ 23.211091] simple_lmk: Killing android.carrier with adj 906 to free 37420 kiB [ 23.211160] simple_lmk: Killing oadcastreceiver with adj 906 to free 36784 kiB
yes, looks like simple_lmk has at least 2 locked victims. And I have no idea why you do not see anything else in dmesg. May be debug_locks_off() was already called.
But see above, "grep lockdep" won't work. Perhaps you can do "grep -e WARNING -e BUG -e locking".
Oleg.
Quoted 9 lines · Oleg Nesterov
I am starting to think I am ;)
If you have task1 != task2 this code
task_lock(task1); task_lock(task2);should trigger
print_deadlock_bug(),task1->alloc_lockandtask2->alloc_lockare the "same" lock from lockdep pov, held_lock's will have the samehlock_class().
Okay, I've stubbed out debug_locks_off(), and lockdep is now complaining about a bunch of false positives so it is _really_ enabled this time. I grepped for lockdep last time to try and find a concise way to show over email that lockdep didn't complain, but that backfired. Here is better evidence:
taimen:/ # dmesg | grep simple_lmk
[ 58.349917] simple_lmk: Killing droid.deskclock with adj 906 to free 47548 KiB [ 58.354748] simple_lmk: Killing .android.dialer with adj 906 to free 36576 KiB [ 58.355030] simple_lmk: Killing rbandroid.sleep with adj 904 to free 50016 KiB [ 58.582833] simple_lmk: Killing oadcastreceiver with adj 904 to free 43044 KiB [ 58.587731] simple_lmk: Killing .apps.wellbeing with adj 902 to free 48128 KiB [ 58.588084] simple_lmk: Killing android.carrier with adj 902 to free 43636 KiB [ 58.671857] simple_lmk: Killing ndroid.keychain with adj 902 to free 39992 KiB [ 58.675622] simple_lmk: Killing gs.intelligence with adj 900 to free 49572 KiB [ 58.675770] simple_lmk: Killing le.modemservice with adj 800 to free 41976 KiB [ 58.762060] simple_lmk: Killing ndroid.settings with adj 700 to free 74708 KiB [ 58.763238] simple_lmk: Killing roid.apps.turbo with adj 700 to free 54660 KiB [ 58.873337] simple_lmk: Killing d.process.acore with adj 700 to free 48540 KiB [ 58.873513] simple_lmk: Killing d.process.media with adj 500 to free 46188 KiB [ 58.873713] simple_lmk: Killing putmethod.latin with adj 200 to free 67304 KiB [ 59.014046] simple_lmk: Killing android.vending with adj 201 to free 54900 KiB [ 59.017623] simple_lmk: Killing rak.mibandtools with adj 201 to free 44552 KiB [ 59.018423] simple_lmk: Killing eport.trebuchet with adj 100 to free 106208 KiB [ 59.223633] simple_lmk: Killing id.printspooler with adj 900 to free 39664 KiB [ 59.223762] simple_lmk: Killing gle.android.gms with adj 100 to free 64176 KiB [ 70.955204] simple_lmk: Killing .apps.translate with adj 906 to free 47564 KiB [ 70.955633] simple_lmk: Killing cloudprint:sync with adj 906 to free 31932 KiB [ 70.955787] simple_lmk: Killing oid.apps.photos with adj 904 to free 50204 KiB [ 71.060789] simple_lmk: Killing ecamera.android with adj 906 to free 32232 KiB [ 71.061074] simple_lmk: Killing webview_service with adj 906 to free 26028 KiB [ 71.061199] simple_lmk: Killing com.whatsapp with adj 904 to free 49484 KiB [ 71.190625] simple_lmk: Killing rbandroid.sleep with adj 906 to free 54724 KiB [ 71.190775] simple_lmk: Killing android.vending with adj 906 to free 39848 KiB [ 71.191303] simple_lmk: Killing m.facebook.orca with adj 904 to free 72296 KiB [ 71.342042] simple_lmk: Killing droid.deskclock with adj 902 to free 49284 KiB [ 71.342240] simple_lmk: Killing .apps.wellbeing with adj 900 to free 47632 KiB [ 71.342529] simple_lmk: Killing le.modemservice with adj 800 to free 33648 KiB [ 71.482391] simple_lmk: Killing d.process.media with adj 800 to free 40676 KiB [ 71.482511] simple_lmk: Killing rdog.challegram with adj 700 to free 71920 KiB
taimen:/ #
The first simple_lmk message appears at 58.349917. And based on the timestamps, it's clear that simple_lmk called task_lock() for multiple different tasks, which is the pattern you think should cause lockdep to complain. But here is the full dmesg starting from that point:
[ 58.349917] simple_lmk: Killing droid.deskclock with adj 906 to free 47548 KiB [ 58.354748] simple_lmk: Killing .android.dialer with adj 906 to free 36576 KiB [ 58.355030] simple_lmk: Killing rbandroid.sleep with adj 904 to free 50016 KiB [ 58.432654] binder_alloc: 2284: binder_alloc_buf failed to map pages in userspace, no vma [ 58.432671] binder: 1206:1218 transaction failed 29189/-3, size 76-0 line 3189 [ 58.582833] simple_lmk: Killing oadcastreceiver with adj 904 to free 43044 KiB [ 58.587731] simple_lmk: Killing .apps.wellbeing with adj 902 to free 48128 KiB [ 58.588084] simple_lmk: Killing android.carrier with adj 902 to free 43636 KiB [ 58.590785] binder: undelivered transaction 58370, process died. [ 58.671857] simple_lmk: Killing ndroid.keychain with adj 902 to free 39992 KiB [ 58.675622] simple_lmk: Killing gs.intelligence with adj 900 to free 49572 KiB [ 58.675770] simple_lmk: Killing le.modemservice with adj 800 to free 41976 KiB [ 58.736678] binder: undelivered transaction 58814, process died. [ 58.736733] binder: release 3099:3128 transaction 57832 in, still active [ 58.736744] binder: send failed reply for transaction 57832 to 1876:3090 [ 58.736761] binder: undelivered TRANSACTION_COMPLETE [ 58.736766] binder: undelivered transaction 58752, process died. [ 58.762060] simple_lmk: Killing ndroid.settings with adj 700 to free 74708 KiB [ 58.763238] simple_lmk: Killing roid.apps.turbo with adj 700 to free 54660 KiB [ 58.863590] binder: release 1876:3089 transaction 58117 out, still active [ 58.863606] binder: undelivered TRANSACTION_COMPLETE [ 58.873337] simple_lmk: Killing d.process.acore with adj 700 to free 48540 KiB [ 58.873513] simple_lmk: Killing d.process.media with adj 500 to free 46188 KiB [ 58.873713] simple_lmk: Killing putmethod.latin with adj 200 to free 67304 KiB [ 59.014046] simple_lmk: Killing android.vending with adj 201 to free 54900 KiB [ 59.017623] simple_lmk: Killing rak.mibandtools with adj 201 to free 44552 KiB [ 59.018423] simple_lmk: Killing eport.trebuchet with adj 100 to free 106208 KiB [ 59.028460] binder: 1206:1206 transaction failed 29189/-22, size 100-0 line 3052 [ 59.142592] binder_alloc: 2814: binder_alloc_buf, no vma [ 59.142620] binder: 1206:1218 transaction failed 29189/-3, size 76-0 line 3189 [ 59.223633] simple_lmk: Killing id.printspooler with adj 900 to free 39664 KiB [ 59.223762] simple_lmk: Killing gle.android.gms with adj 100 to free 64176 KiB [ 59.540176] binder: undelivered transaction 59447, process died. [ 59.540763] binder: undelivered transaction 59446, process died. [ 59.815404] binder: 1206:3140 transaction failed 29189/0, size 12-0 line 2992 [ 59.815418] binder: send failed reply for transaction 58117, target dead [ 60.977609] binder: 2105:2308 transaction failed 29189/-22, size 168-0 line 3052 [ 63.040202] FG: fg_get_battery_temp: batt temperature original:350, tuned:309 [ 63.040219] lge_battery: bm_watch_work: PRESENT:1, CHG_STAT:1, THM_STAT:2, BAT_TEMP:309, BAT_VOLT:4148182, VOTE_CUR:1000000, SET_CUR:1000000 [ 63.076086] msm-dwc3 a800000.ssusb: Avail curr from USB = 2 [ 63.077278] PMI: smblib_handle_switcher_power_ok: Weak charger detected: voting 500mA ICL [ 63.080014] PMI: smblib_handle_switcher_power_ok: Reverse boost detected: voting 0mA to suspend input [ 63.081886] FG: fg_get_battery_temp: batt temperature original:350, tuned:310 [ 63.093639] fts_touch 5-0049: [FTS] Received Charger Disconnected Event [ 63.104656] healthd: battery l=100 v=4148 t=31.0 h=2 st=3 c=748 fc=3229000 cc=289 chg= [ 63.122546] healthd: battery l=100 v=4148 t=31.0 h=2 st=3 c=748 fc=3229000 cc=289 chg= [ 63.135620] FG: fg_get_battery_temp: batt temperature original:350, tuned:310 [ 63.156383] FG: fg_get_battery_temp: batt temperature original:350, tuned:310 [ 63.156897] FG: fg_get_battery_temp: batt temperature original:350, tuned:310 [ 63.160481] FG: fg_get_battery_temp: batt temperature original:350, tuned:310 [ 63.185029] healthd: battery l=100 v=4148 t=31.0 h=2 st=3 c=748 fc=3229000 cc=289 chg= [ 63.189015] healthd: battery l=100 v=4148 t=31.0 h=2 st=3 c=748 fc=3229000 cc=289 chg= [ 63.212484] lge_battery: bm_check_status: wake_unlocked: present[0] chg_state[2] vbus[0] [ 63.213039] FG: fg_get_battery_temp: batt temperature original:350, tuned:310 [ 63.231096] FG: fg_get_battery_temp: batt temperature original:350, tuned:310 [ 63.233981] healthd: battery l=100 v=4148 t=31.0 h=2 st=3 c=748 fc=3229000 cc=289 chg= [ 63.234663] msm-dwc3 a800000.ssusb: dwc3_msm_suspend: Calling suspend 1996 [ 63.249755] msm-dwc3 a800000.ssusb: DWC3 in low power mode [ 63.250247] healthd: battery l=100 v=4148 t=31.0 h=2 st=3 c=748 fc=3229000 cc=289 chg= [ 63.250430] android_work: sent uevent USB_STATE=DISCONNECTED [ 63.294456] msm-dwc3 a800000.ssusb: Avail curr from USB = 0 [ 70.492114] binder: undelivered transaction 86938, process died. [ 70.955204] simple_lmk: Killing .apps.translate with adj 906 to free 47564 KiB [ 70.955633] simple_lmk: Killing cloudprint:sync with adj 906 to free 31932 KiB [ 70.955787] simple_lmk: Killing oid.apps.photos with adj 904 to free 50204 KiB [ 71.060789] simple_lmk: Killing ecamera.android with adj 906 to free 32232 KiB [ 71.061074] simple_lmk: Killing webview_service with adj 906 to free 26028 KiB [ 71.061199] simple_lmk: Killing com.whatsapp with adj 904 to free 49484 KiB [ 71.164996] binder: undelivered transaction 87881, process died. [ 71.190625] simple_lmk: Killing rbandroid.sleep with adj 906 to free 54724 KiB [ 71.190775] simple_lmk: Killing android.vending with adj 906 to free 39848 KiB [ 71.191303] simple_lmk: Killing m.facebook.orca with adj 904 to free 72296 KiB [ 71.342042] simple_lmk: Killing droid.deskclock with adj 902 to free 49284 KiB [ 71.342240] simple_lmk: Killing .apps.wellbeing with adj 900 to free 47632 KiB [ 71.342529] simple_lmk: Killing le.modemservice with adj 800 to free 33648 KiB [ 71.482391] simple_lmk: Killing d.process.media with adj 800 to free 40676 KiB [ 71.482511] simple_lmk: Killing rdog.challegram with adj 700 to free 71920 KiB
No lockdep warnings!
Sultan
Steven Rostedt
reviewerQuoted 10 lines · On Mon, 13 May 2019 09:45:55 -0700 Sultan Alsawaf <[email protected]>…
Quoted 9 lines · Oleg Nesterov
I am starting to think I am ;)
If you have task1 != task2 this code
task_lock(task1); task_lock(task2);should trigger
print_deadlock_bug(),task1->alloc_lockandtask2->alloc_lockare the "same" lock from lockdep pov, held_lock's will have the samehlock_class().
OK, this has gotten my attention.
This thread is quite long, do you have a git repo I can look at, and also where is the first task_lock() taken before the find_lock_task_mm()?
-- Steve
Quoted 7 lines · Steven Rostedt
OK, this has gotten my attention.
This thread is quite long, do you have a git repo I can look at, and also where is the first
task_lock()taken before thefind_lock_task_mm()?-- Steve
Hi Steve,
This is the git repo I work on: https://github.com/kerneltoast/android_kernel_google_wahoo
With the newest simple_lmk iteration being this commit: https://github.com/kerneltoast/android_kernel_google_wahoo/commit/6b145b8c28b39f7047393169117f72ea7387d91c
This repo is based off the 4.4 kernel that Google ships on the Pixel 2/2XL.
simple_lmk iterates through the entire task list more than once and locks potential victims using find_lock_task_mm(). It keeps these potential victims locked across the multiple times that the task list is iterated.
The locking pattern that Oleg said should cause lockdep to complain is that iterating through the entire task list more than once can lead to locking the same task that was locked earlier with find_lock_task_mm(), and thus deadlock. But there is a check in simple_lmk that avoids locking potential victims that were already found, which avoids the deadlock, but lockdep doesn't know about the check (which is done with vtsk_is_duplicate()) and should therefore complain.
Lockdep does not complain though.
Sultan
Quoted 6 lines · Sultan Alsawaf
Quoted 4 lines · Oleg Nesterov
I am starting to think I am ;)
If you have task1 != task2 this code
Quoted 8 lines
Quoted 5 lines
task_lock(task1); task_lock(task2);should trigger
print_deadlock_bug(),task1->alloc_lockandtask2->alloc_lockare the "same" lock from lockdep pov, held_lock's will have the samehlock_class().Okay, I've stubbed out
debug_locks_off(), and lockdep is now complaining about a bunch of false positives so it is _really_ enabled this time.
Could you explain in detail what exactly did you do and what do you see in dmesg?
Just in case, lockdep complains only once, print_circular_bug() does debug_locks_off() so it it has already reported another false positive __lock_acquire() will simply return after that.
Oleg.