module: fix virtual memory wasted on finit_module()
Subsystems (3)
Luis Chamberlain
maintainercover module: fix virtual memory wasted on finit_module()
The graph from my v3 patch series [0] which tries to resolve the virtual memory lost bytes due to duplicates says it all:
+----------------------------------------------------------------------------+
14GB |-+ + + + + *+ +-|
| **** |
| *** |
| ** |
12GB |-+ ** +-|
| ** |
| ** |
| ** |
| ** |
10GB |-+ ** +-|
| ** |
| ** |
| ** |
8GB |-+ ** +-|
waste | ** ### |
| ** #### |
| ** ####### |
6GB |-+ **** #### +-|
| * #### |
| * #### |
| ***** #### |
4GB |-+ ** #### +-|
| ** #### |
| ** #### |
| ** #### |
2GB |-+ ** ##### +-|
| * #### |
| * #### Before ******* |
| **## + + + + After ####### |
+----------------------------------------------------------------------------+
0 50 100 150 200 250 300CPUs count
So we really need to debug to see WTF, because really, WTF. The first patch tries to answer the question if the issue is module auto-loading being abused and that causing the issues. The patch proves that the answer is no, but it does also help us find *a few* requests which can get a bit of love to avoid duplicates. My system at least found one. So it adds a debugging facility to let you do that.
As I was writing the commit log for my first patch series [0] I was noting that this is it... and the obvious conclusion is that the culprit is udev issuing requests per CPU for tons of modules. I didn't feel comfortable in writing that this is it and we can't really do anything before really trying hard. So I gave it a good 'ol college try. At first I wondered if we could use file descriptor hints to just exlude users early on boot before SYSTEM_RUNNING. I couldn't find much, but if there are some ways to do that -- then the last patch can be simplified to do just that. The second patch proves essentially that we can just send -EBUSY to duplicate requests, at least for duplicate module loads and the world doesn't fall apart. It *would* solve the issue. The patch however borrows tons of the code from the first, and if we're realy going to rely on something like that we may as well share. But I'm hopeful that perhaps there are some jucier file descriptor tricks we can use to just make a file mutually exlusivive and introduce a new kread which lets finit_module() use that. The saving grace is that at least all finit_module() calls *wait*, contray to request_module() calls and so the solution can be much simpler.
The end result is 0 wasted virtual memory bytes.
Any ideas how not to make patch 2 suck as-is ?
Yes -- we can also go fix udev, or libkmod, and that's what should be done. However, it seems silly to not fix if the fix is as trivial as patch 2 demonstrates.
If you want to test / muck with all this you can use my branch 20230413-module-alloc-opts [1]:
[0] https://lkml.kernel.org/r/[email protected]
[1] https://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/linux.git/log/?h=20230413-module-alloc-opts
Luis Chamberlain (2):
module: add debugging auto-load duplicate module support
kread: avoid duplicates
fs/kernel_read_file.c | 150 +++++++++++++++++++++++++ kernel/module/Kconfig | 40 +++++++ kernel/module/Makefile | 1 + kernel/module/dups.c | 234 +++++++++++++++++++++++++++++++++++++++ kernel/module/internal.h | 15 +++ kernel/module/kmod.c | 23 +++- kernel/module/main.c | 6 +- 7 files changed, 463 insertions(+), 6 deletions(-) create mode 100644 kernel/module/dups.c
Signature · 1 line
2.39.2
Luis Chamberlain
maintainerRFC 1/2 module: add debugging auto-load duplicate module support
This adds debugging support to the kernel module auto-loader to easily detect and deal with duplicate module requests. To aid with possible bootup failure issues it will supress the waste in virtual memory when races happen before userspace loads a module and the kernel is still issuing requests for the same module.
Folks debugging virtual memory abuse on bootup can and should enable this to see what WARN()s come on, to see if module auto-loading is to blame for their woes.
--- kernel/module/Kconfig | 40 +++++++ kernel/module/Makefile | 1 + kernel/module/dups.c | 234 +++++++++++++++++++++++++++++++++++++++ kernel/module/internal.h | 15 +++ kernel/module/kmod.c | 23 +++- 5 files changed, 309 insertions(+), 4 deletions(-) create mode 100644 kernel/module/dups.c
kernel/module/Kconfig +40 -0
@@ -57,6 +57,46 @@ config MODULE_STATS If unsure, say N. +config MODULE_AUTOLOAD_SUPRESS_DUPS + bool "Debug duplicate modules with auto-loading" + help + Module autoloading allows in-kernel code to request modules through + the *request_module*() API calls. This in turn just calls userspace + modprobe. Although modprobe checks to see if a module is already + loaded before trying to load a module there is a small time window in + which multiple duplicate requests can end up in userspace and multiple + modprobe calls race calling finit_module() around the same time for + duplicate modules. The finit_module() system call can consume in the + worst case more than twice the respective module size in virtual + memory for each duplicate module requests. Although duplicate module + requests are non-fatal virtual memory is a limited resource and each + duplicate module request ends up just wasting virtual memory. + + This debugging facility will create WARN() splats for duplicate module + requests to help identify if module auto-loading is the culprit to your + woes. Since virtual memory abuse caused by duplicate module requests + could render a system unusable this functionality will also suppresses + the waste in virtual memory caused by duplicate requests by sharing + races in requests for the same module to a single unified request. + Once a non-wait request_module() call completes a module should be + loaded and modprobe should simply not allow new finit_module() calls. + + Enable this functionality to try to debug virtual memory abuse during + boot on systems and identify if the abuse was due to module + auto-loading. + + If the first module request used request_module_nowait() we cannot + use that as the anchor to wait for duplicate module requests, since + users of request_module() do want a proper return value. If a call + for the same module happened earlier with request_module() though, + then a duplicate request_module_nowait() would be detected. + + You want to enable this if you want to debug and see if duplicate + module auto-loading might be causing virtual memory abuse during + bootup. A kernel trace will be provided for each duplicate request. + + Disable this if you are on production. + endif # MODULE_DEBUG config MODULE_FORCE_LOAD
kernel/module/Makefile +1 -0
@@ -10,6 +10,7 @@ KCOV_INSTRUMENT_module.o := n obj-y += main.o obj-y += strict_rwx.o obj-y += kmod.o +obj-$(CONFIG_MODULE_AUTOLOAD_SUPRESS_DUPS) += dups.o obj-$(CONFIG_MODULE_DECOMPRESS) += decompress.o obj-$(CONFIG_MODULE_SIG) += signing.o obj-$(CONFIG_LIVEPATCH) += livepatch.o
kernel/module/dups.c new file +234 -0
234 lines not on this page · raw message
kernel/module/internal.h +15 -0
@@ -3,6 +3,7 @@* * Copyright (C) 2012 Red Hat, Inc. All Rights Reserved. * Written by David Howells ([email protected]) +* Copyright (C) 2023 Luis Chamberlain <[email protected]> */ #include <linux/elf.h>
@@ -217,6 +218,20 @@ static inline void mod_stat_bump_becoming(struct load_info *info, int flags)#endif /* CONFIG_MODULE_STATS */ +#ifdef CONFIG_MODULE_AUTOLOAD_SUPRESS_DUPS +bool kmod_dup_request_exists_wait (char *module_name,bool wait,int *dup_ret); +void kmod_dup_request_announce (char *module_name,int ret); +#else +static inline bool kmod_dup_request_exists_wait (char *module_name,bool wait,int *dup_ret) +{ +return false; +} + +static inline void kmod_dup_request_announce (char *module_name,int ret) +{ +} +#endif + #ifdef CONFIG_MODULE_UNLOAD_TAINT_TRACKING struct mod_unload_taint { struct list_head list;
kernel/module/kmod.c +19 -4
@@ -1,6 +1,9 @@/* * kmod - the kernel module loader +* +* Copyright (C) 2023 Luis Chamberlain <[email protected]> */ +#include <linux/module.h> #include <linux/sched.h> #include <linux/sched/task.h>
@@ -27,6 +30,7 @@#include <linux/uaccess.h> #include <trace/events/module.h> +#include "internal.h" /* * Assuming:
@@ -65,7 +69,7 @@ static void free_modprobe_argv(struct subprocess_info *info)kfree (info->argv); } -static int call_modprobe (char *module_name,int wait) +static int call_modprobe (char *orig_module_name,int wait) {struct subprocess_info *info;static char *envp[] = {
@@ -74,12 +78,14 @@ static int call_modprobe(char *module_name, int wait)"PATH=/sbin:/usr/sbin:/bin:/usr/bin" , NULL }; +char *module_name; +int ret;char **argv =kmalloc (sizeof (char *[5 ]),GFP_KERNEL );if (!argv) goto out; - module_name =kstrdup (module_name,GFP_KERNEL ); + module_name =kstrdup (orig_module_name,GFP_KERNEL );if (!module_name) goto free_argv;
@@ -94,13 +100,16 @@ static int call_modprobe(char *module_name, int wait)if (!info) goto free_module_name; -return call_usermodehelper_exec (info, wait |UMH_KILLABLE ); + ret =call_usermodehelper_exec (info, wait |UMH_KILLABLE ); +kmod_dup_request_announce (orig_module_name, ret); +return ret; free_module_name:kfree (module_name); free_argv:kfree (argv); out: +kmod_dup_request_announce (orig_module_name, -ENOMEM );return -ENOMEM ; }
@@ -124,7 +133,7 @@ int __request_module(bool wait, const char *fmt, ...) {va_list args;char module_name[MODULE_NAME_LEN ]; -int ret; +int ret, dup_ret;/* * We don't allow synchronous module loading from async. Module
@@ -156,8 +165,14 @@ int __request_module(bool wait, const char *fmt, ...)trace_module_request (module_name, wait, _RET_IP_); +if (kmod_dup_request_exists_wait (module_name, wait, &dup_ret)) { + ret = dup_ret; + goto out; + } + ret =call_modprobe (module_name, wait ?UMH_WAIT_PROC :UMH_WAIT_EXEC ); +out:up (&kmod_concurrent_max);return ret;
Signature · 1 line
2.39.2
Luis Chamberlain
maintainerRFC 2/2 kread: avoid duplicates
In reply to Luis ChamberlainWith this we run into 0 wasted virtual memory bytes.
--- fs/kernel_read_file.c | 150 ++++++++++++++++++++++++++++++++++++++++++ kernel/module/main.c | 6 +- 2 files changed, 154 insertions(+), 2 deletions(-)
fs/kernel_read_file.c +150 -0
@@ -4,6 +4,7 @@#include <linux/kernel_read_file.h> #include <linux/security.h> #include <linux/vmalloc.h> +#include <linux/fdtable.h> /** * kernel_read_file() - read file contents into a kernel buffer
@@ -171,17 +172,166 @@ ssize_t kernel_read_file_from_path_initns(const char *path, loff_t offset, }EXPORT_SYMBOL_GPL (kernel_read_file_from_path_initns); +DEFINE_MUTEX (kread_dup_mutex); +static LIST_HEAD (kread_dup_reqs); + +struct kread_dup_req { +struct list_head list; +char name[PATH_MAX ]; +struct completion first_req_done; +struct work_struct complete_work; +struct delayed_work delete_work; +int dup_ret; +}; + +static struct kread_dup_req *kread_dup_request_lookup (char *name) +{ +struct kread_dup_req *kread_req; + +list_for_each_entry_rcu (kread_req, &kread_dup_reqs, list, +lockdep_is_held (&kread_dup_mutex)) { +if (strlen (kread_req->name) ==strlen (name) && + !memcmp (kread_req->name, name,strlen (name))) { +return kread_req; + } + } + +return NULL ; +} + +static void kread_dup_request_delete (struct work_struct *work) +{ +struct kread_dup_req *kread_req; + kread_req = container_of(to_delayed_work(work), struct kread_dup_req, delete_work); + +mutex_lock (&kread_dup_mutex); +list_del_rcu (&kread_req->list); +synchronize_rcu (); +mutex_unlock (&kread_dup_mutex); +kfree (kread_req); +} + +static void kread_dup_request_complete (struct work_struct *work) +{ +struct kread_dup_req *kread_req; + + kread_req = container_of(work, struct kread_dup_req, complete_work); + +complete_all (&kread_req->first_req_done); +queue_delayed_work (system_wq, &kread_req->delete_work,60 *HZ ); +} + +static bool kread_dup_request_exists_wait (char *name,int *dup_ret) +{ +struct kread_dup_req *kread_req, *new_kread_req; +int ret; + +/* +* Pre-allocate the entry in case we have to use it later +* to avoid contention with the mutex. +*/ + new_kread_req =kzalloc (sizeof (*new_kread_req),GFP_KERNEL ); +if (!new_kread_req) +return false; + +memcpy (new_kread_req->name, name,strlen (name)); +INIT_WORK (&new_kread_req->complete_work, kread_dup_request_complete); +INIT_DELAYED_WORK (&new_kread_req->delete_work, kread_dup_request_delete); +init_completion (&new_kread_req->first_req_done); + +mutex_lock (&kread_dup_mutex); + + kread_req =kread_dup_request_lookup (name); +if (!kread_req) { +/* +* There was no duplicate, just add the request so we can +* keep tab on duplicates later. +*/ +//pr_info("New kread request for %s\n", name); +list_add_rcu (&new_kread_req->list, &kread_dup_reqs); +mutex_unlock (&kread_dup_mutex); +return false; + } +mutex_unlock (&kread_dup_mutex); + +/* We are dealing with a duplicate request now */ + +kfree (new_kread_req); + +//pr_warn("kread: duplicate request for file %s\n", name); + + ret =wait_for_completion_state (&kread_req->first_req_done, +TASK_UNINTERRUPTIBLE |TASK_KILLABLE ); +if (ret) { + *dup_ret = ret; +return true; + } + +/* breath */ +schedule_timeout (2 *HZ ); + + *dup_ret = kread_req->dup_ret; + +return true; +} + +void kread_dup_request_announce (char *name,int ret) +{ +struct kread_dup_req *kread_req; + +mutex_lock (&kread_dup_mutex); + + kread_req =kread_dup_request_lookup (name); +if (!kread_req) + goto out; + + kread_req->dup_ret = ret; + +/* +* If we complete() here we may allow duplicate threads +* to continue before the first one that submitted the +* request. We're in no rush but avoid boot delays caused +* by these threads waiting too long. +*/ +queue_work (system_wq, &kread_req->complete_work); + +out: +mutex_unlock (&kread_dup_mutex); +} +ssize_t kernel_read_file_from_fd (int fd,loff_t offset,void **buf,size_t buf_size,size_t *file_size,enum kernel_read_file_id id) {struct fd f =fdget (fd);ssize_t ret = -EBADF ; +char *name, *path; +int dup_ret;if (!f.file || !(f.file->f_mode &FMODE_READ )) goto out; + path =kzalloc (PATH_MAX ,GFP_KERNEL ); +if (!path) +return -ENOMEM ; + + name =file_path (f.file, path,PATH_MAX ); +if (IS_ERR (name)) { + ret =PTR_ERR (name); + goto out_mem; + } + +if (kread_dup_request_exists_wait (name, &dup_ret)) { + ret = -EBUSY ; + goto out_mem; + } + ret =kernel_read_file (f.file, offset, buf, buf_size, file_size, id); + +kread_dup_request_announce (name, ret); + +out_mem: +kfree (path); out:fdput (f);return ret;
kernel/module/main.c +4 -2
@@ -3080,8 +3080,10 @@ SYSCALL_DEFINE3(finit_module, int, fd, const char __user *, uargs, int, flags) len =kernel_read_file_from_fd (fd,0 , &buf,INT_MAX ,NULL ,READING_MODULE );if (len <0 ) { -mod_stat_inc (&failed_kreads); -mod_stat_add_long (len, &invalid_kread_bytes); +if (len != -EBUSY ) { +mod_stat_inc (&failed_kreads); +mod_stat_add_long (len, &invalid_kread_bytes); + }return len; }
Signature · 1 line
2.39.2
Re: [RFC 2/2] kread: avoid duplicates
Quoted 1 line · Luis Chamberlain
With this we run into 0 wasted virtual memory bytes.
This changelog does not make any sense at all, sorry. What are you doing here and why?
Quoted 20 lines
Signed-off-by Luis Chamberlain <[email protected]>--- fs/kernel_read_file.c | 150 ++++++++++++++++++++++++++++++++++++++++++ kernel/module/main.c | 6 +- 2 files changed, 154 insertions(+), 2 deletions(-)fs/kernel_read_file.c +1
-0@@ -4,6 +4,7 @@#include <linux/kernel_read_file.h> #include <linux/security.h> #include <linux/vmalloc.h> +#include <linux/fdtable.h> /** * kernel_read_file() - read file contents into a kernel buffer
fs/kernel_read_file.c ssize_t kernel_read_file_from_path_initns(const char *path, loff_t offset, · line 175
Quoted 4 lines
fs/kernel_read_file.c +1
-0@@ -171,17 +172,166 @@ ssize_t kernel_read_file_from_path_initns(const char *path, loff_t offset, }EXPORT_SYMBOL_GPL (kernel_read_file_from_path_initns); +DEFINE_MUTEX (kread_dup_mutex);
static?
I stopped reading here :)
thanks,
greg k-h
Luis Chamberlain
maintainerRe: [RFC 2/2] kread: avoid duplicates
Quoted 5 lines · Greg KH
Quoted 1 line · Luis Chamberlain
With this we run into 0 wasted virtual memory bytes.
This changelog does not make any sense at all, sorry. What are you doing here and why?
It's an RFC and the cover letter described the motivation looking for ideas for an alternative, and it is the reason I was so terse on the commit log.
Luis
Re: [RFC 2/2] kread: avoid duplicates
In reply to Luis ChamberlainQuoted 1 line · Luis Chamberlain
With this we run into 0 wasted virtual memory bytes.
Avoid what duplicates?
Luis Chamberlain
maintainerRe: [RFC 2/2] kread: avoid duplicates
Quoted 4 lines · Christoph Hellwig
Quoted 1 line · Luis Chamberlain
With this we run into 0 wasted virtual memory bytes.
Avoid what duplicates?
David Hildenbrand had reported that with over 400 CPUs vmap space runs out and it seems it was related to module loading. I took a look and confirmed it. Module loading ends up requiring in the worst case 3 vmalloc allocations, so typically at least twice the size of the module size and in the worst case just add the decompressed module size:
a) initial kernel_read*() call
b) optional module decompression
c) the actual module data copy we will keep
Duplicate module requests that come from userspace end up being thrown in the trash bin, as only one module will be allocated. Although there are checks for a module prior to requesting a module udev still doesn't do the best of a job to avoid that and so we end up with tons of duplicate module requests. We're talking about gigabytes of vmalloc bytes just lost because of this for large systems and megabytes for average systems. So for example with just 255 CPUs we can loose about 13.58 GiB, and for 8 CPUs about 226.53 MiB.
I have patches to curtail 1/2 of that space by doing a check in kernel before we do the allocation in c) if the module is already present. For
a) it is harder because userspace just passes a file descriptor. But since we can get the file path without the vmalloc this RFC suggest maybe we can add a new kernel_read*() for module loading where it makes sense to have only one read happen at a time.
Luis
Re: [RFC 2/2] kread: avoid duplicates
Quoted 17 lines · Luis Chamberlain
Quoted 4 lines · Christoph Hellwig
Quoted 1 line · Luis Chamberlain
With this we run into 0 wasted virtual memory bytes.
Avoid what duplicates?
David Hildenbrand had reported that with over 400 CPUs vmap space runs out and it seems it was related to module loading. I took a look and confirmed it. Module loading ends up requiring in the worst case 3 vmalloc allocations, so typically at least twice the size of the module size and in the worst case just add the decompressed module size:
a) initial kernel_read*() call
b) optional module decompression
c) the actual module data copy we will keep
Quoted 8 lines
Duplicate module requests that come from userspace end up being thrown in the trash bin, as only one module will be allocated. Although there are checks for a module prior to requesting a module udev still doesn't do the best of a job to avoid that and so we end up with tons of duplicate module requests. We're talking about gigabytes of vmalloc bytes just lost because of this for large systems and megabytes for average systems. So for example with just 255 CPUs we can loose about 13.58 GiB, and for 8 CPUs about 226.53 MiB.
How does the memory get "lost"? Shouldn't it be properly freed when the duplicate module load fails?
thanks,
greg k-h
Luis Chamberlain
maintainerRe: [RFC 2/2] kread: avoid duplicates
Quoted 27 lines · Greg KH
Quoted 17 lines · Luis Chamberlain
Quoted 4 lines · Christoph Hellwig
Quoted 1 line · Luis Chamberlain
With this we run into 0 wasted virtual memory bytes.
Avoid what duplicates?
David Hildenbrand had reported that with over 400 CPUs vmap space runs out and it seems it was related to module loading. I took a look and confirmed it. Module loading ends up requiring in the worst case 3 vmalloc allocations, so typically at least twice the size of the module size and in the worst case just add the decompressed module size:
a) initial kernel_read*() call
b) optional module decompression
c) the actual module data copy we will keepQuoted 8 lines
Duplicate module requests that come from userspace end up being thrown in the trash bin, as only one module will be allocated. Although there are checks for a module prior to requesting a module udev still doesn't do the best of a job to avoid that and so we end up with tons of duplicate module requests. We're talking about gigabytes of vmalloc bytes just lost because of this for large systems and megabytes for average systems. So for example with just 255 CPUs we can loose about 13.58 GiB, and for 8 CPUs about 226.53 MiB.
Quoted 2 lines
How does the memory get "lost"? Shouldn't it be properly freed when the duplicate module load fails?
Yes memory gets freed, but since virtual memory space can be limitted it also means you can end up eventually getting to the point -ENOMEMs will happen as you have more CPUS and you cannot use virtual memory for other things during kernel bootup and bootup fails. This is apparently exacerbated with KASAN enabled.
Luis
Re: [RFC 2/2] kread: avoid duplicates
Quoted 28 lines · Luis Chamberlain
Quoted 27 lines · Greg KH
Quoted 25 lines · Luis Chamberlain
Quoted 4 lines · Christoph Hellwig
Quoted 1 line · Luis Chamberlain
With this we run into 0 wasted virtual memory bytes.
Avoid what duplicates?
David Hildenbrand had reported that with over 400 CPUs vmap space runs out and it seems it was related to module loading. I took a look and confirmed it. Module loading ends up requiring in the worst case 3 vmalloc allocations, so typically at least twice the size of the module size and in the worst case just add the decompressed module size:
a) initial kernel_read*() call
b) optional module decompression
c) the actual module data copy we will keepDuplicate module requests that come from userspace end up being thrown in the trash bin, as only one module will be allocated. Although there are checks for a module prior to requesting a module udev still doesn't do the best of a job to avoid that and so we end up with tons of duplicate module requests. We're talking about gigabytes of vmalloc bytes just lost because of this for large systems and megabytes for average systems. So for example with just 255 CPUs we can loose about 13.58 GiB, and for 8 CPUs about 226.53 MiB.
Quoted 8 lines
Quoted 2 lines
How does the memory get "lost"? Shouldn't it be properly freed when the duplicate module load fails?
Yes memory gets freed, but since virtual memory space can be limitted it also means you can end up eventually getting to the point -ENOMEMs will happen as you have more CPUS and you cannot use virtual memory for other things during kernel bootup and bootup fails. This is apparently exacerbated with KASAN enabled.
Then why not just rate-limit the module loader in userspace on such large systems if that's an issue? No kernel changes needed to do that.
thanks,
greg k-h
Luis Chamberlain
maintainerRe: [RFC 2/2] kread: avoid duplicates
Quoted 32 lines · Greg KH
Quoted 31 lines · Luis Chamberlain
Quoted 29 lines · Greg KH
Quoted 17 lines · Luis Chamberlain
Quoted 4 lines · Christoph Hellwig
Quoted 1 line · Luis Chamberlain
With this we run into 0 wasted virtual memory bytes.
Avoid what duplicates?
David Hildenbrand had reported that with over 400 CPUs vmap space runs out and it seems it was related to module loading. I took a look and confirmed it. Module loading ends up requiring in the worst case 3 vmalloc allocations, so typically at least twice the size of the module size and in the worst case just add the decompressed module size:
a) initial kernel_read*() call
b) optional module decompression
c) the actual module data copy we will keepQuoted 8 lines
Duplicate module requests that come from userspace end up being thrown in the trash bin, as only one module will be allocated. Although there are checks for a module prior to requesting a module udev still doesn't do the best of a job to avoid that and so we end up with tons of duplicate module requests. We're talking about gigabytes of vmalloc bytes just lost because of this for large systems and megabytes for average systems. So for example with just 255 CPUs we can loose about 13.58 GiB, and for 8 CPUs about 226.53 MiB.
How does the memory get "lost"? Shouldn't it be properly freed when the duplicate module load fails?
Quoted 8 lines
Quoted 5 lines
Yes memory gets freed, but since virtual memory space can be limitted it also means you can end up eventually getting to the point -ENOMEMs will happen as you have more CPUS and you cannot use virtual memory for other things during kernel bootup and bootup fails. This is apparently exacerbated with KASAN enabled.
Then why not just rate-limit the module loader in userspace on such large systems if that's an issue? No kernel changes needed to do that.
We can certainly just take a stance punt this as a userspace problem. I thought it would be good to see what a kernel style of workaround would look like for us to evluate.
Luis
Re: [RFC 2/2] kread: avoid duplicates
In reply to Luis ChamberlainQuoted 29 lines · Luis Chamberlain
Quoted 4 lines · Christoph Hellwig
Quoted 1 line · Luis Chamberlain
With this we run into 0 wasted virtual memory bytes.
Avoid what duplicates?
David Hildenbrand had reported that with over 400 CPUs vmap space runs out and it seems it was related to module loading. I took a look and confirmed it. Module loading ends up requiring in the worst case 3 vmalloc allocations, so typically at least twice the size of the module size and in the worst case just add the decompressed module size:
a) initial kernel_read*() call
b) optional module decompression
c) the actual module data copy we will keepDuplicate module requests that come from userspace end up being thrown
in the trash bin, as only one module will be allocated. Although there
are checks for a module prior to requesting a module udev still doesn't
do the best of a job to avoid that and so we end up with tons of duplicate module requests. We're talking about gigabytes of vmalloc bytes just lost because of this for large systems and megabytes for average systems. So for example with just 255 CPUs we can loose about 13.58 GiB, and for 8 CPUs about 226.53 MiB.
Quoted 9 lines
I have patches to curtail 1/2 of that space by doing a check in kernel
before we do the allocation in c) if the module is already present. For
a) it is harder because userspace just passes a file descriptor. But since we can get the file path without the vmalloc this RFC suggest maybe we can add a new kernel_read*() for module loading where it makes
sense to have only one read happen at a time.
I'm wondering how difficult it would be to just try to remove the vmallocs in (a) and (b) and operate on a list of pages.
So the operations before module_patient_check_exists() are now:
1. decompressing (vmalloc)
2. signature check (vmalloc)
3. elf_validity_cache_copy()
4. early_mod_check() -> module_patient_check_exists()
Right? Then after that a bunch of arch code and other code outside of modules operates on the vmalloc, so this other code would take a large amount of changes to switch to a list of pages.
But did you consider teaching just 1-3 to operate on a list of pages? And then move module_patient_check_exists() a little up in the order? After module_patient_check_exists() you could vmap() the pages and hand it off to the existing code located all over the place.
Then you can catch the duplicate requests before any vmalloc happens. It also takes (a) and (b) down to one vmalloc even in the normal case, as a side benefit. The changes to the signature check part might be tricky though.
Sorry if this idea is off, I've got a little confused as this series split into all these offshoots series.
Luis Chamberlain
maintainerRe: [RFC 2/2] kread: avoid duplicates
Quoted 40 lines · Rick P
Quoted 29 lines · Luis Chamberlain
Quoted 4 lines · Christoph Hellwig
Quoted 1 line · Luis Chamberlain
With this we run into 0 wasted virtual memory bytes.
Avoid what duplicates?
David Hildenbrand had reported that with over 400 CPUs vmap space runs out and it seems it was related to module loading. I took a look and confirmed it. Module loading ends up requiring in the worst case 3 vmalloc allocations, so typically at least twice the size of the module size and in the worst case just add the decompressed module size:
a) initial kernel_read*() call
b) optional module decompression
c) the actual module data copy we will keepDuplicate module requests that come from userspace end up being thrown
in the trash bin, as only one module will be allocated. Although there
are checks for a module prior to requesting a module udev still doesn't
do the best of a job to avoid that and so we end up with tons of duplicate module requests. We're talking about gigabytes of vmalloc bytes just lost because of this for large systems and megabytes for average systems. So for example with just 255 CPUs we can loose about 13.58 GiB, and for 8 CPUs about 226.53 MiB.Quoted 9 lines
I have patches to curtail 1/2 of that space by doing a check in kernel
before we do the allocation in c) if the module is already present. For
a) it is harder because userspace just passes a file descriptor. But since we can get the file path without the vmalloc this RFC suggest maybe we can add a new kernel_read*() for module loading where it makes
sense to have only one read happen at a time.
Quoted 2 lines
I'm wondering how difficult it would be to just try to remove the vmallocs in (a) and (b) and operate on a list of pages.
Yes I think it's worth long term to do that, if possible with seq reads.
Luis
Luis Chamberlain
maintainerRe: [RFC 2/2] kread: avoid duplicates
Quoted 41 lines · Luis Chamberlain
Quoted 40 lines · Rick P
Quoted 38 lines · Luis Chamberlain
Quoted 4 lines · Christoph Hellwig
Quoted 1 line · Luis Chamberlain
With this we run into 0 wasted virtual memory bytes.
Avoid what duplicates?
David Hildenbrand had reported that with over 400 CPUs vmap space runs out and it seems it was related to module loading. I took a look and confirmed it. Module loading ends up requiring in the worst case 3 vmalloc allocations, so typically at least twice the size of the module size and in the worst case just add the decompressed module size:
a) initial kernel_read*() call
b) optional module decompression
c) the actual module data copy we will keepDuplicate module requests that come from userspace end up being thrown
in the trash bin, as only one module will be allocated. Although there
are checks for a module prior to requesting a module udev still doesn't
do the best of a job to avoid that and so we end up with tons of duplicate module requests. We're talking about gigabytes of vmalloc bytes just lost because of this for large systems and megabytes for average systems. So for example with just 255 CPUs we can loose about 13.58 GiB, and for 8 CPUs about 226.53 MiB.I have patches to curtail 1/2 of that space by doing a check in kernel
before we do the allocation in c) if the module is already present. For
a) it is harder because userspace just passes a file descriptor. But since we can get the file path without the vmalloc this RFC suggest maybe we can add a new kernel_read*() for module loading where it makes
sense to have only one read happen at a time.
Quoted 4 lines
Quoted 2 lines
I'm wondering how difficult it would be to just try to remove the vmallocs in (a) and (b) and operate on a list of pages.
Yes I think it's worth long term to do that, if possible with seq reads.
OK here's what I suggest we do then:
I'll resubmit the first patch which allows us to prove / disprove if module-autoloading is the culprit. With that in place folks can debug their setup and verify how udev is to blame.
I'll drop the second kernel_read*() patch / effort and punt this as a userspace problem as this is also not extremely pressing.
Long term should evaluate how we can avoid vmalloc for the kread and module decompression.
If this really becomes a pressing issue we can revisit if we want an in kernel solution, but at this point that likely would be systems with over 400-500 CPUs with KASAN enabled. Without KASAN the issue should eventually trigger if you're enablig modules but its hard to say at what point you'd hit this issue.
Luis
Luis Chamberlain
maintainerQuoted 15 lines · Luis Chamberlain
At first I wondered if
we could use file descriptor hints to just exlude users early on boot beforeSYSTEM_RUNNING. I couldn't find much, but if there are some ways to do that -- then the last patch can be simplified to do just that. The second patch proves essentially that we can just send-EBUSYto duplicate requests, at least for duplicate module loads and the world doesn't fall apart. It *would* solve the issue. The patch however borrows tons of the code from the first, and if we're realy going to rely on something like that we may as well share. But I'm hopeful that perhaps there are some jucier file descriptor tricks we can use to just make a file mutually exlusivive and introduce a new kread which letsfinit_module()use that. The saving grace is that at least allfinit_module()calls *wait*, contray torequest_module()calls and so the solution can be much simpler.
Quoted 3 lines
The end result is 0 wasted virtual memory bytes.
Any ideas how not to make patch 2 suck as-is ?
The more I think about it, a file descriptor based approach would have to loop over all tasks as we're not sure if userspace would use the same thread and just fork requests or what. One would then have to loop over all tasks and look for files that match the same name. This approach is domain specific to internal kernel reads and I am thinking now it is more appropriate.
Since this is a bootup issue for races with module loading what we could do is just have say kernel_read_file_from_fd_excl() which would just call a shared __kernel_read_file_from_fd(..., bool excl, ...) and the kernel_read_file_from_fd() could just set that excl to false while the new one sets it to true. Then finit_module() could just use that.
Luis