module: avoid userspace pressure on unwanted allocations
Subsystems (4)
Luis Chamberlain
maintainerv2 cover module: avoid userspace pressure on unwanted allocations
This v2 series follows up on the first iteration of these patches [0]. They have the following changes made:
o Rolled in fix for an kmemleak issue reported by Jim Cromie
o Dropped from this series all the semaphore & and simplifications on kmod.c as that should just be sent as a separate bike-shedding opporunity patch series and it does not in any way address the the unwanted allocations.
o The rest of the feedback was just from Greg KH and I've addressed all his feedback. I decided to do away with the debug.c as a separate file and leave the #ifdef CONFIG_MODULE_DEBUG eyesore at the end of main.c. I guess it's not so bad there.
o *Tons* of fixes and enhancements to my counters, including tons of documentation to help ensure we don't loose track of some of the tribal knowledge and so to help ensure we have references to what our accounting looks like. Those large wasted virtual memory allocations on a simple qemu idle boring boot are simply rediculous, I am quite baffled we had not spotted this before, and so it all reveals we have quite a bit of optimizations left to do to make loading modules an even more smoother experience at bootup.
If you'd like a tree this is on my 20230404-module-alloc-opts branch based on modules-next [1].
[0] https://lkml.kernel.org/r/[email protected]
[1] https://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/linux.git/log/?h=20230404-module-alloc-opts
Luis Chamberlain (6):
module: fix kmemleak annotations for non init ELF sections
module: move finished_loading()
module: extract patient module check into helper
module: avoid allocation if module is already present and ready
debugfs: add debugfs_create_atomic64_t for atomic64_t
module: add debug stats to help identify memory pressure
Documentation/core-api/kernel-api.rst | 22 +- fs/debugfs/file.c | 36 +++ include/linux/debugfs.h | 2 + kernel/module/Kconfig | 37 +++ kernel/module/Makefile | 1 + kernel/module/decompress.c | 4 + kernel/module/internal.h | 74 +++++ kernel/module/main.c | 198 ++++++++---- kernel/module/stats.c | 428 ++++++++++++++++++++++++++ kernel/module/tracking.c | 7 +- 10 files changed, 742 insertions(+), 67 deletions(-) create mode 100644 kernel/module/stats.c
Signature · 1 line
2.39.2
Luis Chamberlain
maintainerPATCH v2 1/6 module: fix kmemleak annotations for non init ELF sections
Commit ac3b43283923 ("module: replace module_layout with module_memory") reworked the way to handle memory allocations to make it clearer. But it lost in translation how we handled kmemleak_ignore() or kmemleak_not_leak() for different ELF sections.
Fix this and clarify the comments a bit more.
--- kernel/module/main.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-)
kernel/module/main.c +16 -4
@@ -2233,11 +2233,23 @@ static int move_module(struct module *mod, struct load_info *info) ptr =module_memory_alloc (mod->mem[type].size, type);/* -* The pointer to this block is stored in the module structure -* which is inside the block. Just mark it as not being a -* leak. +* The pointer to these blocks of memory are stored on the module +* structure and we keep that around so long as the module is +* around. We only free that memory when we unload the module. +* Just mark them as not being a leak then. The .init* ELF +* sections *do* get freed after boot so we treat them slightly +* differently and only grey them out -- they work as typical +* memory allocations which *do* eventually get freed. */ -kmemleak_ignore (ptr); +switch (type) { +case MOD_INIT_TEXT :/* fallthrough */ +case MOD_INIT_DATA :/* fallthrough */ +case MOD_INIT_RODATA :/* fallthrough */ +kmemleak_ignore (ptr); +break ; +default : +kmemleak_not_leak (ptr); + }if (!ptr) { t = type; goto out_enomem;
Signature · 1 line
2.39.2
Luis Chamberlain
maintainerPATCH v2 2/6 module: move finished_loading()
In reply to Luis ChamberlainThis has no functional change, just moves a routine earlier as we'll make use of it next.
--- kernel/module/main.c | 42 +++++++++++++++++++++--------------------- 1 file changed, 21 insertions(+), 21 deletions(-)
kernel/module/main.c +21 -21
@@ -2454,27 +2454,6 @@ static int post_relocation(struct module *mod, const struct load_info *info)return module_finalize (info->hdr, info->sechdrs, mod); } -/* Is this module of this name done loading? No locks held. */ -static bool finished_loading (const char *name) -{ -struct module *mod; -bool ret; - -/* -* The module_mutex should not be a heavily contended lock; -* if we get the occasional sleep here, we'll go an extra iteration -* in the wait_event_interruptible(), which is harmless. -*/ -sched_annotate_sleep (); -mutex_lock (&module_mutex); - mod =find_module_all (name,strlen (name), true); - ret = !mod || mod->state ==MODULE_STATE_LIVE - || mod->state ==MODULE_STATE_GOING ; -mutex_unlock (&module_mutex); - -return ret; -} -/* Call module constructors. */ static void do_mod_ctors (struct module *mod) {
@@ -2638,6 +2617,27 @@ static int may_init_module(void)return 0 ; } +/* Is this module of this name done loading? No locks held. */ +static bool finished_loading (const char *name) +{ +struct module *mod; +bool ret; + +/* +* The module_mutex should not be a heavily contended lock; +* if we get the occasional sleep here, we'll go an extra iteration +* in the wait_event_interruptible(), which is harmless. +*/ +sched_annotate_sleep (); +mutex_lock (&module_mutex); + mod =find_module_all (name,strlen (name), true); + ret = !mod || mod->state ==MODULE_STATE_LIVE + || mod->state ==MODULE_STATE_GOING ; +mutex_unlock (&module_mutex); + +return ret; +} +/* * We try to place it in the list now to make sure it's unique before * we dedicate too many resources. In particular, temporary percpu
Signature · 1 line
2.39.2
Luis Chamberlain
maintainerPATCH v2 3/6 module: extract patient module check into helper
In reply to Luis ChamberlainThe patient module check inside add_unformed_module() is large enough as we need it. It is a bit hard to read too, so just move it to a helper and do the inverse checks first to help shift the code and make it easier to read. The new helper then is module_patient_check_exists().
--- kernel/module/main.c | 71 +++++++++++++++++++++++++------------------- 1 file changed, 40 insertions(+), 31 deletions(-)
kernel/module/main.c +40 -31
@@ -2638,6 +2638,43 @@ static bool finished_loading(const char *name)return ret; } +/* Must be called with module_mutex held */ +static int module_patient_check_exists (const char *name) +{ +struct module *old; +int err =0 ; + + old =find_module_all (name,strlen (name), true); +if (old ==NULL ) +return 0 ; + +if (old->state ==MODULE_STATE_COMING + || old->state ==MODULE_STATE_UNFORMED ) { +/* Wait in case it fails to load. */ +mutex_unlock (&module_mutex); + err =wait_event_interruptible (module_wq, +finished_loading (name)); +if (err) +return err; + +/* The module might have gone in the meantime. */ +mutex_lock (&module_mutex); + old =find_module_all (name,strlen (name), true); + } + +/* +* We are here only when the same module was being loaded. Do +* not try to load it again right now. It prevents long delays +* caused by serialized module load failures. It might happen +* when more devices of the same type trigger load of +* a particular module. +*/ +if (old && old->state ==MODULE_STATE_LIVE ) +return -EEXIST ; +else +return -EBUSY ; +} +/* * We try to place it in the list now to make sure it's unique before * we dedicate too many resources. In particular, temporary percpu
@@ -2646,41 +2683,14 @@ static bool finished_loading(const char *name)static int add_unformed_module (struct module *mod) {int err; -struct module *old; mod->state =MODULE_STATE_UNFORMED ;mutex_lock (&module_mutex); - old =find_module_all (mod->name,strlen (mod->name), true); -if (old !=NULL ) { -if (old->state ==MODULE_STATE_COMING - || old->state ==MODULE_STATE_UNFORMED ) { -/* Wait in case it fails to load. */ -mutex_unlock (&module_mutex); - err =wait_event_interruptible (module_wq, -finished_loading (mod->name)); -if (err) - goto out_unlocked; - -/* The module might have gone in the meantime. */ -mutex_lock (&module_mutex); - old =find_module_all (mod->name,strlen (mod->name), - true); - } - -/* -* We are here only when the same module was being loaded. Do -* not try to load it again right now. It prevents long delays -* caused by serialized module load failures. It might happen -* when more devices of the same type trigger load of -* a particular module. -*/ -if (old && old->state ==MODULE_STATE_LIVE ) - err = -EEXIST ; -else - err = -EBUSY ; + err =module_patient_check_exists (mod->name); +if (err) goto out; - } +mod_update_bounds (mod);list_add_rcu (&mod->list, &modules);mod_tree_insert (mod);
@@ -2688,7 +2698,6 @@ static int add_unformed_module(struct module *mod) out:mutex_unlock (&module_mutex); -out_unlocked:return err; }
Signature · 1 line
2.39.2
Luis Chamberlain
maintainerPATCH v2 4/6 module: avoid allocation if module is already present and ready
In reply to Luis Chamberlainload_module() will allocate a struct module before even checking if the module is already loaded. This can create unecessary memory pressure since we can easily just check if the module is already present early with the copy of the module information from userspace after we've validated it a bit.
This can only be an issue if a system is getting hammered with userspace loading modules. Note that there are two ways to load modules, one is kernel moduile auto-loading (request_module() calls in-kernel) and the other is modprobe calls from userspace. The auto-loading is in-kernel, that pings back to userspace to just call modprobe. We already have a way to restrict the amount of concurrent kernel auto-loads in a given time, however that does not stop a system from issuing tons of system calls to load a module and for the races to exist. Userspace itself *is* supposed to check if a module is present before loading it. But we're observing situations where tons of the same module are in effect being loaded. Although some of these are acknolwedged as in-kernel bugs such as the ACPI frequency modules, issues for which we already have fixes merged or are working towards, but we can also help a bit more in the modules side to avoid those dramatic situations. All that is just memory being allocated to then be thrown away.
To avoid memory pressure for such stupid cases put a stop gap for them. We now check for the module being present *before* allocation, and then right after we are going to add it to the system.
On a 8vcpu 8 GiB RAM system using kdevops and testing against selftests kmod.sh -t 0008 I see a saving in the *highest* side of memory consumption of up to ~ 84 MiB with the Linux kernel selftests kmod test 0008. With the new stress-ng module test I see a 145 MiB difference in max memory consumption with 100 ops. The stress-ng module ops tests can be pretty pathalogical -- it is not realistic, however it was used to finally successfully reproduce issues which are only reported to happen on system with over 400 CPUs [0] by just usign 100 ops on a 8vcpu 8 GiB RAM system.
This can be observed and visualized below. The time it takes to run the test is also not affected.
The kmod tests 0008:
The gnuplot is set to a range from 400000 KiB (390 Mib) - 580000 (566 Mib) given the tests peak around that range.
cat kmod.plot
set term dumb
set output fileout
set yrange [400000:580000]
plot filein with linespoints title "Memory usage (KiB)"
Before:
root@kmod ~ # /data/linux-next/tools/testing/selftests/kmod/kmod.sh -t 0008 root@kmod ~ # free -k -s 1 -c 40 | grep Mem | awk '{print $3}' > log-0008-before.txt ^C root@kmod ~ # sort -n -r log-0008-before.txt | head -1
528732
So ~516.33 MiB
After:
root@kmod ~ # /data/linux-next/tools/testing/selftests/kmod/kmod.sh -t 0008 root@kmod ~ # free -k -s 1 -c 40 | grep Mem | awk '{print $3}' > log-0008-after.txt ^C
root@kmod ~ # sort -n -r log-0008-after.txt | head -1 442516
So ~432.14 MiB
That's about 84 ~MiB in savings in the worst case. The graphs:
root@kmod ~ # gnuplot -e "filein='log-0008-before.txt'; fileout='graph-0008-before.txt'" kmod.plot root@kmod ~ # gnuplot -e "filein='log-0008-after.txt'; fileout='graph-0008-after.txt'" kmod.plot
root@kmod ~ # cat graph-0008-before.txt
580000 +-----------------------------------------------------------------+
| + + + + + + + |
560000 |-+ Memory usage (KiB) ***A***-|
| |
540000 |-+ +-|
| |
| *A *AA*AA*A*AA *A*AA A*A*A *AA*A*AA*A A |
520000 |-+A*A*AA *AA*A *A*AA*A*AA *A*A A *A+-|
|*A |
500000 |-+ +-|
| |
480000 |-+ +-|
| |
460000 |-+ +-|
| |
| |
440000 |-+ +-|
| |
420000 |-+ +-|
| + + + + + + + | 400000 +-----------------------------------------------------------------+
0 5 10 15 20 25 30 35 40
root@kmod ~ # cat graph-0008-after.txt
580000 +-----------------------------------------------------------------+
| + + + + + + + |
560000 |-+ Memory usage (KiB) ***A***-|
| |
540000 |-+ +-|
| |
| |
520000 |-+ +-|
| |
500000 |-+ +-|
| |
480000 |-+ +-|
| |
460000 |-+ +-|
| |
| *A *A*A |
440000 |-+A*A*AA*A A A*A*AA A*A*AA*A*AA*A*AA*A*AA*AA*A*AA*A*AA-|
|*A *A*AA*A |
420000 |-+ +-|
| + + + + + + + | 400000 +-----------------------------------------------------------------+
0 5 10 15 20 25 30 35 40
The stress-ng module tests:
This is used to run the test to try to reproduce the vmap issues reported by David:
echo 0 > /proc/sys/vm/oom_dump_tasks ./stress-ng --module 100 --module-name xfs
Prior to this commit:
root@kmod ~ # free -k -s 1 -c 40 | grep Mem | awk '{print $3}' > baseline-stress-ng.txt root@kmod ~ # sort -n -r baseline-stress-ng.txt | head -1
5046456
After this commit:
root@kmod ~ # free -k -s 1 -c 40 | grep Mem | awk '{print $3}' > after-stress-ng.txt root@kmod ~ # sort -n -r after-stress-ng.txt | head -1
4896972
5046456 - 4896972
149484
149484/1024
145.98046875000000000000
So this commit using stress-ng reveals saving about 145 MiB in memory using 100 ops from stress-ng which reproduced the vmap issue reported.
cat kmod.plot
set term dumb
set output fileout
set yrange [4700000:5070000]
plot filein with linespoints title "Memory usage (KiB)"
root@kmod ~ # gnuplot -e "filein='baseline-stress-ng.txt'; fileout='graph-stress-ng-before.txt'" kmod-simple-stress-ng.plot root@kmod ~ # gnuplot -e "filein='after-stress-ng.txt'; fileout='graph-stress-ng-after.txt'" kmod-simple-stress-ng.plot
root@kmod ~ # cat graph-stress-ng-before.txt
+---------------------------------------------------------------+
5.05e+06 |-+ + A + + + + + + +-|
| * Memory usage (KiB) ***A*** |
| * A |
5e+06 |-+ ** ** +-|
| ** * * A |
4.95e+06 |-+ * * A * A* +-|
| * * A A * * * * A |
| * * * * * * *A * * * A * |
4.9e+06 |-+ * * * A*A * A*AA*A A *A **A **A*A *+-|
| A A*A A * A * * A A * A * ** |
| * ** ** * * * * * * * |
4.85e+06 |-+ A A A ** * * ** *-|
| * * * * ** * |
| * A * * * * |
4.8e+06 |-+ * * * A A-|
| * * * |
4.75e+06 |-+ * * * +-|
| * ** |
| * + + + + + + ** + | 4.7e+06 +---------------------------------------------------------------+
0 5 10 15 20 25 30 35 40
root@kmod ~ # cat graph-stress-ng-after.txt
+---------------------------------------------------------------+
5.05e+06 |-+ + + + + + + + +-|
| Memory usage (KiB) ***A*** |
| |
5e+06 |-+ +-|
| |
4.95e+06 |-+ +-|
| |
| |
4.9e+06 |-+ *AA +-|
| A*AA*A*A A A*AA*AA*A*AA*A A A A*A *AA*A*A A A*AA*AA |
| * * ** * * * ** * *** * |
4.85e+06 |-+* *** * * * * *** A * * +-|
| * A * * ** * * A * * |
| * * * * ** * * |
4.8e+06 |-+* * * A * * * +-|
| * * * A * * |
4.75e+06 |-* * * * * +-|
| * * * * * |
| * + * *+ + + + + * *+ | 4.7e+06 +---------------------------------------------------------------+
0 5 10 15 20 25 30 35 40
[0] https://lkml.kernel.org/r/[email protected]
--- kernel/module/main.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
kernel/module/main.c +5 -1
@@ -2797,7 +2797,11 @@ static int early_mod_check(struct load_info *info, int flags)if (err)return err; -return 0 ; +mutex_lock (&module_mutex); + err =module_patient_check_exists (info->mod->name); +mutex_unlock (&module_mutex); + +return err; }/*
Signature · 1 line
2.39.2
Luis Chamberlain
maintainerPATCH v2 5/6 debugfs: add debugfs_create_atomic64_t for atomic64_t
In reply to Luis ChamberlainSometimes you want to add debugfs entries for atomic counters which can be pretty large using atomic64_t. Add support for these.
--- fs/debugfs/file.c | 36 ++++++++++++++++++++++++++++++++++++ include/linux/debugfs.h | 2 ++ 2 files changed, 38 insertions(+)
fs/debugfs/file.c +36 -0
@@ -780,6 +780,42 @@ void debugfs_create_atomic_t(const char *name, umode_t mode, }EXPORT_SYMBOL_GPL (debugfs_create_atomic_t); +static int debugfs_atomic64_t_set (void *data,u64 val) +{ +atomic64_set ((atomic64_t *)data, val); +return 0 ; +} +static int debugfs_atomic64_t_get (void *data,u64 *val) +{ + *val =atomic64_read ((atomic64_t *)data); +return 0 ; +} +DEFINE_DEBUGFS_ATTRIBUTE_SIGNED (fops_atomic64_t, debugfs_atomic64_t_get, + debugfs_atomic64_t_set,"%lld\n" ); +DEFINE_DEBUGFS_ATTRIBUTE_SIGNED (fops_atomic64_t_ro, debugfs_atomic64_t_get,NULL , +"%lld\n" ); +DEFINE_DEBUGFS_ATTRIBUTE_SIGNED (fops_atomic64_t_wo,NULL , debugfs_atomic64_t_set, +"%lld\n" ); + +/** +* debugfs_create_atomic64_t - create a debugfs file that is used to read and +* write an atomic64_t value +* @name: a pointer to a string containing the name of the file to create. +* @mode: the permission that the file should have +* @parent: a pointer to the parent dentry for this file. This should be a +* directory dentry if set. If this parameter is %NULL, then the +* file will be created in the root of the debugfs filesystem. +* @value: a pointer to the variable that the file should read to and write +* from. +*/ +void debugfs_create_atomic64_t (const char *name,umode_t mode, +struct dentry *parent,atomic64_t *value) +{ +debugfs_create_mode_unsafe (name, mode, parent, value, &fops_atomic64_t, + &fops_atomic64_t_ro, &fops_atomic64_t_wo); +} +EXPORT_SYMBOL_GPL (debugfs_create_atomic64_t); +ssize_t debugfs_read_file_bool (struct file *file,char __user *user_buf,size_t count,loff_t *ppos) {
include/linux/debugfs.h +2 -0
@@ -136,6 +136,8 @@ void debugfs_create_size_t(const char *name, umode_t mode, struct dentry *parent, size_t *value);void debugfs_create_atomic_t (const char *name,umode_t mode,struct dentry *parent,atomic_t *value); +void debugfs_create_atomic64_t (const char *name,umode_t mode, +struct dentry *parent,atomic64_t *value);void debugfs_create_bool (const char *name,umode_t mode,struct dentry *parent,bool *value); void debugfs_create_str(const char *name, umode_t mode,
Signature · 1 line
2.39.2
Luis Chamberlain
maintainerPATCH v2 6/6 module: add debug stats to help identify memory pressure
In reply to Luis ChamberlainLoading modules with finit_module() can end up using vmalloc(), vmap() and vmalloc() again, for a total of up to 3 separate allocations in the worst case for a single module. We always kernel_read*() the module, that's a vmalloc(). Then vmap() is used for the module decompression, and if so the last read buffer is freed as we use the now decompressed module buffer to stuff data into our copy module. The last allocation is specific to each architectures but pretty much that's generally a series of vmalloc() calls or a variation of vmalloc to handle ELF sections with special permissions.
Evaluation with new stress-ng module support [1] with just 100 ops is proving that you can end up using GiBs of data easily even with all care we have in the kernel and userspace today in trying to not load modules which are already loaded. 100 ops seems to resemble the sort of pressure a system with about 400 CPUs can create on module loading. Although issues relating to duplicate module requests due to each CPU inucurring a new module reuest is silly and some of these are being fixed, we currently lack proper tooling to help diagnose easily what happened, when it happened and who likely is to blame -- userspace or kernel module autoloading.
Provide an initial set of stats which use debugfs to let us easily scrape post-boot information about failed loads. This sort of information can be used on production worklaods to try to optimize *avoiding* redundant memory pressure using finit_module().
The following screen shot, of a simple 8vcpu 8 GiB KVM guest shows 226.53 MiB are wasted in virtual memory allocations which due to duplicate module requests during boot. It also shows an average module memory size of 167.10 KiB and an an average module .text + .init.text size of 61.13 KiB. The end shows all modules which were detected as duplicate requests and whether or not they failed early after just the first kernel_read*() call or late after we've already allocated the private space for the module in layout_and_allocate(). A system with module decompression would reveal more wasted virtual memory space.
We should put effort now into identifying the source of these duplicate module requests and trimming these down as much possible. Larger systems will obviously show much more wasted virtual memory allocations.
root@kmod ~ # cat /sys/kernel/debug/modules/stats
Mods ever loaded 67
Mods failed on kread 0
Mods failed on decompress 0
Mods failed on becoming 83
Mods failed on load 16
Total module size 11464704
Total mod text size 4194304
Failed kread bytes 0
Failed decompress bytes 0
Failed becoming bytes 228959096
Failed kmod bytes 8578080
Virtual mem wasted bytes 237537176
Average mod size 171115
Average mod text size 62602
Avg fail becoming bytes 2758544
Average fail load bytes 536130Duplicate failed modules:
module-name How-many-times Reason
kvm_intel 7 Becoming
kvm 7 Becoming
irqbypass 6 Becoming & Load
crct10dif_pclmul 7 Becoming & Load
ghash_clmulni_intel 7 Becoming & Load
sha512_ssse3 6 Becoming & Load
sha512_generic 7 Becoming & Load
aesni_intel 7 Becoming
crypto_simd 7 Becoming & Load
cryptd 3 Becoming & Load
evdev 1 Becoming
serio_raw 1 Becoming
nvme 3 Becoming
nvme_core 3 Becoming
t10_pi 3 Becoming
virtio_pci 3 Becoming
crc32_pclmul 6 Becoming & Load
crc64_rocksoft 3 Becoming
crc32c_intel 3 Becoming
virtio_pci_modern_dev 2 Becoming
virtio_pci_legacy_dev 1 Becoming
crc64 2 Becoming
virtio 2 Becoming
virtio_ring 2 Becoming
[0] https://github.com/ColinIanKing/stress-ng.git
[1] echo 0 > /proc/sys/vm/oom_dump_tasks
./stress-ng --module 100 --module-name xfs
--- Documentation/core-api/kernel-api.rst | 22 +- kernel/module/Kconfig | 37 +++ kernel/module/Makefile | 1 + kernel/module/decompress.c | 4 + kernel/module/internal.h | 74 +++++ kernel/module/main.c | 65 +++- kernel/module/stats.c | 428 ++++++++++++++++++++++++++ kernel/module/tracking.c | 7 +- 8 files changed, 625 insertions(+), 13 deletions(-) create mode 100644 kernel/module/stats.c
Documentation/core-api/kernel-api.rst +20 -2
@@ -220,12 +220,30 @@ relay interface Module Support ============== -Module Loading --------------- +Kernel module auto-loading +-------------------------- .. kernel-doc:: kernel/module/kmod.c :export: +Module debugging +---------------- + +.. kernel-doc:: kernel/module/stats.c + :doc: module debugging statistics overview + +dup_failed_modules - tracks duplicate failed modules +**************************************************** + +.. kernel-doc:: kernel/module/stats.c + :doc: dup_failed_modules - tracks duplicate failed modules + +module statistics debugfs counters +********************************** + +.. kernel-doc:: kernel/module/stats.c + :doc: module statistics debugfs counters + Inter Module support --------------------
kernel/module/Kconfig +37 -0
@@ -22,6 +22,43 @@ menuconfig MODULES if MODULES +config MODULE_DEBUG + bool "Module debugging" + depends on DEBUG_FS + help + Allows you to enable / disable features which can help you debug + modules. You don't need these options in produciton systems. You can + and probably should enable this prior to making your kernel + produciton ready though. + +if MODULE_DEBUG + +config MODULE_STATS + bool "Module statistics" + depends on DEBUG_FS + help + This option allows you to maintain a record of module statistics. + For example each all modules size, average size, text size, and + failed modules and the size for each of those. For failed + modules we keep track of module which failed due to either the + existing module taking too long to load or that module already + was loaded. + + You should enable this if you are debugging production loads + and want to see if userspace or the kernel is doing stupid things + with loading modules when it shouldn't or if you want to help + optimize userspace / kernel space module autoloading schemes. + You might want to do this because failed modules tend to use + use up significan amount of memory, and so you'd be doing everyone + a favor in avoiding these failure proactively. + + This functionality is also useful for those experimenting with + module .text ELF section optimization. + + If unsure, say N. + +endif # MODULE_DEBUG + config MODULE_FORCE_LOAD bool "Forced module loading" default n
kernel/module/Makefile +1 -0
@@ -21,3 +21,4 @@ obj-$(CONFIG_SYSFS) += sysfs.o obj-$(CONFIG_KGDB_KDB) += kdb.o obj-$(CONFIG_MODVERSIONS) += version.o obj-$(CONFIG_MODULE_UNLOAD_TAINT_TRACKING) += tracking.o +obj-$(CONFIG_MODULE_STATS) += stats.o
kernel/module/decompress.c +4 -0
@@ -297,6 +297,10 @@ int module_decompress(struct load_info *info, const void *buf, size_t size)ssize_t data_size;int error; +#if defined(CONFIG_MODULE_STATS ) + info->compressed_len = size; +#endif +/* * Start with number of pages twice as big as needed for * compressed data.
kernel/module/internal.h +74 -0
86 lines not on this page · raw message
kernel/module/main.c +59 -6
171 lines not on this page · raw message
kernel/module/stats.c new file +428 -0
428 lines not on this page · raw message
kernel/module/tracking.c +2 -5
@@ -15,6 +15,7 @@#include "internal.h" static LIST_HEAD (unloaded_tainted_modules); +extern struct dentry *mod_debugfs_root;int try_add_tainted_module (struct module *mod) {
@@ -120,12 +121,8 @@ static const struct file_operations unloaded_tainted_modules_fops = {static int __init unloaded_tainted_modules_init (void ) { -struct dentry *dir; - - dir =debugfs_create_dir ("modules" ,NULL ); -debugfs_create_file ("unloaded_tainted" ,0444 , dir,NULL , +debugfs_create_file ("unloaded_tainted" ,0444 , mod_debugfs_root,NULL , &unloaded_tainted_modules_fops); -return 0 ; }module_init (unloaded_tainted_modules_init);
Signature · 1 line
2.39.2
Re: [PATCH v2 1/6] module: fix kmemleak annotations for non init ELF sections
In reply to Luis ChamberlainQuoted 11 lines · Luis Chamberlain
Commit
ac3b43283923("module: replacemodule_layoutwithmodule_memory") reworked the way to handle memory allocations to make it clearer. But it lost in translation how we handledkmemleak_ignore()orkmemleak_not_leak()for different ELF sections.Fix this and clarify the comments a bit more.
Fixes ac3b43283923 ("module: replace module_layout with module_memory")Reported-by Jim Cromie <[email protected]>Signed-off-by Luis Chamberlain <[email protected]>
Thanks for the fix!
Quoted 39 lines
--- kernel/module/main.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-)kernel/module/main.c +16
-5@@ -2233,11 +2233,23 @@ static int move_module(struct module *mod, struct load_info *info) ptr =module_memory_alloc (mod->mem[type].size, type);/* -* The pointer to this block is stored in the module structure -* which is inside the block. Just mark it as not being a -* leak. +* The pointer to these blocks of memory are stored on the module +* structure and we keep that around so long as the module is +* around. We only free that memory when we unload the module. +* Just mark them as not being a leak then. The .init* ELF +* sections *do* get freed after boot so we treat them slightly +* differently and only grey them out -- they work as typical +* memory allocations which *do* eventually get freed. */ -kmemleak_ignore (ptr); +switch (type) { +case MOD_INIT_TEXT :/* fallthrough */ +case MOD_INIT_DATA :/* fallthrough */ +case MOD_INIT_RODATA :/* fallthrough */ +kmemleak_ignore (ptr); +break ; +default : +kmemleak_not_leak (ptr); + }if (!ptr) { t = type; goto out_enomem; --2.39.2
Re: [PATCH v2 5/6] debugfs: add debugfs_create_atomic64_t for atomic64_t
In reply to Luis ChamberlainQuoted 3 lines · Luis Chamberlain
Sometimes you want to add debugfs entries for atomic counters which can be pretty large using
atomic64_t. Add support for these.
So I realize why you use atomic64, but I really suspect you'd be better off with just the regular "atomic_long".
This is not some debug stat that we care deeply about on 32-bit, and "atomic64" is often really really nasty on 32-bit architectures.
For example, on x86, instead of being a single instruction, it ends up being a cmpxchg loop. In fact, even a single atomic read is a cmpxchg (admittedly without the need for looping).
And yeah, I realize that we don't have a "atomic_long" debugfs interface either. But I think we could just use atomic_long for the module code (avoiding all the horrors of 64-bit atomics on 32-bit architectures), and then using just 'var->counter' for the value. It's not like the debugfs stuff actually does any truly atomic updates.
So something like
debugfs_create_ulong(... &val->counter ..);
instead of
debugfs_create_atomic64(... &val ..);
Hmm?
I dunno. I just think this is not something that may be worth introducing a new thing for, when it is *so* painful on 32-bit, and doesn't seem worth it.
Linus
Luis Chamberlain
maintainerRe: [PATCH v2 5/6] debugfs: add debugfs_create_atomic64_t for atomic64_t
Quoted 2 lines · Linus Torvalds
So I realize why you use atomic64, but I really suspect you'd be better off with just the regular "
atomic_long".
<-- snip -->
Quoted 9 lines
So something like
debugfs_create_ulong(... &val->counter ..);instead of
debugfs_create_atomic64(... &val ..);Hmm?
We already have debugfs_create_ulong(), it just uses unsigned long with no atomic_long. I can just use that then.
Luis
Luis Chamberlain
maintainerRe: [PATCH v2 5/6] debugfs: add debugfs_create_atomic64_t for atomic64_t
Quoted 10 lines · Luis Chamberlain
Quoted 2 lines · Linus Torvalds
So I realize why you use atomic64, but I really suspect you'd be better off with just the regular "
atomic_long".<-- snip -->
Quoted 4 lines
So something like
debugfs_create_ulong(... &val->counter ..);
Quoted 8 lines
Quoted 5 lines
instead of
debugfs_create_atomic64(... &val ..);Hmm?
We already have
debugfs_create_ulong(), it just uses unsigned long with noatomic_long. I can just use that then.
Oh but I don't get the atomic incs, so we'd need debugfs_create_atomic_long_t().
Luis
Re: [PATCH v2 5/6] debugfs: add debugfs_create_atomic64_t for atomic64_t
Quoted 2 lines · Luis Chamberlain
Oh but I don't get the atomic incs, so we'd need
debugfs_create_atomic_long_t().
No, you misunderstand what I meant.
Just use "atomic_long_t" in the module code.
But then the debugfs code should do
debugfs_create_ulong(... &val->counter ..);
to expose said atomic_long values.
No need for new debugfs interfaces.
Because "atomic_long" is just a regular "long" as far as plain read/set operations are concerned - which is all that the debugfs code does anyway.
So I think you can do something like
atomic_long_t total_mod_size;
...
debugfs_create_ulong("total_mod_size",
0400, mod_debugfs_root,
&total_mod_size.counter);but I didn't actually try to compile that kind of version.
(I think "counter" is actually a _signed_ long, so maybe the types don't match).
Linus
Luis Chamberlain
maintainerRe: [PATCH v2 5/6] debugfs: add debugfs_create_atomic64_t for atomic64_t
Quoted 11 lines · Linus Torvalds
Quoted 2 lines · Luis Chamberlain
Oh but I don't get the atomic incs, so we'd need
debugfs_create_atomic_long_t().debugfs_create_ulong("total_mod_size",
0400, mod_debugfs_root, &total_mod_size.counter);but I didn't actually try to compile that kind of version.
(I think "counter" is actually a _signed_ long, so maybe the types don't match).
I see yes, sadly we'd have to cast to (unsigned long *) to make that work as atomic_long counters are long long int:
debugfs_create_ulong("total_mod_size",
0400, mod_debugfs_root,
- &total_mod_size.counter);
+ (unsigned long *)&total_mod_size.counter);
That's:
unsigned long min bits 32
long long min bits 64
But since we'd be doing our accounting in atomic_long and just use debugfs for prints I think that's fine. Just a bit ugly.
Luis
Re: [PATCH v2 2/6] module: move finished_loading()
In reply to Luis ChamberlainQuoted 5 lines · Luis Chamberlain
This has no functional change, just moves a routine earlier as we'll make use of it next.
Signed-off-by Luis Chamberlain <[email protected]>---
I'd simply squash into #3, as that's short enough that the move doesn't add significant noise. Anyhow:
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v2 3/6] module: extract patient module check into helper
In reply to Luis ChamberlainQuoted 21 lines · Luis Chamberlain
The patient module check inside
add_unformed_module()is large enough as we need it. It is a bit hard to read too, so just move it to a helper and do the inverse checks first to help shift the code and make it easier to read. The new helper then ismodule_patient_check_exists().Signed-off-by Luis Chamberlain <[email protected]>--- kernel/module/main.c | 71 +++++++++++++++++++++++++------------------- 1 file changed, 40 insertions(+), 31 deletions(-)kernel/module/main.c +2
-0@@ -2638,6 +2638,43 @@ static bool finished_loading(const char *name)return ret; } +/* Must be called with module_mutex held */ +static int module_patient_check_exists (const char *name)
kernel/module/main.c static bool finished_loading(const char *name) · line 2652
Quoted 10 lines
kernel/module/main.c +10
-0+{ +struct module *old; +int err =0 ; + + old =find_module_all (name,strlen (name), true); +if (old ==NULL ) +return 0 ; + +if (old->state ==MODULE_STATE_COMING + || old->state ==MODULE_STATE_UNFORMED ) {
I never understood why people prefer to prefix the || on a newline. But it seems to be a thing in the module/ world :)
kernel/module/main.c static bool finished_loading(const char *name) · line 2658
Quoted 6 lines
kernel/module/main.c +6
-0+/* Wait in case it fails to load. */ +mutex_unlock (&module_mutex); + err =wait_event_interruptible (module_wq, +finished_loading (name)); +if (err) +return err;
You return with the mutex unlocked. The caller will unlock again ...
Quoted 7 lines
kernel/module/main.c +7
-0+ +/* The module might have gone in the meantime. */ +mutex_lock (&module_mutex); + old =find_module_all (name,strlen (name), true); + } + +/*
kernel/module/main.c static bool finished_loading(const char *name) · line 2675
Quoted 10 lines
kernel/module/main.c +10
-0+* We are here only when the same module was being loaded. Do +* not try to load it again right now. It prevents long delays +* caused by serialized module load failures. It might happen +* when more devices of the same type trigger load of +* a particular module. +*/ +if (old && old->state ==MODULE_STATE_LIVE ) +return -EEXIST ; +else +return -EBUSY ;
You can drop the else and return right away.
kernel/module/main.c static bool finished_loading(const char *name) · line 2679
Quoted 4 lines
kernel/module/main.c +2
-0+} +/* * We try to place it in the list now to make sure it's unique before
Besides, LGTM.
Signature · 3 lines
Thanks, David / dhildenb
Luis Chamberlain
maintainerRe: [PATCH v2 3/6] module: extract patient module check into helper
Quoted 26 lines · David Hildenbrand
Quoted 25 lines · Luis Chamberlain
The patient module check inside
add_unformed_module()is large enough as we need it. It is a bit hard to read too, so just move it to a helper and do the inverse checks first to help shift the code and make it easier to read. The new helper then ismodule_patient_check_exists().Signed-off-by Luis Chamberlain <[email protected]>--- kernel/module/main.c | 71 +++++++++++++++++++++++++------------------- 1 file changed, 40 insertions(+), 31 deletions(-)kernel/module/main.c +7
-0@@ -2638,6 +2638,43 @@ static bool finished_loading(const char *name)return ret; } +/* Must be called with module_mutex held */ +static int module_patient_check_exists (const char *name) +{ +struct module *old; +int err =0 ; + + old =find_module_all (name,strlen (name), true);
Quoted 8 lines
kernel/module/main.c static bool finished_loading(const char *name) · line 2652
Quoted 5 lines
kernel/module/main.c +5
-0+if (old ==NULL ) +return 0 ; + +if (old->state ==MODULE_STATE_COMING + || old->state ==MODULE_STATE_UNFORMED ) {I never understood why people prefer to prefix the || on a newline. But it seems to be a thing in the module/ world :)
Yeah the other way seems better, I'll make it pretty.
Quoted 8 lines
kernel/module/main.c static bool finished_loading(const char *name) · line 2658
Quoted 6 lines
kernel/module/main.c +6
-0+/* Wait in case it fails to load. */ +mutex_unlock (&module_mutex); + err =wait_event_interruptible (module_wq, +finished_loading (name)); +if (err) +return err;You return with the mutex unlocked. The caller will unlock again ...
Fixed now by moving the mutex below up after the wait_event_interruptible(), thanks.
Quoted 11 lines
Quoted 11 lines
kernel/module/main.c +11
-0+ +/* The module might have gone in the meantime. */ +mutex_lock (&module_mutex); + old =find_module_all (name,strlen (name), true); + } + +/* +* We are here only when the same module was being loaded. Do +* not try to load it again right now. It prevents long delays +* caused by serialized module load failures. It might happen +* when more devices of the same type trigger load of
Quoted 8 lines
kernel/module/main.c static bool finished_loading(const char *name) · line 2675
Quoted 6 lines
kernel/module/main.c +6
-0+* a particular module. +*/ +if (old && old->state ==MODULE_STATE_LIVE ) +return -EEXIST ; +else +return -EBUSY ;You can drop the else and return right away.
Will make it pretty like that, thanks.
Luis
Luis Chamberlain
maintainerRe: [PATCH v2 2/6] module: move finished_loading()
In reply to David HildenbrandQuoted 9 lines · David Hildenbrand
Quoted 5 lines · Luis Chamberlain
This has no functional change, just moves a routine earlier as we'll make use of it next.
Signed-off-by Luis Chamberlain <[email protected]>---
I'd simply squash into #3, as that's short enough that the move doesn't add significant noise. Anyhow:
I'll fold that, thanks.
Quoted 1 line
Reviewed-by David Hildenbrand <[email protected]>
What would be *really* nice, if you can, is an output of the new module debugfs stats on your big system. It would be nice to also see the stats if you revert the patch "module: avoid allocation if module is already present and ready".
The delta between those stats should give us a more realistic analysis of probable savings due to that patch on virtual memory on bootup on a large system. In particular the delta between "Virtual mem wasted bytes".
Luis
RE: [PATCH v2 5/6] debugfs: add debugfs_create_atomic64_t for atomic64_t
In reply to Luis ChamberlainFrom: Luis Chamberlain Luis Chamberlain
Quoted 23 lines
Sent: 05 April 2023 17:53
Quoted 11 lines · Linus Torvalds
Quoted 2 lines · Luis Chamberlain
Oh but I don't get the atomic incs, so we'd need
debugfs_create_atomic_long_t().debugfs_create_ulong("total_mod_size",
0400, mod_debugfs_root, &total_mod_size.counter);but I didn't actually try to compile that kind of version.
(I think "counter" is actually a _signed_ long, so maybe the types don't match).
I see yes, sadly we'd have to cast to (unsigned long *) to make that work as
atomic_longcounters are long long int:debugfs_create_ulong("total_mod_size", 0400, mod_debugfs_root,(quoted diff) +1
-1- &total_mod_size.counter); + (unsigned long *)&total_mod_size.counter);
Quoted 7 lines
That's:
unsigned long min bits 32
long long min bits 64But since we'd be doing our accounting in
atomic_longand just use debugfs for prints I think that's fine. Just a bit ugly.
That isn't going to work.
It is pretty much never right to do *(integer_type *)&integer_variable.
But are you really sure 'atomic_long' is 'long long' doesn't sound right at all.
'long long' is 128bit on 64bit and 64bit on 32bit - so is always a double-register access.
This is worse than atomic_u64.
I should probably try to lookup and/or measure the performance of atomic operations (esp. cmpxchg) on x86. Historically they were a separate read and write bus cycles with the 'lock' signal asserted (and still are if they cross cache line boundaries).
But it is possible that at least some of the locked operations (esp. the xchg ones) are implemented within the cache itself so are just a single cpu -> cache operation.
If not it is actually possible that the _local variants that seem to have been added should not use the locked instructions!
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Re: [PATCH v2 5/6] debugfs: add debugfs_create_atomic64_t for atomic64_t
Le 06/04/2023 à 10:15, David Laight a écrit :
Quoted 32 lines
From: Luis Chamberlain Luis Chamberlain
Quoted 23 lines
Sent: 05 April 2023 17:53
Quoted 11 lines · Linus Torvalds
Quoted 2 lines · Luis Chamberlain
Oh but I don't get the atomic incs, so we'd need
debugfs_create_atomic_long_t().debugfs_create_ulong("total_mod_size", 0400, mod_debugfs_root, &total_mod_size.counter);but I didn't actually try to compile that kind of version.
(I think "counter" is actually a _signed_ long, so maybe the types don't match).
I see yes, sadly we'd have to cast to (unsigned long *) to make that work as
atomic_longcounters are long long int:debugfs_create_ulong("total_mod_size", 0400, mod_debugfs_root,(quoted diff) +1
-1- &total_mod_size.counter); + (unsigned long *)&total_mod_size.counter);Quoted 7 lines
That's:
unsigned long min bits 32
long long min bits 64But since we'd be doing our accounting in
atomic_longand just use debugfs for prints I think that's fine. Just a bit ugly.
Quoted 8 lines
That isn't going to work.
It is pretty much never right to do *(integer_type *)&integer_variable.But are you really sure '
atomic_long' is 'long long' doesn't sound right at all.
'long long' is 128bit on 64bit and 64bit on 32bit - so is always a double-register access.
This is worse thanatomic_u64.
On powerpc 'long long' is 64 bits on both PPC32 and PPC64.
Christophe
Quoted 17 lines
I should probably try to lookup and/or measure the performance of atomic operations (esp. cmpxchg) on x86. Historically they were a separate read and write bus cycles with the 'lock' signal asserted (and still are if they cross cache line boundaries).
But it is possible that at least some of the locked operations (esp. the xchg ones) are implemented within the cache itself so are just a single cpu -> cache operation.
If not it is actually possible that the_localvariants that seem to have been added should not use the locked instructions!David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
RE: [PATCH v2 5/6] debugfs: add debugfs_create_atomic64_t for atomic64_t
From: Christophe Leroy
Quoted 1 line
Sent: 06 April 2023 14:38
...
Quoted 1 line
On powerpc 'long long' is 64 bits on both PPC32 and PPC64.
It probably in on x85-64 as well.
By brain is getting frazzled.
On 64bit long long ought to be 128bit.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Re: [PATCH v2 5/6] debugfs: add debugfs_create_atomic64_t for atomic64_t
Quoted 9 lines · David Laight
From: Christophe Leroy
Quoted 1 line
Sent: 06 April 2023 14:38
...
Quoted 1 line
On powerpc 'long long' is 64 bits on both PPC32 and PPC64.
It probably in on x85-64 as well.
By brain is getting frazzled.On 64bit long long ought to be 128bit.
I might have been living under a rock, but the only requirement I am aware of is for long long to be at least 64bit wide.
AFAIK, that is also the case on s390x and aarch64.
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v2 1/6] module: fix kmemleak annotations for non init ELF sections
In reply to Luis ChamberlainQuoted 19 lines · Luis Chamberlain
kernel/module/main.c +7
-3@@ -2233,11 +2233,23 @@ static int move_module(struct module *mod, struct load_info *info) ptr =module_memory_alloc (mod->mem[type].size, type);/* -* The pointer to this block is stored in the module structure -* which is inside the block. Just mark it as not being a -* leak. +* The pointer to these blocks of memory are stored on the module +* structure and we keep that around so long as the module is +* around. We only free that memory when we unload the module. +* Just mark them as not being a leak then. The .init* ELF +* sections *do* get freed after boot so we treat them slightly +* differently and only grey them out -- they work as typical +* memory allocations which *do* eventually get freed. */
kernel/module/main.c static int move_module(struct module *mod, struct load_info *info) · line 2252
Quoted 10 lines
kernel/module/main.c +9
-1-kmemleak_ignore (ptr); +switch (type) { +case MOD_INIT_TEXT :/* fallthrough */ +case MOD_INIT_DATA :/* fallthrough */ +case MOD_INIT_RODATA :/* fallthrough */ +kmemleak_ignore (ptr); +break ; +default : +kmemleak_not_leak (ptr); + }
This works as well but if you want to keep it simple, just call kmemleak_not_leak() in all cases. When freeing the init sections, they would be removed from the kmemleak tracing anyway.
Signature · 1 line
Catalin
Luis Chamberlain
maintainerRe: [PATCH v2 1/6] module: fix kmemleak annotations for non init ELF sections
Quoted 26 lines · Catalin Marinas
Quoted 25 lines · Luis Chamberlain
kernel/module/main.c +12
-4@@ -2233,11 +2233,23 @@ static int move_module(struct module *mod, struct load_info *info) ptr =module_memory_alloc (mod->mem[type].size, type);/* -* The pointer to this block is stored in the module structure -* which is inside the block. Just mark it as not being a -* leak. +* The pointer to these blocks of memory are stored on the module +* structure and we keep that around so long as the module is +* around. We only free that memory when we unload the module. +* Just mark them as not being a leak then. The .init* ELF +* sections *do* get freed after boot so we treat them slightly +* differently and only grey them out -- they work as typical +* memory allocations which *do* eventually get freed. */ -kmemleak_ignore (ptr); +switch (type) { +case MOD_INIT_TEXT :/* fallthrough */ +case MOD_INIT_DATA :/* fallthrough */ +case MOD_INIT_RODATA :/* fallthrough */ +kmemleak_ignore (ptr);
Quoted 8 lines
kernel/module/main.c static int move_module(struct module *mod, struct load_info *info) · line 2252
Quoted 4 lines
kernel/module/main.c +4
-0+break ; +default : +kmemleak_not_leak (ptr); + }This works as well but if you want to keep it simple, just call
kmemleak_not_leak()in all cases. When freeing the init sections, they would be removed from the kmemleak tracing anyway.
It is up to you as you were the one who originally used different calls here, so I didn't want to change the old mechanism. Changing it to use kmemleak_not_leak() would be a functional change, do we loose anything for using kmemleak_not_leak() for all? Ie, why had you used a different set of calls when you first added this depending on the if its init or not? Is the value no longer there?
Luis