static_call() vs __exit fixes
Peter Zijlstra
maintainerPATCH 1/3 static_call: Fix static_call_set_init()
In reply to Peter ZijlstraIt turns out that static_call_set_init() does not preserve the other flags; IOW. it clears TAIL if it was set.
--- kernel/static_call.c | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-)
kernel/static_call.c +10 -7
@@ -35,27 +35,30 @@ static inline void *static_call_addr(strreturn (void *)((long )site->addr + (long )&site->addr); } +static inline unsigned long __static_call_key (const struct static_call_site *site) +{ +return (long )site->key + (long )&site->key; +}static inline struct static_call_key *static_call_key (const struct static_call_site *site) { -return (struct static_call_key *) - (((long )site->key + (long )&site->key) & ~STATIC_CALL_SITE_FLAGS ); +return (void *)(__static_call_key (site) & ~STATIC_CALL_SITE_FLAGS ); }/* These assume the key is word-aligned. */ static inline bool static_call_is_init (struct static_call_site *site) { -return ((long )site->key + (long )&site->key) &STATIC_CALL_SITE_INIT ; +return __static_call_key (site) &STATIC_CALL_SITE_INIT ; }static inline bool static_call_is_tail (struct static_call_site *site) { -return ((long )site->key + (long )&site->key) &STATIC_CALL_SITE_TAIL ; +return __static_call_key (site) &STATIC_CALL_SITE_TAIL ; }static inline void static_call_set_init (struct static_call_site *site) { - site->key = ((long )static_call_key (site) |STATIC_CALL_SITE_INIT ) - + site->key = (__static_call_key (site) |STATIC_CALL_SITE_INIT ) - (long )&site->key; }
@@ -190,7 +193,7 @@ void __static_call_update(struct static_ }arch_static_call_transform (site_addr,NULL , func, -static_call_is_tail (site)); +static_call_is_tail (site)); } }
@@ -349,7 +352,7 @@ static int static_call_add_module(structstruct static_call_site *site;for (site = start; site != stop; site++) { -unsigned long s_key = (long )site->key + (long )&site->key; +unsigned long s_key =__static_call_key (site);unsigned long addr = s_key & ~STATIC_CALL_SITE_FLAGS ;unsigned long key;
Peter Zijlstra
maintainerPATCH 2/3 static_call: Align static_call_is_init() patching condition
In reply to Peter ZijlstraThe intent is to avoid writing init code after init (because the text might have been freed). The code is needlessly different between jump_label and static_call and not obviously correct.
The existing code relies on the fact that the module loader clears the init layout, such that within_module_init() always fails, while jump_label relies on the module state which is more obvious and matches the kernel logic.
--- kernel/static_call.c | 14 ++++---------- 1 file changed, 4 insertions(+), 10 deletions(-)
kernel/static_call.c +4 -10
@@ -149,6 +149,7 @@ void __static_call_update(struct static_ };for (site_mod = &first; site_mod; site_mod = site_mod->next) { +bool init = system_state <SYSTEM_RUNNING ;struct module *mod = site_mod->mod;if (!site_mod->sites) {
@@ -168,6 +169,7 @@ void __static_call_update(struct static_if (mod) { stop = mod->static_call_sites + mod->num_static_call_sites; + init = mod->state ==MODULE_STATE_COMING ; }#endif
@@ -175,16 +177,8 @@ void __static_call_update(struct static_ site < stop && static_call_key(site) == key; site++) {void *site_addr =static_call_addr (site); -if (static_call_is_init (site)) { -/* -* Don't write to call sites which were in -* initmem and have since been freed. -*/ -if (!mod && system_state >=SYSTEM_RUNNING ) -continue ; -if (mod && !within_module_init ((unsigned long )site_addr, mod)) -continue ; - } +if (!init &&static_call_is_init (site)) +continue ;if (!kernel_text_address ((unsigned long )site_addr)) { WARN_ONCE(1, "can't patch static call site at %pS",
Peter Zijlstra
maintainerPATCH 3/3 static_call: Fix static_call_update() sanity check
In reply to Peter ZijlstraSites that match init_section_contains() get marked as INIT. For built-in code init_sections contains both __init and __exit text. OTOH kernel_text_address() only explicitly includes __init text (and there are no __exit text markers).
Match what jump_label already does and ignore the warning for INIT sites. Also see the excellent changelog for commit: 8f35eaa5f2de ("jump_label: Don't warn on __exit jump entries")
--- kernel/jump_label.c | 8 ++++++++ kernel/static_call.c | 11 ++++++++++- 2 files changed, 18 insertions(+), 1 deletion(-)
kernel/jump_label.c +8 -0
@@ -407,6 +407,14 @@ static bool jump_label_can_update(structreturn false;if (!kernel_text_address (jump_entry_code (entry))) { +/* +* This skips patching __exit, which is part of +* init_section_contains() but is not part of +* kernel_text_address(). +* +* Skipping __exit is fine since it will never +* be executed. +*/ WARN_ONCE (!jump_entry_is_init (entry),"can't patch jump_label at %pS" , (void *)jump_entry_code (entry));
kernel/static_call.c +10 -1
@@ -181,7 +181,16 @@ void __static_call_update(struct static_continue ;if (!kernel_text_address ((unsigned long )site_addr)) { -WARN_ONCE (1 ,"can't patch static call site at %pS" , +/* +* This skips patching __exit, which is part of +* init_section_contains() but is not part of +* kernel_text_address(). +* +* Skipping __exit is fine since it will never +* be executed. +*/ +WARN_ONCE (!static_call_is_init (site), +"can't patch static call site at %pS" , site_addr);continue ; }
Peter Zijlstra
maintainercover static_call() vs __exit fixes
Hi,
After more poking a new set of patches to fix static_call() vs __exit functions. These patches replace the patch I posted yesterday:
https://lkml.kernel.org/r/[email protected]
Since I've reproduced the problem locally, and these patches do seem to fully cure things, I'll shortly queue them for tip/locking/urgent.
Peter Zijlstra
maintainerRe: [PATCH 3/3] static_call: Fix static_call_update() sanity check
In reply to Peter ZijlstraQuoted 8 lines · Peter Zijlstra
Sites that match
init_section_contains()get marked as INIT. For built-in codeinit_sectionscontains both__initand__exittext. OTOHkernel_text_address()only explicitly includes__inittext (and there are no__exittext markers).Match what
jump_labelalready does and ignore the warning for INIT sites. Also see the excellent changelog for commit:8f35eaa5f2de("jump_label: Don't warn on__exitjump entries")
Note that I initially had a different fix and thought jump_label was broken for not patching, but then found the above commit.
Quoted 11 lines · Peter Zijlstra
Hi,
After more poking a new set of patches to fix
static_call()vs__exitfunctions. These patches replace the patch I posted yesterday:https://lkml.kernel.org/r/[email protected]
Since I've reproduced the problem locally, and these patches do seem to fully cure things, I'll shortly queue them for tip/locking/urgent.
Thanks Peter for these fixes, works fine for me.
FWIW:
-Sumit
Re: [PATCH 3/3] static_call: Fix static_call_update() sanity check
In reply to Peter ZijlstraQuoted 5 lines · Peter Zijlstra
kernel/static_call.c +3
-1if (!kernel_text_address ((unsigned long )site_addr)) { - WARN_ONCE(1, "can't patch static call site at %pS", +/* +* This skips patching __exit, which is part of +* init_section_contains() but is not part of
kernel/static_call.c void __static_call_update(struct static_ · line 196
Quoted 10 lines
kernel/static_call.c +7
-0+* kernel_text_address(). +* +* Skipping __exit is fine since it will never +* be executed. +*/ +WARN_ONCE (!static_call_is_init (site), +"can't patch static call site at %pS" , site_addr);continue ; }
It might be good to clarify the situation for __exit in modules in the comment and/or changelog, as they both seem to be implicitly talking only about __exit in vmlinux.
For CONFIG_MODULE_UNLOAD, the code ends up in the normal text area, so static_call_is_init() is false and kernel_text_address() is true.
For !CONFIG_MODULE_UNLOAD, the code gets discarded during module load, so static_call_is_init() and kernel_text_address() are both false. I guess that will trigger a warning?
Signature · 1 line
Josh
Peter Zijlstra
maintainerRe: [PATCH 3/3] static_call: Fix static_call_update() sanity check
Quoted 5 lines · Josh Poimboeuf
kernel/static_call.c void __static_call_update(struct static_ · line 185
Quoted 4 lines · Peter Zijlstra
kernel/static_call.c +2
-1if (!kernel_text_address ((unsigned long )site_addr)) { - WARN_ONCE(1, "can't patch static call site at %pS", +/* +* This skips patching __exit, which is part of
This skips patching built-in __exit, ...
?
Quoted 4 lines
kernel/jump_label.c static bool jump_label_can_update(struct · line 415
Quoted 4 lines
kernel/jump_label.c +4
-0+* init_section_contains() but is not part of +* kernel_text_address(). +* +* Skipping __exit is fine since it will never
+ built-in, again
Quoted 11 lines
kernel/static_call.c void __static_call_update(struct static_ · line 196
Quoted 7 lines
kernel/static_call.c +4
-0+* be executed. +*/ +WARN_ONCE (!static_call_is_init (site), +"can't patch static call site at %pS" , site_addr);continue ; }It might be good to clarify the situation for
__exitin modules in the comment and/or changelog, as they both seem to be implicitly talking only about__exitin vmlinux.
Correct.
Quoted 6 lines
For
CONFIG_MODULE_UNLOAD, the code ends up in the normal text area, sostatic_call_is_init()is false andkernel_text_address()is true.For !CONFIG_MODULE_UNLOAD, the code gets discarded during module load, so
static_call_is_init()andkernel_text_address()are both false. I guess that will trigger a warning?
Oh gawd, more variants.
Afaict MODULE_UNLOAD, by virtue of that #ifdef in rewrite_section_headers() won't even load the .exit sections. Afaict that will break: alterative, jump_label and static_call patching all in one go.
Quoted 9 lines · Peter Zijlstra
Hi,
After more poking a new set of patches to fix
static_call()vs__exitfunctions. These patches replace the patch I posted yesterday:https://lkml.kernel.org/r/[email protected]
Since I've reproduced the problem locally, and these patches do seem to fully cure things, I'll shortly queue them for tip/locking/urgent.
For all:
/Jarkkko
[tip: locking/urgent] static_call: Fix static_call_set_init()
In reply to Peter ZijlstraThe following commit has been merged into the locking/urgent branch of tip:
The following commit has been merged into the locking/urgent branch of tip:
Commit-ID: 68b1eddd421d2b16c6655eceb48918a1e896bbbc Gitweb: https://git.kernel.org/tip/68b1eddd421d2b16c6655eceb48918a1e896bbbc Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 18 Mar 2021 11:27:19 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 19 Mar 2021 13:16:44 +01:00
static_call: Fix static_call_set_init()
It turns out that static_call_set_init() does not preserve the other flags; IOW. it clears TAIL if it was set.
--- kernel/static_call.c | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-)
kernel/static_call.c +10 -7
@@ -35,27 +35,30 @@ static inline void *static_call_addr(struct static_call_site *site)return (void *)((long )site->addr + (long )&site->addr); } +static inline unsigned long __static_call_key (const struct static_call_site *site) +{ +return (long )site->key + (long )&site->key; +}static inline struct static_call_key *static_call_key (const struct static_call_site *site) { -return (struct static_call_key *) - (((long )site->key + (long )&site->key) & ~STATIC_CALL_SITE_FLAGS ); +return (void *)(__static_call_key (site) & ~STATIC_CALL_SITE_FLAGS ); }/* These assume the key is word-aligned. */ static inline bool static_call_is_init (struct static_call_site *site) { -return ((long )site->key + (long )&site->key) &STATIC_CALL_SITE_INIT ; +return __static_call_key (site) &STATIC_CALL_SITE_INIT ; }static inline bool static_call_is_tail (struct static_call_site *site) { -return ((long )site->key + (long )&site->key) &STATIC_CALL_SITE_TAIL ; +return __static_call_key (site) &STATIC_CALL_SITE_TAIL ; }static inline void static_call_set_init (struct static_call_site *site) { - site->key = ((long )static_call_key (site) |STATIC_CALL_SITE_INIT ) - + site->key = (__static_call_key (site) |STATIC_CALL_SITE_INIT ) - (long )&site->key; }
@@ -190,7 +193,7 @@ void __static_call_update(struct static_call_key *key, void *tramp, void *func) }arch_static_call_transform (site_addr,NULL , func, -static_call_is_tail (site)); +static_call_is_tail (site)); } }
@@ -349,7 +352,7 @@ static int static_call_add_module(struct module *mod)struct static_call_site *site;for (site = start; site != stop; site++) { -unsigned long s_key = (long )site->key + (long )&site->key; +unsigned long s_key =__static_call_key (site);unsigned long addr = s_key & ~STATIC_CALL_SITE_FLAGS ;unsigned long key;
[tip: locking/urgent] static_call: Align static_call_is_init() patching condition
In reply to Peter ZijlstraThe following commit has been merged into the locking/urgent branch of tip:
The following commit has been merged into the locking/urgent branch of tip:
Commit-ID: 698bacefe993ad2922c9d3b1380591ad489355e9 Gitweb: https://git.kernel.org/tip/698bacefe993ad2922c9d3b1380591ad489355e9 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 18 Mar 2021 11:29:56 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 19 Mar 2021 13:16:44 +01:00
static_call: Align static_call_is_init() patching condition
The intent is to avoid writing init code after init (because the text might have been freed). The code is needlessly different between jump_label and static_call and not obviously correct.
The existing code relies on the fact that the module loader clears the init layout, such that within_module_init() always fails, while jump_label relies on the module state which is more obvious and matches the kernel logic.
--- kernel/static_call.c | 14 ++++---------- 1 file changed, 4 insertions(+), 10 deletions(-)
kernel/static_call.c +4 -10
@@ -149,6 +149,7 @@ void __static_call_update(struct static_call_key *key, void *tramp, void *func) };for (site_mod = &first; site_mod; site_mod = site_mod->next) { +bool init = system_state <SYSTEM_RUNNING ;struct module *mod = site_mod->mod;if (!site_mod->sites) {
@@ -168,6 +169,7 @@ void __static_call_update(struct static_call_key *key, void *tramp, void *func)if (mod) { stop = mod->static_call_sites + mod->num_static_call_sites; + init = mod->state ==MODULE_STATE_COMING ; }#endif
@@ -175,16 +177,8 @@ void __static_call_update(struct static_call_key *key, void *tramp, void *func) site < stop && static_call_key(site) == key; site++) {void *site_addr =static_call_addr (site); -if (static_call_is_init (site)) { -/* -* Don't write to call sites which were in -* initmem and have since been freed. -*/ -if (!mod && system_state >=SYSTEM_RUNNING ) -continue ; -if (mod && !within_module_init ((unsigned long )site_addr, mod)) -continue ; - } +if (!init &&static_call_is_init (site)) +continue ;if (!kernel_text_address ((unsigned long )site_addr)) { WARN_ONCE(1, "can't patch static call site at %pS",
[tip: locking/urgent] static_call: Fix static_call_update() sanity check
In reply to Peter ZijlstraThe following commit has been merged into the locking/urgent branch of tip:
The following commit has been merged into the locking/urgent branch of tip:
Commit-ID: 38c93587375053c5b9ef093f4a5ea754538cba32 Gitweb: https://git.kernel.org/tip/38c93587375053c5b9ef093f4a5ea754538cba32 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 18 Mar 2021 11:31:51 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 19 Mar 2021 13:16:44 +01:00
static_call: Fix static_call_update() sanity check
Sites that match init_section_contains() get marked as INIT. For built-in code init_sections contains both __init and __exit text. OTOH kernel_text_address() only explicitly includes __init text (and there are no __exit text markers).
Match what jump_label already does and ignore the warning for INIT sites. Also see the excellent changelog for commit: 8f35eaa5f2de ("jump_label: Don't warn on __exit jump entries")
--- kernel/jump_label.c | 8 ++++++++ kernel/static_call.c | 11 ++++++++++- 2 files changed, 18 insertions(+), 1 deletion(-)
kernel/jump_label.c +8 -0
@@ -407,6 +407,14 @@ static bool jump_label_can_update(struct jump_entry *entry, bool init)return false;if (!kernel_text_address (jump_entry_code (entry))) { +/* +* This skips patching built-in __exit, which +* is part of init_section_contains() but is +* not part of kernel_text_address(). +* +* Skipping built-in __exit is fine since it +* will never be executed. +*/ WARN_ONCE (!jump_entry_is_init (entry),"can't patch jump_label at %pS" , (void *)jump_entry_code (entry));
kernel/static_call.c +10 -1
@@ -181,7 +181,16 @@ void __static_call_update(struct static_call_key *key, void *tramp, void *func)continue ;if (!kernel_text_address ((unsigned long )site_addr)) { -WARN_ONCE (1 ,"can't patch static call site at %pS" , +/* +* This skips patching built-in __exit, which +* is part of init_section_contains() but is +* not part of kernel_text_address(). +* +* Skipping built-in __exit is fine since it +* will never be executed. +*/ +WARN_ONCE (!static_call_is_init (site), +"can't patch static call site at %pS" , site_addr);continue ; }
Peter Zijlstra
maintainerRe: [PATCH 3/3] static_call: Fix static_call_update() sanity check
In reply to Peter ZijlstraQuoted 10 lines · Peter Zijlstra
Quoted 6 lines
For
CONFIG_MODULE_UNLOAD, the code ends up in the normal text area, sostatic_call_is_init()is false andkernel_text_address()is true.For !CONFIG_MODULE_UNLOAD, the code gets discarded during module load, so
static_call_is_init()andkernel_text_address()are both false. I guess that will trigger a warning?Oh gawd, more variants.
Afaict
MODULE_UNLOAD, by virtue of that #ifdef in
!MODULE_UNLOAD (obv)
Quoted 3 lines
rewrite_section_headers()won't even load the .exit sections. Afaict that will break: alterative,jump_labelandstatic_callpatching all in one go.
Jessica, can you explain how !MODULE_UNLOAD is supposed to work? Alternatives, jump_labels and static_call all can have relocations into __exit code. Not loading it at all would be BAD.
For alternatives all we really need it to discard it after init, for jump_label and static_call we additinoally need to the code to identify as init (ie, within_module_init() must return true for it).
Re: [PATCH 2/3] static_call: Align static_call_is_init() patching condition
In reply to Peter ZijlstraQuoted 17 lines · Peter Zijlstra
The intent is to avoid writing init code after init (because the text might have been freed). The code is needlessly different between
jump_labelandstatic_calland not obviously correct.The existing code relies on the fact that the module loader clears the init layout, such that
within_module_init()always fails, whilejump_labelrelies on the module state which is more obvious and matches the kernel logic.Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/static_call.c | 14 ++++---------- 1 file changed, 4 insertions(+), 10 deletions(-)
kernel/static_call.c void __static_call_update(struct static_ · line 152
Quoted 4 lines
kernel/static_call.c +1
-0@@ -149,6 +149,7 @@ void __static_call_update(struct static_ };for (site_mod = &first; site_mod; site_mod = site_mod->next) { +bool init = system_state <SYSTEM_RUNNING ;
I recently had occasion to look at whether that would be a suitable condition for knowing whether __init stuff was gone, but concluded that it's not. Maybe I'm wrong. init/main.c:
free_initmem();
mark_readonly();
/*
* Kernel mappings are now finalized - update the userspace
page-table
* to finalize PTI.
*/
pti_finalize();
system_state = SYSTEM_RUNNING;So ISTM there's window where system_state < SYSTEM_RUNNING but accessing init stuff is a bad idea. If you're PID1 it's all fine, but I suppose other kernel threads could end up calling static_call_update. And just moving the system_state setting up before the *free_initmem() calls doesn't really solve anything because TOCTOU.
Dunno, probably overkill, but perhaps we could have an atomic_t (or refcount, whatever) init_ref inited to 1, with init_ref_get() doing an inc_unless_zero, and iff you get a ref, you're free to call (/patch) __init functions and access __initdata, but must do init_ref_put(), with PID1 dropping its initial ref and waiting for it to drop to 0 before doing the *free_initmem() calls.
Rasmus
Peter Zijlstra
maintainerRe: [PATCH 2/3] static_call: Align static_call_is_init() patching condition
Quoted 20 lines · Rasmus Villemoes
Quoted 19 lines · Peter Zijlstra
The intent is to avoid writing init code after init (because the text might have been freed). The code is needlessly different between
jump_labelandstatic_calland not obviously correct.The existing code relies on the fact that the module loader clears the init layout, such that
within_module_init()always fails, whilejump_labelrelies on the module state which is more obvious and matches the kernel logic.Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/static_call.c | 14 ++++---------- 1 file changed, 4 insertions(+), 10 deletions(-)kernel/static_call.c +0
-0@@ -149,6 +149,7 @@ void __static_call_update(struct static_ };
Quoted 6 lines
kernel/static_call.c void __static_call_update(struct static_ · line 152
Quoted 2 lines
kernel/static_call.c +1
-0for (site_mod = &first; site_mod; site_mod = site_mod->next) { +bool init = system_state <SYSTEM_RUNNING ;I recently had occasion to look at whether that would be a suitable condition for knowing whether
__initstuff was gone, but concluded that it's not. Maybe I'm wrong.init/main.c:
Ha, me too:
https://lkml.kernel.org/r/YFMToXI/[email protected]
and I share your concern.
Quoted 6 lines
Dunno, probably overkill, but perhaps we could have an
atomic_t(or refcount, whatever)init_refinited to 1, withinit_ref_get()doing aninc_unless_zero, and iff you get a ref, you're free to call (/patch)__initfunctions and access__initdata, but must doinit_ref_put(), with PID1 dropping its initial ref and waiting for it to drop to 0 before doing the*free_initmem()calls.
I'd as soon simply add another SYSTEM state. That way we don't have to worry about who else looks at RUNNING for what etc..
Re: [PATCH 2/3] static_call: Align static_call_is_init() patching condition
Quoted 9 lines · Peter Zijlstra
Quoted 6 lines
Dunno, probably overkill, but perhaps we could have an
atomic_t(or refcount, whatever)init_refinited to 1, withinit_ref_get()doing aninc_unless_zero, and iff you get a ref, you're free to call (/patch)__initfunctions and access__initdata, but must doinit_ref_put(), with PID1 dropping its initial ref and waiting for it to drop to 0 before doing the*free_initmem()calls.I'd as soon simply add another SYSTEM state. That way we don't have to worry about who else looks at RUNNING for what etc..
I don't understand. How would that solve the
PID1 PIDX
ok = system_state < INIT_MEM_GONE;
system_state = INIT_MEM_GONE;free_initmem();system_state = RUNNING;
if (ok)
poke init mem
race? I really don't see any way arbitrary threads can reliably check how far PID1 has progressed at one point in time and use that information (a few lines) later to access init memory without synchronizing with PID1.
AFAICT, having an atomic_t object representing the init memory and a "no, you can't have a new reference" would guarantee correctness of the jump_label/static_call patching: If we get a reference, we do the patching just as for the rest of the kernel .text. If we don't, i.e. observe atomic_load(init_ref)==0, that may not necessarily mean that PID1 has actually discarded the memory yet, but no thread can currently or in the future actually run __init code, so it need not be patched.
Rasmus
Re: [PATCH 2/3] static_call: Align static_call_is_init() patching condition
Quoted 22 lines · Rasmus Villemoes
Quoted 9 lines · Peter Zijlstra
Quoted 6 lines
Dunno, probably overkill, but perhaps we could have an
atomic_t(or refcount, whatever)init_refinited to 1, withinit_ref_get()doing aninc_unless_zero, and iff you get a ref, you're free to call (/patch)__initfunctions and access__initdata, but must doinit_ref_put(), with PID1 dropping its initial ref and waiting for it to drop to 0 before doing the*free_initmem()calls.I'd as soon simply add another SYSTEM state. That way we don't have to worry about who else looks at RUNNING for what etc..
I don't understand. How would that solve the
PID1 PIDX
ok = system_state < INIT_MEM_GONE;
system_state=INIT_MEM_GONE;free_initmem();system_state= RUNNING;
if (ok)
poke init mem
Quoted 6 lines
race? I really don't see any way arbitrary threads can reliably check how far PID1 has progressed at one point in time and use that information (a few lines) later to access init memory without synchronizing with PID1.
AFAICT, having an
atomic_tobject representing the init memory
something like
init/main.c +14 -0
@@ -1417,6 +1417,18 @@ void __weak free_initmem(void)free_initmem_default (POISON_FREE_INITMEM ); } +static atomic_t init_mem_ref =ATOMIC_INIT (1 ); +static DECLARE_COMPLETION (init_mem_may_go); +bool init_mem_get (void ) +{ +return atomic_inc_not_zero (&init_mem_ref); +} +void init_mem_put (void ) +{ +if (atomic_dec_and_test (&init_mem_ref)) +complete (&init_mem_may_go); +} +static int __ref kernel_init (void *unused) {int ret;
@@ -1424,6 +1436,8 @@ static int __ref kernel_init(void *unused)kernel_init_freeable ();/* need to finish all async __init code before freeing the memory */ async_synchronize_full (); +init_mem_put (); +wait_for_completion (&init_mem_may_go);kprobe_free_init_mem ();ftrace_free_init_mem ();kgdb_free_init_mem ();
Steven Rostedt
reviewerRe: [PATCH 3/3] static_call: Fix static_call_update() sanity check
In reply to Peter ZijlstraQuoted 3 lines · On Fri, 19 Mar 2021 13:57:38 +0100 Peter Zijlstra <[email protected]> wrote:
Jessica, can you explain how !MODULE_UNLOAD is supposed to work? Alternatives,
jump_labelsandstatic_callall can have relocations into__exitcode. Not loading it at all would be BAD.
According to the description:
" Without this option you will not be able to unload any
modules (note that some modules may not be unloadable anyway), which makes your kernel smaller, faster and simpler.
If unsure, say Y."
Seems there's no reason to load the "exit" portion, as that's what makes it "smaller".
Would making __exit code the same as init code work? That is, load it just like module init code is loaded, and free it when the init code is freed (hopefully keeping the kernel still "smaller, faster and simpler").
-- Steve
Peter Zijlstra
maintainerRe: [PATCH 3/3] static_call: Fix static_call_update() sanity check
Quoted 2 lines · Steven Rostedt
Would making
__exitcode the same as init code work? That is, load it just like module init code is loaded, and free it when the init code is freed
As stated, yes. But it must then also identify as init through within_module_init().
Steven Rostedt
reviewerRe: [PATCH 3/3] static_call: Fix static_call_update() sanity check
Quoted 6 lines · On Fri, 19 Mar 2021 20:34:24 +0100 Peter Zijlstra <[email protected]> wrote:
Quoted 2 lines · Steven Rostedt
Would making
__exitcode the same as init code work? That is, load it just like module init code is loaded, and free it when the init code is freedAs stated, yes. But it must then also identify as init through
within_module_init().
I think that's doable. Since the usecases for that appear to be mostly about "think code may no longer exist after it is used". Thus, having exit code act just like init code when UNLOAD is not set, appears appropriate.
Jessica, please correct me if I'm wrong.
Thanks,
-- Steve
Re: [PATCH 3/3] static_call: Fix static_call_update() sanity check
In reply to Steven Rostedt+++ Steven Rostedt [19/03/21 14:00 -0400]:
Quoted 9 lines
Quoted 3 lines · On Fri, 19 Mar 2021 13:57:38 +0100 Peter Zijlstra <[email protected]> wrote:
Jessica, can you explain how !MODULE_UNLOAD is supposed to work? Alternatives,
jump_labelsandstatic_callall can have relocations into__exitcode. Not loading it at all would be BAD.According to the description:
Quoted 7 lines
" Without this option you will not be able to unload any
modules (note that some modules may not be unloadable anyway), which makes your kernel smaller, faster and simpler.
If unsure, say Y."Seems there's no reason to load the "exit" portion, as that's what makes it "smaller".
Exactly. If you disable MODULE_UNLOAD, then you don't intend to ever unload any modules, and so you'll never end up calling the module's cleanup/exit function. That code would basically be never used, so that's why it's not loaded in the first place.
Peter Zijlstra
maintainerRe: [PATCH 3/3] static_call: Fix static_call_update() sanity check
Quoted 15 lines · Jessica Yu
+++ Steven Rostedt [19/03/21 14:00 -0400]:
Quoted 14 lines
Quoted 3 lines · On Fri, 19 Mar 2021 13:57:38 +0100 Peter Zijlstra <[email protected]> wrote:
Jessica, can you explain how !MODULE_UNLOAD is supposed to work? Alternatives,
jump_labelsandstatic_callall can have relocations into__exitcode. Not loading it at all would be BAD.According to the description:
" Without this option you will not be able to unload any
modules (note that some modules may not be unloadable anyway), which makes your kernel smaller, faster and simpler.
If unsure, say Y."
Quoted 7 lines
Quoted 2 lines
Seems there's no reason to load the "exit" portion, as that's what makes it "smaller".
Exactly. If you disable
MODULE_UNLOAD, then you don't intend to ever unload any modules, and so you'll never end up calling the module's cleanup/exit function. That code would basically be never used, so that's why it's not loaded in the first place.
As explained, that's broken. Has always been for as long as we've had alternatives.
Re: [PATCH 3/3] static_call: Fix static_call_update() sanity check
In reply to Steven Rostedt+++ Steven Rostedt [19/03/21 16:57 -0400]:
Quoted 7 lines
Quoted 4 lines · On Fri, 19 Mar 2021 20:34:24 +0100 Peter Zijlstra <[email protected]> wrote:
Quoted 2 lines · Steven Rostedt
Would making
__exitcode the same as init code work? That is, load it just like module init code is loaded, and free it when the init code is freed
Quoted 8 lines
Quoted 2 lines
As stated, yes. But it must then also identify as init through
within_module_init().I think that's doable. Since the usecases for that appear to be mostly about "think code may no longer exist after it is used". Thus, having exit code act just like init code when UNLOAD is not set, appears appropriate.
Jessica, please correct me if I'm wrong.
It should be doable. If you want the exit sections to be treated the same as module init, the following patch should stuff any exit sections into the module init "region" (completely untested). Hence it should be freed together with the init sections and it would identify as init through within_module_init(). Let me know if this works for you.
---
kernel/module.c +5 -6
@@ -2802,7 +2802,11 @@ void * __weak module_alloc(unsigned long size)bool __weak module_init_section (const char *name) { -return strstarts (name,".init" ); +#ifndef CONFIG_UNLOAD_MODULE +return strstarts (name,".init" ) ||module_exit_section (name); +#else + return strstarts(name, ".init") +#endif }bool __weak module_exit_section (const char *name)
@@ -3116,11 +3120,6 @@ static int rewrite_section_headers(struct load_info *info, int flags)*/ shdr->sh_addr = (size_t )info->hdr + shdr->sh_offset; -#ifndef CONFIG_MODULE_UNLOAD -/* Don't load .exit sections */ -if (module_exit_section (info->secstrings+shdr->sh_name)) - shdr->sh_flags &= ~(unsigned long )SHF_ALLOC ; -#endif }/* Track but don't keep modinfo and version sections. */
Peter Zijlstra
maintainerRe: [PATCH 3/3] static_call: Fix static_call_update() sanity check
Quoted 5 lines · Jessica Yu
It should be doable. If you want the exit sections to be treated the same as module init, the following patch should stuff any exit sections into the module init "region" (completely untested). Hence it should be freed together with the init sections and it would identify as init through
within_module_init(). Let me know if this works for you.
That does indeed seem to DTRT from a quick scan of module.c. Very nice tidy patch. I was afraid it'd be much worse.
Assuming it actually works; for your Changelog:
"Dynamic code patching (alternatives, jump_label and static_call) can have sites in __exit code, even it __exit is never executed. Therefore __exit must be present at runtime, at least for as long as __init code is.
Additionally, for jump_label and static_call, the __exit sites must also identify as within_module_init(), such that the infrastructure is aware to never touch them after module init -- alternatives are only ran once at init and hence don't have this particular constraint.
By making __exit identify as __init for UNLOAD_MODULE, the above is satisfied."
Thanks!
Quoted 32 lines
---
kernel/module.c +5
-6@@ -2802,7 +2802,11 @@ void * __weak module_alloc(unsigned long size)bool __weak module_init_section (const char *name) { -return strstarts (name,".init" ); +#ifndef CONFIG_UNLOAD_MODULE +return strstarts (name,".init" ) ||module_exit_section (name); +#else + return strstarts(name, ".init") +#endif }bool __weak module_exit_section (const char *name)@@ -3116,11 +3120,6 @@ static int rewrite_section_headers(struct load_info *info, int flags)*/ shdr->sh_addr = (size_t )info->hdr + shdr->sh_offset; -#ifndef CONFIG_MODULE_UNLOAD -/* Don't load .exit sections */ -if (module_exit_section (info->secstrings+shdr->sh_name)) - shdr->sh_flags &= ~(unsigned long )SHF_ALLOC ; -#endif }/* Track but don't keep modinfo and version sections. */
Peter Zijlstra
maintainerRe: [PATCH 2/3] static_call: Align static_call_is_init() patching condition
In reply to Rasmus VillemoesQuoted 15 lines · Rasmus Villemoes
Quoted 9 lines · Peter Zijlstra
Quoted 6 lines
Dunno, probably overkill, but perhaps we could have an
atomic_t(or refcount, whatever)init_refinited to 1, withinit_ref_get()doing aninc_unless_zero, and iff you get a ref, you're free to call (/patch)__initfunctions and access__initdata, but must doinit_ref_put(), with PID1 dropping its initial ref and waiting for it to drop to 0 before doing the*free_initmem()calls.I'd as soon simply add another SYSTEM state. That way we don't have to worry about who else looks at RUNNING for what etc..
I don't understand. How would that solve the
PID1 PIDX
Quoted 8 lines
ok = system_state < INIT_MEM_GONE;
system_state=INIT_MEM_GONE;free_initmem();system_state= RUNNING;
if (ok)
poke init memrace?
Argh, I meant to put it before SMP bringup, but then still have to run the smp_init calls :/
N/m, you're quite right.
Re: [PATCH 3/3] static_call: Fix static_call_update() sanity check
In reply to Peter Zijlstra+++ Peter Zijlstra [22/03/21 17:54 +0100]:
Quoted 18 lines · Jessica Yu
Quoted 6 lines
It should be doable. If you want the exit sections to be treated the same as module init, the following patch should stuff any exit sections into the module init "region" (completely untested). Hence it should be freed together with the init sections and it would identify as init through
within_module_init(). Let me know if this works for you.That does indeed seem to DTRT from a quick scan of
module.c. Very nice tidy patch. I was afraid it'd be much worse.Assuming it actually works; for your Changelog:
"Dynamic code patching (alternatives,
jump_labelandstatic_call) can have sites in__exitcode, even it__exitis never executed. Therefore__exitmust be present at runtime, at least for as long as__initcode is.
Quoted 7 lines
Additionally, for
jump_labelandstatic_call, the__exitsites must also identify aswithin_module_init(), such that the infrastructure is aware to never touch them after module init -- alternatives are only ran once at init and hence don't have this particular constraint.By making
__exitidentify as__initforUNLOAD_MODULE, the above is satisfied."
Thanks a lot for the changelog :-) I'll turn this into a formal patch after some testing tomorrow.
Jessica