Skip to contentSkip to conversation

Deny the patched modules to be removed

PATCH In review Miroslav Benes 3 participants 8 messages Jun 7, 2018 – Jun 11, 2018 Live patching

lore
Share & tools
lore

Miroslav Benes

author maintainer
#0 lore raw

cover Deny the patched modules to be removed

Quoting the second patch's changelog:

Josh reported a bug:

When the object to be patched is a module, and that module is rmmod'ed and reloaded, it fails to load with:

module: x86/modules: Skipping invalid relocation target, existing value is nonzero for type 2, loc 00000000ba0302e9, val ffffffffa03e293c
livepatch: failed to initialize patch 'livepatch_nfsd' for module 'nfsd' (-8)
livepatch: patch 'livepatch_nfsd' failed for module 'nfsd', refusing to load module 'nfsd'

The livepatch module has a relocation which references a symbol in the _previous_ loading of nfsd. When apply_relocate_add() tries to replace the old relocation with a new one, it sees that the previous one is nonzero and it errors out.

On ppc64le, we have a similar issue:

module_64: livepatch_nfsd: Expected nop after call, got e8410018 at e_show+0x60/0x548 [livepatch_nfsd]
livepatch: failed to initialize patch 'livepatch_nfsd' for module 'nfsd' (-8)
livepatch: patch 'livepatch_nfsd' failed for module 'nfsd', refusing to load module 'nfsd'

He also proposed three different solutions. We could remove the error check in apply_relocate_add() introduced by commit eda9cec4c9a1 ("x86/module: Detect and skip invalid relocations"). However the check is useful for detecting corrupted modules.

We could also reverse the relocation patching (clear all relocation targets on x86_64, or return back nops on powerpc) in klp_unpatch_object(). The solution is not universal and is too much arch-specific.

We decided to deny the patched modules to be removed. If it proves to be a major drawback for users, we can still implement a different approach.

There were many subtle details to check, so it may be broken somewhere. Hopefully it is not, but please review carefully.

Miroslav Benes (3):
livepatch: Nullify obj->mod in klp_module_coming()'s error path
livepatch: Deny the patched modules to be removed
module: Remove superfluous call to klp_module_going()

 kernel/livepatch/core.c | 25 +++++++++++++++++++++++--
 kernel/module.c         |  1 -
 2 files changed, 23 insertions(+), 3 deletions(-)
Signature · 1 line
2.17.0
author maintainer
#1 lore raw

PATCH 1/3 livepatch: Nullify obj->mod in klp_module_coming()'s error path

klp_module_coming() is called for every module appearing in the system. It sets obj->mod to a patched module for klp_object obj. Unfortunately it leaves it set even if an error happens later in the function and the patched module is not allowed to be loaded.

klp_is_object_loaded() uses obj->mod variable and could currently give a wrong return value. The bug is probably harmless as of now, but we're gonna rely on klp_is_object_loaded() and correct obj->mod much more and the bug would be visible then.

---
 kernel/livepatch/core.c | 1 +
 1 file changed, 1 insertion(+)
kernel/livepatch/core.c +1 -0
index 3a4656fb7047..36eb5cf38766 100644
@@ -1015,6 +1015,7 @@ int klp_module_coming(struct module *mod)
 	pr_warn("patch '%s' failed for module '%s', refusing to load module '%s'\n",
 		patch->mod->name, obj->mod->name, obj->mod->name);
 	mod->klp_alive = false;
+	obj->mod = NULL;
 	klp_cleanup_module_patches_limited(mod, patch);
 	mutex_unlock(&klp_mutex);
 
Signature · 1 line
2.17.0
maintainer Reviewed-by on v1 1/3
#2 lore raw

Re: [PATCH 1/3] livepatch: Nullify obj->mod in klp_module_coming()'s error path

Quoted 11 lines · Miroslav Benes

klp_module_coming() is called for every module appearing in the system. It sets obj->mod to a patched module for klp_object obj. Unfortunately it leaves it set even if an error happens later in the function and the patched module is not allowed to be loaded.

klp_is_object_loaded() uses obj->mod variable and could currently give a wrong return value. The bug is probably harmless as of now, but we're gonna rely on klp_is_object_loaded() and correct obj->mod much more and the bug would be visible then.

Great catch!

Best Regards,
Petr

author maintainer
#3 lore raw

PATCH 2/3 livepatch: Deny the patched modules to be removed

In reply to Miroslav Benes

Josh reported a bug:

When the object to be patched is a module, and that module is rmmod'ed and reloaded, it fails to load with:

module: x86/modules: Skipping invalid relocation target, existing value is nonzero for type 2, loc 00000000ba0302e9, val ffffffffa03e293c
livepatch: failed to initialize patch 'livepatch_nfsd' for module 'nfsd' (-8)
livepatch: patch 'livepatch_nfsd' failed for module 'nfsd', refusing to load module 'nfsd'

The livepatch module has a relocation which references a symbol in the _previous_ loading of nfsd. When apply_relocate_add() tries to replace the old relocation with a new one, it sees that the previous one is nonzero and it errors out.

On ppc64le, we have a similar issue:

module_64: livepatch_nfsd: Expected nop after call, got e8410018 at e_show+0x60/0x548 [livepatch_nfsd]
livepatch: failed to initialize patch 'livepatch_nfsd' for module 'nfsd' (-8)
livepatch: patch 'livepatch_nfsd' failed for module 'nfsd', refusing to load module 'nfsd'

He also proposed three different solutions. We could remove the error check in apply_relocate_add() introduced by commit eda9cec4c9a1 ("x86/module: Detect and skip invalid relocations"). However the check is useful for detecting corrupted modules.

We could also reverse the relocation patching (clear all relocation targets on x86_64, or return back nops on powerpc) in klp_unpatch_object(). The solution is not universal and is too much arch-specific.

We decided to deny the patched modules to be removed. If it proves to be a major drawback for users, we can still implement a different approach.

The reference of a patched module has to be taken regardless of a patch's state. Thus it is not taken and dropped in enable/disable paths, but in register/unregister paths.

The new behaviour should not collide with mod->klp_alive variable.

---
 kernel/livepatch/core.c | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)
kernel/livepatch/core.c +22 -2
index 36eb5cf38766..2fa9eaee7bb5 100644
@@ -645,6 +645,10 @@ static void klp_free_object_loaded(struct klp_object *obj)
 {
 	struct klp_func *func;
 
+	/*
+	 * Drop the reference from klp_init_object_loaded()
+	 */
+	module_put(obj->mod);
 	obj->mod = NULL;
 
 	klp_for_each_func(obj, func)
@@ -663,6 +667,9 @@ static void klp_free_objects_limited(struct klp_patch *patch,
 	for (obj = patch->objs; obj->funcs && obj != limit; obj++) {
 		klp_free_funcs_limited(obj, NULL);
 		kobject_put(&obj->kobj);
+
+		if (klp_is_module(obj) && klp_is_object_loaded(obj))
+			module_put(obj->mod);
 	}
 }
 
@@ -739,6 +746,14 @@ static int klp_init_object_loaded(struct klp_patch *patch,
 		}
 	}
 
+	/*
+	 * Do not allow patched modules to be removed.
+	 *
+	 * All callers of klp_init_object_loaded() set obj->mod.
+	 */
+	if (klp_is_module(obj) && !try_module_get(obj->mod))
+		return -ENODEV;
+
 	return 0;
 }
 
@@ -984,7 +999,7 @@ int klp_module_coming(struct module *mod)
 			if (ret) {
 				pr_warn("pre-patch callback failed for object '%s'\n",
 					obj->name);
-				goto err;
+				goto err_after_init;
 			}
 
 			ret = klp_patch_object(obj);
@@ -993,7 +1008,7 @@ int klp_module_coming(struct module *mod)
 					patch->mod->name, obj->mod->name, ret);
 
 				klp_post_unpatch_callback(obj);
-				goto err;
+				goto err_after_init;
 			}
 
 			if (patch != klp_transition_patch)
@@ -1007,6 +1022,11 @@ int klp_module_coming(struct module *mod)
 
 	return 0;
 
+err_after_init:
+	/*
+	 * Drop the reference from klp_init_object_loaded().
+	 */
+	module_put(obj->mod);
 err:
 	/*
 	 * If a patch is unsuccessfully applied, return
Signature · 1 line
2.17.0
maintainer
#4 lore raw

Re: [PATCH 2/3] livepatch: Deny the patched modules to be removed

Quoted 13 lines · Miroslav Benes

We decided to deny the patched modules to be removed. If it proves to be a major drawback for users, we can still implement a different approach.

The reference of a patched module has to be taken regardless of a patch's state. Thus it is not taken and dropped in enable/disable paths, but in register/unregister paths.

kernel/livepatch/core.c +0 -0
@@ -739,6 +746,14 @@ static int klp_init_object_loaded(struct klp_patch *patch,
 		}
 	}
 

kernel/livepatch/core.c static int klp_init_object_loaded(struct klp_patch *patch, · line 758

Quoted 10 lines
kernel/livepatch/core.c +8 -0
+	/*
+	 * Do not allow patched modules to be removed.
+	 *
+	 * All callers of klp_init_object_loaded() set obj->mod.
+	 */
+	if (klp_is_module(obj) && !try_module_get(obj->mod))
+		return -ENODEV;
+
 	return 0;
 }

This looks strange. We try to take the reference after we do all actions needed for a loaded module. There is nothing that would prevent obj->mod to get into the going state.

Note that it was safe (except for the relocated symbols) because the module was not able to really go until it called klp_module_going(). But you removed this last break
by the 3rd patch.

I suggest another approach:

I would rename klp_find_object_module() to klp_get_object_module() and get the reference there. Note that it should be fine to call it later in klp_init_object() (before klp_init_object_loaded()). This would solve klp_init_patch().

We will also need to get the reference in klp_module_coming(). It can be called anywhere there. If it fails, we will block loading the module.

Note that the mod->klp_alive logic will still be necessary. It solves various races when both the patch and the patched module are loaded in parallel.

Sigh, this also means that we still could call klp_init_object_loaded() and apply reloacations even when the patched module fails to loaded later from other reasons. This means that this approach does not solve the original problem completely.

I wonder how complicated would be the arch-specific part that would clean up relocations when the module is unloaded.

Best Regards,
Petr

author maintainer
#5 lore raw

Re: [PATCH 2/3] livepatch: Deny the patched modules to be removed

Quoted 14 lines · Petr Mladek
Quoted 13 lines · Miroslav Benes

We decided to deny the patched modules to be removed. If it proves to be a major drawback for users, we can still implement a different approach.

The reference of a patched module has to be taken regardless of a patch's state. Thus it is not taken and dropped in enable/disable paths, but in register/unregister paths.

kernel/livepatch/core.c +0 -0
@@ -739,6 +746,14 @@ static int klp_init_object_loaded(struct klp_patch *patch,
 		}
 	}
 
Quoted 14 lines

kernel/livepatch/core.c static int klp_init_object_loaded(struct klp_patch *patch, · line 758

Quoted 10 lines
kernel/livepatch/core.c +8 -0
+	/*
+	 * Do not allow patched modules to be removed.
+	 *
+	 * All callers of klp_init_object_loaded() set obj->mod.
+	 */
+	if (klp_is_module(obj) && !try_module_get(obj->mod))
+		return -ENODEV;
+
 	return 0;
 }

This looks strange. We try to take the reference after we do all actions needed for a loaded module. There is nothing that would prevent obj->mod to get into the going state.

Yes, there was an intention to keep the number of necessary modifications of the error paths as low as possible.

Quoted 4 lines

Note that it was safe (except for the relocated symbols) because the module was not able to really go until it called klp_module_going(). But you removed this last break
by the 3rd patch.

Yes, try_module_get() would return an error here.

Quoted 10 lines

I suggest another approach:

I would rename klp_find_object_module() to klp_get_object_module() and get the reference there. Note that it should be fine to call it later in klp_init_object() (before klp_init_object_loaded()). This would solve klp_init_patch().

We will also need to get the reference in klp_module_coming(). It can be called anywhere there. If it fails, we will block loading the module.

I can as well move the above hunk up in klp_init_object_loaded(), no? Not that I'm seeing a problem to have it in klp_find_object_module().

Quoted 8 lines

Note that the mod->klp_alive logic will still be necessary. It solves various races when both the patch and the patched module are loaded in parallel.

Sigh, this also means that we still could call klp_init_object_loaded() and apply reloacations even when the patched module fails to loaded later from other reasons. This means that this approach does not solve the original problem completely.

That would be in klp_module_coming(). Hm, you're right.

Quoted 2 lines

I wonder how complicated would be the arch-specific part that would clean up relocations when the module is unloaded.

I don't think it would be complicated. We just want to avoid it, because it is arch-specific. It is more difficult to maintain such code.

Regards,
Miroslav

author maintainer
#6 lore raw

PATCH 3/3 module: Remove superfluous call to klp_module_going()

In reply to Miroslav Benes

Now that patched modules cannot suddenly disappear we could theoretically remove klp_module_going() altogether. Unfortunately we cannot do that in practice. Loading of a patched module may fail and we need to execute right the actions implemented in klp_module_going().

Remove the call from delete_module syscall though, because that one is really superfluous.

---
 kernel/module.c | 1 -
 1 file changed, 1 deletion(-)
kernel/module.c +0 -1
index a6e43a5806a1..af29a0d3708f 100644
@@ -1019,7 +1019,6 @@ SYSCALL_DEFINE2(delete_module, const char __user *, name_user,
 		mod->exit();
 	blocking_notifier_call_chain(&module_notify_list,
 				     MODULE_STATE_GOING, mod);
-	klp_module_going(mod);
 	ftrace_release_mod(mod);
 
 	async_synchronize_full();
Signature · 1 line
2.17.0
Acked-by on v1 3/3
#7 lore raw

Re: [PATCH 3/3] module: Remove superfluous call to klp_module_going()

+++ Miroslav Benes [07/06/18 11:29 +0200]:

Quoted 9 lines

Now that patched modules cannot suddenly disappear we could theoretically remove klp_module_going() altogether. Unfortunately we cannot do that in practice. Loading of a patched module may fail and we need to execute right the actions implemented in klp_module_going().

Remove the call from delete_module syscall though, because that one is really superfluous.

Quoted 19 lines

---
kernel/module.c | 1 -

1 file changed, 1 deletion(-)
kernel/module.c +0 -1
index a6e43a5806a1..af29a0d3708f 100644
@@ -1019,7 +1019,6 @@ SYSCALL_DEFINE2(delete_module, const char __user *, name_user,
 		mod->exit();
 	blocking_notifier_call_chain(&module_notify_list,
 				     MODULE_STATE_GOING, mod);
-	klp_module_going(mod);
 	ftrace_release_mod(mod);
 
 	async_synchronize_full();
Signature · 1 line
2.17.0