Skip to contentSkip to conversation

livepatch/stacktrace: Clean up of reliable stacktrace errors

PATCH Superseded Petr Mladek 5 participants 18 messages Apr 24, 2019 – Apr 29, 2019 Live patching

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Petr Mladek

author maintainer
#0 lore raw

cover livepatch/stacktrace: Clean up of reliable stacktrace errors

I ended with three patches in the end. The first one is the same as before [1]. Then I added two patches for handling of the stacktrace error messages.

[1] https://lkml.kernel.org/r/[email protected]

Petr Mladek (3):
livepatch: Convert error about unsupported reliable stacktrace into a
warning
stacktrace: Remove superfluous WARN_ONCE() from
save_stack_trace_tsk_reliable()
livepatch: Cleanup message handling in klp_try_switch_task()

 kernel/livepatch/core.c       |  5 ++---
 kernel/livepatch/transition.c | 19 ++++++++++++++-----
 kernel/stacktrace.c           |  1 -
 3 files changed, 16 insertions(+), 9 deletions(-)
Signature · 1 line
2.16.4
author maintainer
#1 lore raw

PATCH 1/3 livepatch: Convert error about unsupported reliable stacktrace into a warning

The commit d0807da78e11d46f ("livepatch: Remove immediate feature") caused that any livepatch was refused when reliable stacktraces were not supported on the given architecture.

The limitation is too strong. User space processes are safely migrated even when entering or leaving the kernel. Kthreads transition would need to get forced. But it is safe when:

+ The livepatch does not change the semantic of the code.
+ Callbacks do not depend on a safely finished transition.

---
 kernel/livepatch/core.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
kernel/livepatch/core.c +2 -3
index eb0ee10a1981..14f33ab6c583 100644
@@ -1003,11 +1003,10 @@ int klp_enable_patch(struct klp_patch *patch)
 		return -ENODEV;
 
 	if (!klp_have_reliable_stack()) {
-		pr_err("This architecture doesn't have support for the livepatch consistency model.\n");
-		return -EOPNOTSUPP;
+		pr_warn("This architecture doesn't have support for the livepatch consistency model.\n");
+		pr_warn("The livepatch transition may never complete.\n");
 	}
 
-
 	mutex_lock(&klp_mutex);
 
 	ret = klp_init_patch_early(patch);
Signature · 1 line
2.16.4
Acked-by on v1 1/3
#2 lore raw

Re: [PATCH 1/3] livepatch: Convert error about unsupported reliable stacktrace into a warning

Quoted 38 lines · Petr Mladek

The commit d0807da78e11d46f ("livepatch: Remove immediate feature") caused that any livepatch was refused when reliable stacktraces were not supported on the given architecture.

The limitation is too strong. User space processes are safely migrated even when entering or leaving the kernel. Kthreads transition would need to get forced. But it is safe when:

+ The livepatch does not change the semantic of the code.
+ Callbacks do not depend on a safely finished transition.

---
 kernel/livepatch/core.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
kernel/livepatch/core.c +2 -3
index eb0ee10a1981..14f33ab6c583 100644
@@ -1003,11 +1003,10 @@ int klp_enable_patch(struct klp_patch *patch)
 		return -ENODEV;
 
 	if (!klp_have_reliable_stack()) {
-		pr_err("This architecture doesn't have support for the livepatch consistency model.\n");
-		return -EOPNOTSUPP;
+		pr_warn("This architecture doesn't have support for the livepatch consistency model.\n");
+		pr_warn("The livepatch transition may never complete.\n");
 	}
 
-
 	mutex_lock(&klp_mutex);
 
 	ret = klp_init_patch_early(patch);
Signature · 1 line
2.16.4
Signature · 1 line
Josh
maintainer Acked-by on v1 1/3
#3 lore raw

Re: [PATCH 1/3] livepatch: Convert error about unsupported reliable stacktrace into a warning

In reply to Petr Mladek
Quoted 8 lines · Petr Mladek

The commit d0807da78e11d46f ("livepatch: Remove immediate feature") caused that any livepatch was refused when reliable stacktraces were not supported on the given architecture.

The limitation is too strong. User space processes are safely migrated even when entering or leaving the kernel. Kthreads transition would need to get forced. But it is safe when:

Quoted 5 lines

+ The livepatch does not change the semantic of the code.
+ Callbacks do not depend on a safely finished transition.

Miroslav

Reviewed-by on v1 1/3
#4 lore raw

Re: [PATCH 1/3] livepatch: Convert error about unsupported reliable stacktrace into a warning

In reply to Petr Mladek
Quoted 8 lines · Petr Mladek

The commit d0807da78e11d46f ("livepatch: Remove immediate feature") caused that any livepatch was refused when reliable stacktraces were not supported on the given architecture.

The limitation is too strong. User space processes are safely migrated even when entering or leaving the kernel. Kthreads transition would need to get forced. But it is safe when:

Quoted 5 lines

+ The livepatch does not change the semantic of the code.
+ Callbacks do not depend on a safely finished transition.

Signature · 1 line
Kamalesh
author maintainer
#5 lore raw

Re: [PATCH 1/3] livepatch: Convert error about unsupported reliable stacktrace into a warning

In reply to Petr Mladek
Quoted 37 lines · Petr Mladek

The commit d0807da78e11d46f ("livepatch: Remove immediate feature") caused that any livepatch was refused when reliable stacktraces were not supported on the given architecture.

The limitation is too strong. User space processes are safely migrated even when entering or leaving the kernel. Kthreads transition would need to get forced. But it is safe when:

+ The livepatch does not change the semantic of the code.
+ Callbacks do not depend on a safely finished transition.

---
 kernel/livepatch/core.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
kernel/livepatch/core.c +2 -3
index eb0ee10a1981..14f33ab6c583 100644
@@ -1003,11 +1003,10 @@ int klp_enable_patch(struct klp_patch *patch)
 		return -ENODEV;
 
 	if (!klp_have_reliable_stack()) {
-		pr_err("This architecture doesn't have support for the livepatch consistency model.\n");
-		return -EOPNOTSUPP;
+		pr_warn("This architecture doesn't have support for the livepatch consistency model.\n");
+		pr_warn("The livepatch transition may never complete.\n");
 	}
 
-
 	mutex_lock(&klp_mutex);
 
 	ret = klp_init_patch_early(patch);
Signature · 1 line
2.16.4

I have committed this patch into for-5.2/core branch.

Best Regards,
Petr

PS: I am going to resend 2nd patch separately with more people
interested into kernel/stacktrace.c.

Also I am going to send two separate patches instead of the 3rd one (complete warning removal, static err_buf).

author maintainer
#6 lore raw

PATCH 2/3 stacktrace: Remove superfluous WARN_ONCE() from save_stack_trace_tsk_reliable()

In reply to Petr Mladek

WARN_ONCE() in the generic save_stack_trace_tsk_reliable() is superfluous. The only current user klp_check_stack() writes its own warning when -ENOSYS is returned.

---
 kernel/stacktrace.c | 1 -
 1 file changed, 1 deletion(-)
kernel/stacktrace.c +0 -1
index f8edee9c792d..83ac0ac5ffd9 100644
@@ -74,6 +74,5 @@ __weak int
 save_stack_trace_tsk_reliable(struct task_struct *tsk,
 			      struct stack_trace *trace)
 {
-	WARN_ONCE(1, KERN_INFO "save_stack_tsk_reliable() not implemented yet.\n");
 	return -ENOSYS;
 }
Signature · 1 line
2.16.4
author maintainer
#7 lore raw

Re: [PATCH 2/3] stacktrace: Remove superfluous WARN_ONCE() from save_stack_trace_tsk_reliable()

Quoted 3 lines · Petr Mladek

WARN_ONCE() in the generic save_stack_trace_tsk_reliable() is superfluous. The only current user klp_check_stack() writes its own warning when -ENOSYS is returned.

Quoted 1 line

Best Regards,
Petr

#8 lore raw

Re: [PATCH 2/3] stacktrace: Remove superfluous WARN_ONCE() from save_stack_trace_tsk_reliable()

In reply to Petr Mladek

Adding Thomas because this might slightly conflict with some of the stacktrace.c improvements he's working on.

Quoted 23 lines · Petr Mladek

WARN_ONCE() in the generic save_stack_trace_tsk_reliable() is superfluous. The only current user klp_check_stack() writes its own warning when -ENOSYS is returned.

---
 kernel/stacktrace.c | 1 -
 1 file changed, 1 deletion(-)
kernel/stacktrace.c +0 -1
index f8edee9c792d..83ac0ac5ffd9 100644
@@ -74,6 +74,5 @@ __weak int
 save_stack_trace_tsk_reliable(struct task_struct *tsk,
 			      struct stack_trace *trace)
 {
-	WARN_ONCE(1, KERN_INFO "save_stack_tsk_reliable() not implemented yet.\n");
 	return -ENOSYS;
 }
Signature · 1 line
2.16.4
Signature · 1 line
Josh
Acked-by on v1 2/3
#9 lore raw

Re: [PATCH 2/3] stacktrace: Remove superfluous WARN_ONCE() from save_stack_trace_tsk_reliable()

In reply to Petr Mladek
Quoted 23 lines · Petr Mladek

WARN_ONCE() in the generic save_stack_trace_tsk_reliable() is superfluous. The only current user klp_check_stack() writes its own warning when -ENOSYS is returned.

---
 kernel/stacktrace.c | 1 -
 1 file changed, 1 deletion(-)
kernel/stacktrace.c +0 -1
index f8edee9c792d..83ac0ac5ffd9 100644
@@ -74,6 +74,5 @@ __weak int
 save_stack_trace_tsk_reliable(struct task_struct *tsk,
 			      struct stack_trace *trace)
 {
-	WARN_ONCE(1, KERN_INFO "save_stack_tsk_reliable() not implemented yet.\n");
 	return -ENOSYS;
 }
Signature · 1 line
2.16.4
Signature · 1 line
Josh
Reviewed-by on v1 2/3
#11 lore raw

Re: [PATCH 2/3] stacktrace: Remove superfluous WARN_ONCE() from save_stack_trace_tsk_reliable()

In reply to Petr Mladek
Quoted 5 lines · Petr Mladek

WARN_ONCE() in the generic save_stack_trace_tsk_reliable() is superfluous. The only current user klp_check_stack() writes its own warning when -ENOSYS is returned.

Signature · 1 line
Kamalesh
author maintainer
#12 lore raw

PATCH 3/3 livepatch: Cleanup message handling in klp_try_switch_task()

In reply to Petr Mladek

WARN_ON_ONCE() could not be called safely under rq lock because of console deadlock issues. Fortunately, simple printk_deferred() is enough because the warning is printed from a well defined location and context.

Also klp_try_switch_task() is called under klp_mutex. Therefore, the buffer for debug messages could be static.

---
 kernel/livepatch/transition.c | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)
kernel/livepatch/transition.c +14 -5
index 9c89ae8b337a..e8183d18227f 100644
@@ -254,6 +254,7 @@ static int klp_check_stack_func(struct klp_func *func,
 static int klp_check_stack(struct task_struct *task, char *err_buf)
 {
 	static unsigned long entries[MAX_STACK_ENTRIES];
+	static int enosys_warned;
 	struct stack_trace trace;
 	struct klp_object *obj;
 	struct klp_func *func;
@@ -263,8 +264,16 @@ static int klp_check_stack(struct task_struct *task, char *err_buf)
 	trace.nr_entries = 0;
 	trace.max_entries = MAX_STACK_ENTRIES;
 	trace.entries = entries;
+
 	ret = save_stack_trace_tsk_reliable(task, &trace);
-	WARN_ON_ONCE(ret == -ENOSYS);
+	if (ret == -ENOSYS) {
+		if (!enosys_warned) {
+			printk_deferred(KERN_WARNING "%s: save_stack_trace_tsk_reliable() not supported on this architecture.\n",
+					__func__);
+			enosys_warned = 1;
+		}
+		return ret;
+	}
 	if (ret) {
 		snprintf(err_buf, STACK_ERR_BUF_SIZE,
 			 "%s: %s:%d has an unreliable stack\n",
@@ -297,11 +306,11 @@ static int klp_check_stack(struct task_struct *task, char *err_buf)
  */
 static bool klp_try_switch_task(struct task_struct *task)
 {
+	static char err_buf[STACK_ERR_BUF_SIZE];
 	struct rq *rq;
 	struct rq_flags flags;
 	int ret;
 	bool success = false;
-	char err_buf[STACK_ERR_BUF_SIZE];
 
 	err_buf[0] = '\0';
 
@@ -336,15 +345,15 @@ static bool klp_try_switch_task(struct task_struct *task)
 	task_rq_unlock(rq, task, &flags);
 
 	/*
-	 * Due to console deadlock issues, pr_debug() can't be used while
-	 * holding the task rq lock.  Instead we have to use a temporary buffer
+	 * printk_deferred() need to be used under rq lock to avoid
+	 * console deadlock issues.  But it does not support the dynamic
+	 * debug feature.  Instead we have to use a temporary buffer
 	 * and print the debug message after releasing the lock.
 	 */
 	if (err_buf[0] != '\0')
 		pr_debug("%s", err_buf);
 
 	return success;
-
 }
 
 /*
Signature · 1 line
2.16.4
#13 lore raw

Re: [PATCH 3/3] livepatch: Cleanup message handling in klp_try_switch_task()

Quoted 27 lines · Petr Mladek

WARN_ON_ONCE() could not be called safely under rq lock because of console deadlock issues. Fortunately, simple printk_deferred() is enough because the warning is printed from a well defined location and context.

Also klp_try_switch_task() is called under klp_mutex. Therefore, the buffer for debug messages could be static.

---
 kernel/livepatch/transition.c | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)
kernel/livepatch/transition.c +1 -0
index 9c89ae8b337a..e8183d18227f 100644
@@ -254,6 +254,7 @@ static int klp_check_stack_func(struct klp_func *func,
 static int klp_check_stack(struct task_struct *task, char *err_buf)
 {
 	static unsigned long entries[MAX_STACK_ENTRIES];
+	static int enosys_warned;
 	struct stack_trace trace;
 	struct klp_object *obj;
 	struct klp_func *func;
@@ -263,8 +264,16 @@ static int klp_check_stack(struct task_struct *task, char *err_buf)
 	trace.nr_entries = 0;

kernel/livepatch/transition.c static int klp_check_stack(struct task_struct *task, char *err_buf) · line 273

Quoted 10 lines
kernel/livepatch/transition.c +6 -1
 	trace.max_entries = MAX_STACK_ENTRIES;
 	trace.entries = entries;
+
 	ret = save_stack_trace_tsk_reliable(task, &trace);
-	WARN_ON_ONCE(ret == -ENOSYS);
+	if (ret == -ENOSYS) {
+		if (!enosys_warned) {
+			printk_deferred(KERN_WARNING "%s: save_stack_trace_tsk_reliable() not supported on this architecture.\n",
+					__func__);
+			enosys_warned = 1;

... abusing the fact that you are also printk maintainer :) ... looking at the above, wouldn't it make sense to introduce generic printk_deferred_once() instead?

Signature · 2 lines
Jiri Kosina
SUSE Labs
author maintainer
#14 lore raw

Re: [PATCH 3/3] livepatch: Cleanup message handling in klp_try_switch_task()

Quoted 35 lines · Jiri Kosina
Quoted 33 lines · Petr Mladek

WARN_ON_ONCE() could not be called safely under rq lock because of console deadlock issues. Fortunately, simple printk_deferred() is enough because the warning is printed from a well defined location and context.

Also klp_try_switch_task() is called under klp_mutex. Therefore, the buffer for debug messages could be static.

---
 kernel/livepatch/transition.c | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)
kernel/livepatch/transition.c +3 -1
index 9c89ae8b337a..e8183d18227f 100644
@@ -254,6 +254,7 @@ static int klp_check_stack_func(struct klp_func *func,
 static int klp_check_stack(struct task_struct *task, char *err_buf)
 {
 	static unsigned long entries[MAX_STACK_ENTRIES];
+	static int enosys_warned;
 	struct stack_trace trace;
 	struct klp_object *obj;
 	struct klp_func *func;
@@ -263,8 +264,16 @@ static int klp_check_stack(struct task_struct *task, char *err_buf)
 	trace.nr_entries = 0;
 	trace.max_entries = MAX_STACK_ENTRIES;
 	trace.entries = entries;
+
 	ret = save_stack_trace_tsk_reliable(task, &trace);
-	WARN_ON_ONCE(ret == -ENOSYS);
+	if (ret == -ENOSYS) {
Quoted 8 lines

kernel/livepatch/transition.c static int klp_check_stack(struct task_struct *task, char *err_buf) · line 273

Quoted 4 lines
kernel/livepatch/transition.c +4 -0
+		if (!enosys_warned) {
+			printk_deferred(KERN_WARNING "%s: save_stack_trace_tsk_reliable() not supported on this architecture.\n",
+					__func__);
+			enosys_warned = 1;

... abusing the fact that you are also printk maintainer :) ... looking at the above, wouldn't it make sense to introduce generic printk_deferred_once() instead?

Yeah, I thought about it. Also pr_debug_deferred() would be useful for the other messages passed via err_buf.

Sigh, printk_deferred() is whack-a-mole approach. We use it because we do not have anything better for the deadlocks. I am a bit scared to add more wrappers because it might encourage people to use it more widely, e.g. to avoid softlockups or secure fast paths.

On the other hand, it still looks better to find all occurrences easily instead of hiding them into a custom workarounds.

OK, I am going to prepare new patchset and involve printk people into it.

Best Regards,
Petr

#15 lore raw

Re: [PATCH 3/3] livepatch: Cleanup message handling in klp_try_switch_task()

In reply to Petr Mladek
Quoted 30 lines · Petr Mladek

WARN_ON_ONCE() could not be called safely under rq lock because of console deadlock issues. Fortunately, simple printk_deferred() is enough because the warning is printed from a well defined location and context.

Also klp_try_switch_task() is called under klp_mutex. Therefore, the buffer for debug messages could be static.

---
 kernel/livepatch/transition.c | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)
kernel/livepatch/transition.c +2 -0
index 9c89ae8b337a..e8183d18227f 100644
@@ -254,6 +254,7 @@ static int klp_check_stack_func(struct klp_func *func,
 static int klp_check_stack(struct task_struct *task, char *err_buf)
 {
 	static unsigned long entries[MAX_STACK_ENTRIES];
+	static int enosys_warned;
 	struct stack_trace trace;
 	struct klp_object *obj;
 	struct klp_func *func;
@@ -263,8 +264,16 @@ static int klp_check_stack(struct task_struct *task, char *err_buf)
 	trace.nr_entries = 0;
 	trace.max_entries = MAX_STACK_ENTRIES;
 	trace.entries = entries;
+

kernel/livepatch/transition.c static int klp_check_stack(struct task_struct *task, char *err_buf) · line 276

Quoted 10 lines
kernel/livepatch/transition.c +8 -1
 	ret = save_stack_trace_tsk_reliable(task, &trace);
-	WARN_ON_ONCE(ret == -ENOSYS);
+	if (ret == -ENOSYS) {
+		if (!enosys_warned) {
+			printk_deferred(KERN_WARNING "%s: save_stack_trace_tsk_reliable() not supported on this architecture.\n",
+					__func__);
+			enosys_warned = 1;
+		}
+		return ret;
+	}

We already have a similar printk in patch 1, so is this warning really needed?

Signature · 1 line
Josh
maintainer
#16 lore raw

Re: [PATCH 3/3] livepatch: Cleanup message handling in klp_try_switch_task()

Quoted 36 lines · Josh Poimboeuf
Quoted 35 lines · Petr Mladek

WARN_ON_ONCE() could not be called safely under rq lock because of console deadlock issues. Fortunately, simple printk_deferred() is enough because the warning is printed from a well defined location and context.

Also klp_try_switch_task() is called under klp_mutex. Therefore, the buffer for debug messages could be static.

---
 kernel/livepatch/transition.c | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)
kernel/livepatch/transition.c +5 -1
index 9c89ae8b337a..e8183d18227f 100644
@@ -254,6 +254,7 @@ static int klp_check_stack_func(struct klp_func *func,
 static int klp_check_stack(struct task_struct *task, char *err_buf)
 {
 	static unsigned long entries[MAX_STACK_ENTRIES];
+	static int enosys_warned;
 	struct stack_trace trace;
 	struct klp_object *obj;
 	struct klp_func *func;
@@ -263,8 +264,16 @@ static int klp_check_stack(struct task_struct *task, char *err_buf)
 	trace.nr_entries = 0;
 	trace.max_entries = MAX_STACK_ENTRIES;
 	trace.entries = entries;
+
 	ret = save_stack_trace_tsk_reliable(task, &trace);
-	WARN_ON_ONCE(ret == -ENOSYS);
+	if (ret == -ENOSYS) {
+		if (!enosys_warned) {
+			printk_deferred(KERN_WARNING "%s: save_stack_trace_tsk_reliable() not supported on this architecture.\n",
Quoted 8 lines

kernel/livepatch/transition.c static int klp_check_stack(struct task_struct *task, char *err_buf) · line 276

Quoted 5 lines
kernel/livepatch/transition.c +5 -0
+					__func__);
+			enosys_warned = 1;
+		}
+		return ret;
+	}

We already have a similar printk in patch 1, so is this warning really needed?

I don't think so. pr_warn() in klp_enable_patch() should be enough in my opinion.

However,

if (ret == -ENOSYS)

	return ret;

would be justified, wouldn't it?

Miroslav

#17 lore raw

Re: [PATCH 3/3] livepatch: Cleanup message handling in klp_try_switch_task()

Quoted 2 lines · Miroslav Benes

On Wed, 24 Apr 2019, Josh Poimboeuf wrote:

[...]

Quoted 17 lines
Quoted 5 lines
Quoted 5 lines
kernel/livepatch/transition.c +3 -1
 	ret = save_stack_trace_tsk_reliable(task, &trace);
-	WARN_ON_ONCE(ret == -ENOSYS);
+	if (ret == -ENOSYS) {
+		if (!enosys_warned) {
+			printk_deferred(KERN_WARNING "%s: save_stack_trace_tsk_reliable() not supported on this architecture.\n",
Quoted 8 lines

kernel/livepatch/transition.c static int klp_check_stack(struct task_struct *task, char *err_buf) · line 276

Quoted 5 lines
kernel/livepatch/transition.c +5 -0
+					__func__);
+			enosys_warned = 1;
+		}
+		return ret;
+	}

We already have a similar printk in patch 1, so is this warning really needed?

I don't think so. pr_warn() in klp_enable_patch() should be enough in my opinion.

Quoted 7 lines

However,

if (ret == -ENOSYS)

	return ret;

would be justified, wouldn't it?

Probably an one line comment on why we return, will be helpful.

Signature · 1 line
Kamalesh