Skip to contentSkip to conversation

Add callback-like mechanism before/after ptrace stops

PATCH In review Jan Kiszka 3 participants 15 messages Jul 8, 2021 – Jul 20, 2021

lore
Share & tools
lore

Jan Kiszka

author
#0 lore raw

v2 cover Add callback-like mechanism before/after ptrace stops

Changes in v2:
- use 'bool' in patch 1
- reserve y2038 syscall range before adding new ones in patch 2
- ignore SIG32 (cancellation) in patch 3 to avoid failing tests

This allows debugged real-time processes to execute crucial operations right before and after being stopped by a debugger. Such operations can include pausing physical devices and resuming them in an ordered manner so that no harm is caused to the outer world while debugging takes place and the real-time process can resume normally after the debugging session. It may also involve informing cooperating processes about the ongoing debug session.

The model chosen for this is dedicating a real-time thread to this task. This has the advantage of isolating the thread a bit from the debugged contexts and also avoids having to introduce any kind of asynchronous signaling mechanism. The helper thread will register itself with the core and then run a loop, waiting for the next stop or resume event. At the point the helper receives such an event, all other real-time threads in the process have been stopped or not yet resumed so that the helper cannot interact with them nor should wait for resources they may have locked. The helper thread may also migrate to secondary mode before releasing control, usually only after/before all time-sensitive operations are executed.

These patches have a very long (likely too long) internal history at Siemens, thus have shown to be useful at least for one of our use cases.

Jan

Jan Kiszka (3):
cobalt/kernel: Introduce XNDBGCTRL to block SIGINT/SIGSTOP
cobalt: Add ptrace debugging helper interface testsuite/smokey/gdb: Add test cases for ptrace-based debugging helper

 include/cobalt/Makefile.am          |  1 +
 include/cobalt/ptrace.h             | 37 +++++++++++
 include/cobalt/uapi/Makefile.am     |  1 +
 include/cobalt/uapi/kernel/thread.h |  1 +
 include/cobalt/uapi/ptrace.h        | 24 +++++++
 include/cobalt/uapi/syscall.h       |  3 +
 kernel/cobalt/posix/process.c       | 34 +++++++++-
 kernel/cobalt/posix/process.h       |  5 ++
 kernel/cobalt/posix/syscall.c       | 98 ++++++++++++++++++++++++++++-
 lib/cobalt/Makefile.am              |  1 +
 lib/cobalt/ptrace.c                 | 91 +++++++++++++++++++++++++++
 testsuite/smokey/gdb/gdb.c          | 86 ++++++++++++++++++++++++-
 12 files changed, 377 insertions(+), 5 deletions(-)
 create mode 100644 include/cobalt/ptrace.h
 create mode 100644 include/cobalt/uapi/ptrace.h
 create mode 100644 lib/cobalt/ptrace.c
Signature · 1 line
2.26.2
author
#1 lore raw

PATCH v2 1/3 cobalt/kernel: Introduce XNDBGCTRL to block SIGINT/SIGSTOP

From: Jan Kiszka <[email protected]>

Setting XNDBGCTRL for a thread will prevent its migration to secondary if only the debugging signals SIGINT or SIGSTOP are pending. Such threads will still be able to trigger migration on other signals, such as SIGDEBUG (watchdog), SIGCANCEL (SIGRTMIN) or synchronous SIGTRAP.

This is a building block for the debug helper feature.

---
 include/cobalt/uapi/kernel/thread.h |  1 +
 kernel/cobalt/posix/syscall.c       | 12 +++++++++++-
 2 files changed, 12 insertions(+), 1 deletion(-)
include/cobalt/uapi/kernel/thread.h +1 -0
index 664def08ef..74007ac0d6 100644
@@ -51,6 +51,7 @@
 #define XNTRAPLB  0x00100000 /**< Trap lock break (i.e. may not sleep with sched lock) */
 #define XNDEBUG   0x00200000 /**< User-level debugging enabled */
 #define XNDBGSTOP 0x00400000 /**< Stopped for synchronous debugging */
+#define XNDBGCTRL 0x00800000 /**< Thread is protected against SIGSTOP/SIGINT */
 
 /** @} */
 
kernel/cobalt/posix/syscall.c +11 -1
index cd72b22d14..d46ff103de 100644
@@ -86,6 +86,8 @@ static void prepare_for_signal(struct task_struct *p,
 			       struct pt_regs *regs,
 			       int sysflags)
 {
+	sigset_t pending;
+	bool ignore = false;
 	int notify = 0;
 	spl_t s;
 
@@ -100,13 +102,21 @@ static void prepare_for_signal(struct task_struct *p,
 			xnthread_clear_info(thread, XNBREAK);
 		}
 		xnthread_clear_info(thread, XNKICKED);
+	} else if (xnthread_test_state(thread, XNDBGCTRL)) {
+		/* only ignore thread-specific SIGSTOP/SIGINT */
+		pending = p->pending.signal;
+		sigdelset(&pending, SIGSTOP);
+		sigdelset(&pending, SIGINT);
+		if (sigisemptyset(&pending))
+			ignore = true;
 	}
 
 	xnlock_put_irqrestore(&nklock, s);
 
 	xnthread_test_cancel();
 
-	xnthread_relax(notify, SIGDEBUG_MIGRATE_SIGNAL);
+	if (!ignore)
+		xnthread_relax(notify, SIGDEBUG_MIGRATE_SIGNAL);
 }
 
 static COBALT_SYSCALL(migrate, current, (int domain))
Signature · 1 line
2.26.2
author
#2 lore raw

PATCH v2 2/3 cobalt: Add ptrace debugging helper interface

In reply to Jan Kiszka

From: Jan Kiszka <[email protected]>

This introduces the concept of a debugging helper thread. It can register itself with the cobalt core and then wait for ptrace stop and resumption events. This can be used to bring the connected devices or other parts of the system into a state that can tolerate the potentially long interruption and also synchronize again with it to continue.

On stop events (breakpoints, debugger interceptions), the core will ensure that the helper is run after all primary-mode threads were put on hold and before the debugger will gain control over the whole process. For that purpose, the helper will receive a notification when it was pending on the corresponding event-wait syscall and will release the process into debugging by invoking that syscall to wait for the resumption event.

When the debugger resumes the whole process, the helper is resumed first, right before all threads that will continue in primary mode are released (threads in secondary mode may run earlier but will have to wait when trying to enter primary mode). Again, the helper thread decides when to continue by calling the event-wait system again, in that case in order to wait for the next stop event.

---
 include/cobalt/Makefile.am      |  1 +
 include/cobalt/ptrace.h         | 37 ++++++++++++++
 include/cobalt/uapi/Makefile.am |  1 +
 include/cobalt/uapi/ptrace.h    | 24 +++++++++
 include/cobalt/uapi/syscall.h   |  3 ++
 kernel/cobalt/posix/process.c   | 34 +++++++++++-
 kernel/cobalt/posix/process.h   |  5 ++
 kernel/cobalt/posix/syscall.c   | 86 +++++++++++++++++++++++++++++++
 lib/cobalt/Makefile.am          |  1 +
 lib/cobalt/ptrace.c             | 91 +++++++++++++++++++++++++++++++++
 10 files changed, 281 insertions(+), 2 deletions(-)
 create mode 100644 include/cobalt/ptrace.h
 create mode 100644 include/cobalt/uapi/ptrace.h
 create mode 100644 lib/cobalt/ptrace.c
include/cobalt/Makefile.am +1 -0
index 19e96112e8..e0b203193d 100644
@@ -4,6 +4,7 @@ includesub_HEADERS =	\
 	fcntl.h		\
 	mqueue.h	\
 	pthread.h	\
+	ptrace.h	\
 	sched.h		\
 	semaphore.h	\
 	signal.h	\
include/cobalt/ptrace.h new file +37 -0
new file mode 100644
index 0000000000..f5bec56c9d
@@ -0,0 +1,37 @@
+/*
+ * Copyright (C) Siemens AG, 2015-2021
+ *
+ * Authors:
+ *  Jan Kiszka <[email protected]>
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307 USA.
+ */
+#ifndef _COBALT_PTRACE_H
+#define _COBALT_PTRACE_H
+
+#include <cobalt/uapi/ptrace.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+int cobalt_ptrace_helper_init(void);
+int cobalt_ptrace_event_wait(int event);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* !_COBALT_PTRACE_H */
include/cobalt/uapi/Makefile.am +1 -0
index d887213f8e..41076e23d9 100644
@@ -6,6 +6,7 @@ includesub_HEADERS =	\
 	event.h		\
 	monitor.h	\
 	mutex.h		\
+	ptrace.h	\
 	sched.h		\
 	sem.h		\
 	signal.h	\
include/cobalt/uapi/ptrace.h new file +24 -0
new file mode 100644
index 0000000000..4e61d458c1
@@ -0,0 +1,24 @@
+/*
+ * Copyright (C) Siemens AG, 2015-2021
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA.
+ */
+#ifndef _COBALT_UAPI_PTRACE_H
+#define _COBALT_UAPI_PTRACE_H
+
+#define COBALT_PTRACE_EVENT_STOP	0x1
+#define COBALT_PTRACE_EVENT_RESUME	0x2
+
+#endif /* !_COBALT_UAPI_PTRACE_H */
include/cobalt/uapi/syscall.h +3 -0
index a2795a3644..8d10992284 100644
@@ -128,6 +128,9 @@
 #define sc_cobalt_clock_nanosleep64		105
 #define sc_cobalt_clock_getres64		106
 #define sc_cobalt_clock_adjtime64		107
+/* 108-118 reserved for 64-bit time_t syscalls */
+#define sc_cobalt_ptrace_helper_init		119
+#define sc_cobalt_ptrace_event_wait		120
 
 #define __NR_COBALT_SYSCALLS			128 /* Power of 2 */
 
kernel/cobalt/posix/process.c +32 -2
index 98b7df33fb..ad7def2f27 100644
@@ -45,6 +45,7 @@
 #include <cobalt/kernel/ppd.h>
 #include <cobalt/kernel/vdso.h>
 #include <cobalt/kernel/thread.h>
+#include <cobalt/uapi/ptrace.h>
 #include <cobalt/uapi/signal.h>
 #include <cobalt/uapi/syscall.h>
 #include <pipeline/sched.h>
@@ -683,16 +684,25 @@ void cobalt_stop_debugged_process(struct xnthread *thread)
 	struct cobalt_process *process = process_from_thread(thread);
 	struct cobalt_thread *cth;
 
-	if (process->debugged_threads > 0)
+	if (process->stopped_for_ptrace)
 		return;
 
+	process->stopped_for_ptrace = true;
+
 	list_for_each_entry(cth, &process->thread_list, next) {
-		if (&cth->threadbase == thread)
+		if (&cth->threadbase == thread ||
+		    &cth->threadbase == process->ptrace_helper)
 			continue;
 
 		xnthread_suspend(&cth->threadbase, XNDBGSTOP, XN_INFINITE,
 				 XN_RELATIVE, NULL);
 	}
+
+	if (process->ptrace_helper) {
+		process->pending_ptrace_events |= COBALT_PTRACE_EVENT_STOP;
+		if (xnsynch_wakeup_one_sleeper(&process->ptrace_stop_event))
+			xnsched_run();
+	}
 }
 
 static void cobalt_resume_debugged_process(struct cobalt_process *process)
@@ -705,6 +715,8 @@ static void cobalt_resume_debugged_process(struct cobalt_process *process)
 		if (xnthread_test_state(&cth->threadbase, XNDBGSTOP))
 			xnthread_resume(&cth->threadbase, XNDBGSTOP);
 
+	process->stopped_for_ptrace = false;
+
 	xnsched_unlock();
 }
 
@@ -728,6 +740,17 @@ void cobalt_unregister_debugged_thread(struct xnthread *thread)
 {
 	struct cobalt_process *process = process_from_thread(thread);
 
+	if (process->ptrace_helper && process->debugged_threads == 1 &&
+	    !(process->pending_ptrace_events & COBALT_PTRACE_EVENT_RESUME)) {
+		process->pending_ptrace_events |= COBALT_PTRACE_EVENT_RESUME;
+		if (process->ptrace_helper != thread) {
+			xnthread_clear_state(thread, XNSSTEP);
+			xnthread_set_state(process->ptrace_helper, XNSSTEP);
+		}
+		xnthread_resume(process->ptrace_helper, XNDBGSTOP);
+		return;
+	}
+
 	process->debugged_threads--;
 	xnthread_clear_state(thread, XNSSTEP);
 
@@ -834,6 +857,7 @@ void cobalt_adjust_affinity(struct task_struct *task) /* nklocked, IRQs off */
 
 static void __handle_taskexit_event(struct task_struct *p)
 {
+	struct cobalt_process *process = cobalt_current_process();
 	struct cobalt_ppd *sys_ppd;
 	struct xnthread *thread;
 	spl_t s;
@@ -850,6 +874,11 @@ static void __handle_taskexit_event(struct task_struct *p)
 
 	xnlock_get_irqsave(&nklock, s);
 
+	if (process && process->ptrace_helper == thread) {
+		process->ptrace_helper = NULL;
+		process->pending_ptrace_events = 0;
+	}
+
 	if (xnthread_test_state(thread, XNSSTEP))
 		cobalt_unregister_debugged_thread(thread);
 
@@ -1057,6 +1086,7 @@ static void *cobalt_process_attach(void)
 	INIT_LIST_HEAD(&process->thread_list);
 	xntree_init(&process->usems);
 	bitmap_fill(process->timers_map, CONFIG_XENO_OPT_NRTIMERS);
+	xnsynch_init(&process->ptrace_stop_event, XNSYNCH_FIFO, NULL);
 	cobalt_set_process(process);
 
 	return process;
kernel/cobalt/posix/process.h +5 -0
index 46dff43b7a..c9bccdb361 100644
@@ -22,6 +22,7 @@
 #include <linux/bitmap.h>
 #include <pipeline/thread.h>
 #include <cobalt/kernel/ppd.h>
+#include <cobalt/kernel/synch.h>
 
 #define NR_PERSONALITIES  4
 #if BITS_PER_LONG < NR_PERSONALITIES
@@ -55,6 +56,10 @@ struct cobalt_process {
 	void *priv[NR_PERSONALITIES];
 	int ufeatures;
 	unsigned int debugged_threads;
+	struct xnthread *ptrace_helper;
+	struct xnsynch ptrace_stop_event;
+	unsigned int pending_ptrace_events;
+	bool stopped_for_ptrace;
 };
 
 struct cobalt_resnode {
kernel/cobalt/posix/syscall.c +86 -0

98 lines not on this page · raw message

lib/cobalt/Makefile.am +1 -0
index b3003cd957..3f9136e84c 100644
@@ -24,6 +24,7 @@ libcobalt_la_SOURCES =		\
 	mutex.c			\
 	parse_vdso.c		\
 	printf.c		\
+	ptrace.c		\
 	rtdm.c			\
 	sched.c			\
 	select.c		\
lib/cobalt/ptrace.c new file +91 -0

91 lines not on this page · raw message

Signature · 1 line
2.26.2
author
#3 lore raw

PATCH v2 3/3 testsuite/smokey/gdb: Add test cases for ptrace-based debugging helper

In reply to Jan Kiszka

From: Jan Kiszka <[email protected]>

Extend the gdb test with cases that stress the debugging helper thread. We check if the helper is run on hitting a breakpoint as well as before resuming the process afterwards.

This should be extended in the future by also testing error cases.

---
 testsuite/smokey/gdb/gdb.c | 86 +++++++++++++++++++++++++++++++++++++-
 1 file changed, 84 insertions(+), 2 deletions(-)
testsuite/smokey/gdb/gdb.c +84 -2
index eabf75e6a2..5e7f1c8991 100644
@@ -18,6 +18,7 @@
 #include <semaphore.h>
 #include <sys/wait.h>
 #include <xeno_config.h>
+#include <cobalt/ptrace.h>
 #include <boilerplate/libc.h>
 #include <smokey/smokey.h>
 #include "lib/cobalt/current.h"
@@ -65,10 +66,12 @@ smokey_test_plugin(gdb,
 
 static int pipe_in[2], pipe_out[2];
 static volatile unsigned long long thread_loops, expected_loops;
+static volatile unsigned int stop_events, cont_events;
+static volatile int terminate, ptrace_helper_terminated;
 static volatile int primary_mode;
-static volatile int terminate;
 static int child_terminated;
-static sem_t kickoff_lo;
+static sem_t kickoff_lo, kickoff_helper;
+static pthread_t thread_ptrace_helper;
 
 static void handle_sigchld(int signum)
 {
@@ -184,9 +187,61 @@ static void *thread_hi_func(void *arg)
 	breakpoint_target();
 
 	/* 2nd bp: synchronous continue */
+	breakpoint_target();
+
+	err = sem_post(&kickoff_helper);
+	check_no_error("sem_post", err);
+
+	nanosleep(&ts, NULL);
+
+	/* 3rd bp: synchronous stop with ptrace helper in place */
 	expected_loops = thread_loops;
 	breakpoint_target();
+
+	/* 4th bp: synchronous continue with ptrace helper */
+	breakpoint_target();
+
+	/* 5th bp: terminate ptrace helper (and spinning thread) */
 	terminate = 1;
+	err = pthread_cancel(thread_ptrace_helper);
+	check_no_error("pthread_cancel", err);
+	err = pthread_join(thread_ptrace_helper, NULL);
+	check_no_error("pthread_join", err);
+	ptrace_helper_terminated = 1;
+	breakpoint_target();
+
+	return NULL;
+}
+
+static void *thread_ptrace_helper_func(void *arg)
+{
+	int err, event = COBALT_PTRACE_EVENT_STOP;
+
+	pthread_setname_np(pthread_self(), "pthelper");
+
+	err = sem_wait(&kickoff_helper);
+	check_no_error("sem_wait", err);
+
+	err = pthread_setcanceltype(PTHREAD_CANCEL_ASYNCHRONOUS, NULL);
+	check_no_error("pthread_setcanceltype", err);
+
+	err = cobalt_ptrace_helper_init();
+	check_no_error("cobalt_ptrace_helper_init", err);
+
+	while (1) {
+		err = cobalt_ptrace_event_wait(event);
+		check_no_error("cobalt_ptrace_event_wait", err);
+
+		if (event == COBALT_PTRACE_EVENT_STOP) {
+			stop_events++;
+			event = COBALT_PTRACE_EVENT_RESUME;
+		} else {
+			check("event is COBALT_PTRACE_EVENT_RESUME",
+			      event, COBALT_PTRACE_EVENT_RESUME);
+			cont_events++;
+			event = COBALT_PTRACE_EVENT_STOP;
+		}
+	}
 
 	return NULL;
 }
@@ -223,6 +278,8 @@ static int run_gdb(struct smokey_test *t, int argc, char *const argv[])
 
 		err = sem_init(&kickoff_lo, 0, 0);
 		check_no_error("sem_init", err);
+		err = sem_init(&kickoff_helper, 0, 0);
+		check_no_error("sem_init", err);
 
 		err = pthread_attr_init(&attr);
 		check_no_error("pthread_attr_init", err);
@@ -233,6 +290,13 @@ static int run_gdb(struct smokey_test *t, int argc, char *const argv[])
 		err = pthread_attr_setaffinity_np(&attr, sizeof(cpu_set), &cpu_set);
 		check_no_error("pthread_attr_setaffinity_np", err);
 
+		params.sched_priority = 3;
+		err = pthread_attr_setschedparam(&attr, &params);
+		check_no_error("pthread_attr_setschedparam", err);
+		err = pthread_create(&thread_ptrace_helper, &attr,
+				     thread_ptrace_helper_func, NULL);
+		check_no_error("pthread_create", err);
+
 		params.sched_priority = 2;
 		err = pthread_attr_setschedparam(&attr, &params);
 		check_no_error("pthread_attr_setschedparam", err);
@@ -293,6 +357,24 @@ static int run_gdb(struct smokey_test *t, int argc, char *const argv[])
 
 			smokey_trace("synchronous continue");
 			eval_expression("thread_loops==expected_loops", "1");
+			send_command("c\n", 1);
+
+			smokey_trace("stop with event");
+			eval_expression("thread_loops==expected_loops", "1");
+			eval_expression("stop_events", "1");
+			eval_expression("cont_events", "0");
+			send_command("c\n", 1);
+
+			smokey_trace("continue with event");
+			eval_expression("thread_loops==expected_loops", "1");
+			eval_expression("stop_events", "2");
+			eval_expression("cont_events", "1");
+			/* SIG32 is used by glibc to cancel threads */
+			send_command("handle SIG32 noprint\n", 1);
+			send_command("c\n", 1);
+
+			smokey_trace("terminate ptrace helper");
+			eval_expression("ptrace_helper_terminated", "1");
 
 			send_command("q\n", 0);
 			pause();
Signature · 1 line
2.26.2
author
#4 lore raw
In reply to Jan Kiszka
Quoted 35 lines · Jan Kiszka via Xenomai

Changes in v2:
- use 'bool' in patch 1
- reserve y2038 syscall range before adding new ones in patch 2
- ignore SIG32 (cancellation) in patch 3 to avoid failing tests

This allows debugged real-time processes to execute crucial operations right before and after being stopped by a debugger. Such operations can include pausing physical devices and resuming them in an ordered manner so that no harm is caused to the outer world while debugging takes place and the real-time process can resume normally after the debugging session. It may also involve informing cooperating processes about the ongoing debug session.

The model chosen for this is dedicating a real-time thread to this task. This has the advantage of isolating the thread a bit from the debugged contexts and also avoids having to introduce any kind of asynchronous signaling mechanism. The helper thread will register itself with the core and then run a loop, waiting for the next stop or resume event. At the point the helper receives such an event, all other real-time threads in the process have been stopped or not yet resumed so that the helper cannot interact with them nor should wait for resources they may have locked. The helper thread may also migrate to secondary mode before releasing control, usually only after/before all time-sensitive operations are executed.

These patches have a very long (likely too long) internal history at Siemens, thus have shown to be useful at least for one of our use cases.

Jan

Jan Kiszka (3):
cobalt/kernel: Introduce XNDBGCTRL to block SIGINT/SIGSTOP
cobalt: Add ptrace debugging helper interface testsuite/smokey/gdb: Add test cases for ptrace-based debugging helper

Quoted 17 lines
 include/cobalt/Makefile.am          |  1 +
 include/cobalt/ptrace.h             | 37 +++++++++++
 include/cobalt/uapi/Makefile.am     |  1 +
 include/cobalt/uapi/kernel/thread.h |  1 +
 include/cobalt/uapi/ptrace.h        | 24 +++++++
 include/cobalt/uapi/syscall.h       |  3 +
 kernel/cobalt/posix/process.c       | 34 +++++++++-
 kernel/cobalt/posix/process.h       |  5 ++
 kernel/cobalt/posix/syscall.c       | 98 ++++++++++++++++++++++++++++-
 lib/cobalt/Makefile.am              |  1 +
 lib/cobalt/ptrace.c                 | 91 +++++++++++++++++++++++++++
 testsuite/smokey/gdb/gdb.c          | 86 ++++++++++++++++++++++++-
 12 files changed, 377 insertions(+), 5 deletions(-)
 create mode 100644 include/cobalt/ptrace.h
 create mode 100644 include/cobalt/uapi/ptrace.h
 create mode 100644 lib/cobalt/ptrace.c

32-bit ARM is still showing issues of this series, possibly generic ones. Will continue to debug this today and otherwise pull things out of next again.

Jan

Signature · 2 lines
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux
#5 lore raw
In reply to Jan Kiszka
Quoted 12 lines · Jan Kiszka via Xenomai

The model chosen for this is dedicating a real-time thread to this task. This has the advantage of isolating the thread a bit from the debugged contexts and also avoids having to introduce any kind of asynchronous signaling mechanism. The helper thread will register itself with the core and then run a loop, waiting for the next stop or resume event. At the point the helper receives such an event, all other real-time threads in the process have been stopped or not yet resumed so that the helper cannot interact with them nor should wait for resources they may have locked. The helper thread may also migrate to secondary mode before releasing control, usually only after/before all time-sensitive operations are executed.

Quoted 2 lines

These patches have a very long (likely too long) internal history at Siemens, thus have shown to be useful at least for one of our use cases.

We have a similar but still rather different set of use cases.
A Xenomai 2 based system David and I working on carries a in house patch to deliver
exceptions to userspace in the primary domain. The current patch implements a signal alike interface. So userspace can register a hard real-time handler for e.g. SIGFPE or SIGTRAP.
If a real-time task triggers an exception (e.g. division by zero, hitting a break-point exception) this
exception is translated to a signal and delivered to userspace without switching to the secondary domain.

The monitor thread approach should work too, I guess.
But the main difference is that the application I have in mind needs a notification for exceptions,
plus machine state. This is why the signal model was chosen. Just like on the Linux side
the real-time thread has access to mcontext to figure why the exception happened, etc..

Let me outline two use cases.
1. FPU exception handling:
The application we work on is more a runtime than an application, it runs programs that assume
specific behavior regarding integer and floating point operations. These program have been developed for other operating systems and must not break on
Linux/Xenomai.
For example they assume that a division by zero does not trap but results in 0. So the runtime handles floating point exceptions, looks at the machine state, decodes the current
operation and fixes the result.
Like it or not, we have to do this to keep binary compatibly. Rebuilding these programs
is not possible.

2. Break point handling:
The runtime allows instrumenting of programs it executes. A system operator can mark code paths
to see how often they are executed. So the runtime installs a break point instruction at the desired
place and handles the breakpoint exception in the primary domain where just a counter
is incremented.
Although this slows down execution and hurts the system response time it is still fast enough
to satisfy our real-time requirements.
On the other hand switching to the secondary domain is far too expensive.

Maybe we can combine both approaches? We'll happily port the current patch to Xenomai 3.1
and give the monitor thread approach a try.

author
#6 lore raw
Quoted 36 lines · Richard Weinberger
Quoted 12 lines · Jan Kiszka via Xenomai

The model chosen for this is dedicating a real-time thread to this task. This has the advantage of isolating the thread a bit from the debugged contexts and also avoids having to introduce any kind of asynchronous signaling mechanism. The helper thread will register itself with the core and then run a loop, waiting for the next stop or resume event. At the point the helper receives such an event, all other real-time threads in the process have been stopped or not yet resumed so that the helper cannot interact with them nor should wait for resources they may have locked. The helper thread may also migrate to secondary mode before releasing control, usually only after/before all time-sensitive operations are executed.

Quoted 2 lines

These patches have a very long (likely too long) internal history at Siemens, thus have shown to be useful at least for one of our use cases.

We have a similar but still rather different set of use cases.
A Xenomai 2 based system David and I working on carries a in house patch to deliver
exceptions to userspace in the primary domain. The current patch implements a signal alike interface. So userspace can register a hard real-time handler for e.g. SIGFPE or SIGTRAP.
If a real-time task triggers an exception (e.g. division by zero, hitting a break-point exception) this
exception is translated to a signal and delivered to userspace without switching to the secondary domain.

The monitor thread approach should work too, I guess.
But the main difference is that the application I have in mind needs a notification for exceptions,
plus machine state. This is why the signal model was chosen. Just like on the Linux side
the real-time thread has access to mcontext to figure why the exception happened, etc..

Quoted 15 lines

Let me outline two use cases.
1. FPU exception handling:
The application we work on is more a runtime than an application, it runs programs that assume
specific behavior regarding integer and floating point operations. These program have been developed for other operating systems and must not break on
Linux/Xenomai.
For example they assume that a division by zero does not trap but results in 0. So the runtime handles floating point exceptions, looks at the machine state, decodes the current
operation and fixes the result.
Like it or not, we have to do this to keep binary compatibly. Rebuilding these programs
is not possible.

We have kind of a similar case in our backlog. The application expects to handle certain FPU-related faults gracefully, i.e. without falling out of RT with the causing thread, using a context update to bring it on the road again.

We modeled that case on top of the debug monitor thread, waking it up also on such faults and giving it a chance to remotely fix up the triggering thread.

Quoted 13 lines

2. Break point handling:
The runtime allows instrumenting of programs it executes. A system operator can mark code paths
to see how often they are executed. So the runtime installs a break point instruction at the desired
place and handles the breakpoint exception in the primary domain where just a counter
is incremented.
Although this slows down execution and hurts the system response time it is still fast enough
to satisfy our real-time requirements.
On the other hand switching to the secondary domain is far too expensive.

But this is not debugging, this is UPROBES / UPROBE_EVENTS, no? Maybe better make that work with oob if it does not yet.

Quoted 5 lines

Maybe we can combine both approaches? We'll happily port the current patch to Xenomai 3.1
and give the monitor thread approach a try.

I'd like to see how you implemented that signal-like context switching in your approach. That is where I shied away from, rather using a thread switch plus a (not too nice, granted) mcontext get/set for other threads.

My current understanding is that a real-time domain signal mechanism will require changes in the pipeline patch underneath, thus first of all needs to be sold to the dovetail infrastructure (and then we can discuss if we backport that to ipipe anymore).

That said, I currently have a problem without remote fix-up approach as
well: I-pipe allowed Xenomai to declare a fault resolved, dovetail so far always migrates the causing thread. If we would go that way, we would also need a bit of dovetail changes. For that reason, I started to question the remaining use cases for these self fix-ups internally again, but I cannot tell yet how successful that will be.

Jan

Signature · 2 lines
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux
#7 lore raw
Quoted 8 lines · Jan Kiszka

We have kind of a similar case in our backlog. The application expects to handle certain FPU-related faults gracefully, i.e. without falling out of RT with the causing thread, using a context update to bring it on the road again.

We modeled that case on top of the debug monitor thread, waking it up also on such faults and giving it a chance to remotely fix up the triggering thread.

Yeah, sounds related and I think useful for our case too.

Quoted 14 lines
Quoted 13 lines

2. Break point handling:
The runtime allows instrumenting of programs it executes. A system operator can mark code paths
to see how often they are executed. So the runtime installs a break point instruction at the desired
place and handles the breakpoint exception in the primary domain where just a counter
is incremented.
Although this slows down execution and hurts the system response time it is still fast enough
to satisfy our real-time requirements.
On the other hand switching to the secondary domain is far too expensive.

Quoted 2 lines

But this is not debugging, this is UPROBES / UPROBE_EVENTS, no? Maybe better make that work with oob if it does not yet.

I don't disagree that uprobes might help but the whole point of our patch is exception delivery to userspace. How userspace uses the exception is not set in stone.
For example, the application itself is modifying the code and places breakpoint instructions.
Another obstacle is that the application that runs on top of the Xenomai based framework
is not a regular Linux application, it does not even follow the calling convention, I bet this
makes things with uprobes not easier.

Quoted 10 lines
Quoted 6 lines

Maybe we can combine both approaches? We'll happily port the current patch to Xenomai 3.1
and give the monitor thread approach a try.

I'd like to see how you implemented that signal-like context switching in your approach. That is where I shied away from, rather using a thread switch plus a (not too nice, granted) mcontext get/set for other threads.

The key is ipipe_trap_hook().
__ipipe_notify_trap() passes a struct pt_regs of the trapping task to ipipe_trap_hook(). In ipipe_trap_hook() a signal frame is placed on the stack and
program counter, stack pointer, etc. in pt_regs are set accordingly. After ipipe_trap_hook() returns, the entry logic in arch/arm/ or arch/x86/ will restore
from the modified pt_regs and the trapping task executes the signal handler.

Quoted 4 lines

My current understanding is that a real-time domain signal mechanism will require changes in the pipeline patch underneath, thus first of all needs to be sold to the dovetail infrastructure (and then we can discuss if we backport that to ipipe anymore).

I'm pretty sure our approach has issues but so far no changes to ipipe are needed.

Signature · 2 lines
Thanks,
//richard
author
#8 lore raw
Quoted 60 lines · Richard Weinberger
Quoted 8 lines · Jan Kiszka

We have kind of a similar case in our backlog. The application expects to handle certain FPU-related faults gracefully, i.e. without falling out of RT with the causing thread, using a context update to bring it on the road again.

We modeled that case on top of the debug monitor thread, waking it up also on such faults and giving it a chance to remotely fix up the triggering thread.

Yeah, sounds related and I think useful for our case too.

Quoted 14 lines
Quoted 13 lines

2. Break point handling:
The runtime allows instrumenting of programs it executes. A system operator can mark code paths
to see how often they are executed. So the runtime installs a break point instruction at the desired
place and handles the breakpoint exception in the primary domain where just a counter
is incremented.
Although this slows down execution and hurts the system response time it is still fast enough
to satisfy our real-time requirements.
On the other hand switching to the secondary domain is far too expensive.

Quoted 2 lines

But this is not debugging, this is UPROBES / UPROBE_EVENTS, no? Maybe better make that work with oob if it does not yet.

I don't disagree that uprobes might help but the whole point of our patch is exception delivery to userspace. How userspace uses the exception is not set in stone.
For example, the application itself is modifying the code and places breakpoint instructions.
Another obstacle is that the application that runs on top of the Xenomai based framework
is not a regular Linux application, it does not even follow the calling convention, I bet this
makes things with uprobes not easier.

Quoted 10 lines
Quoted 6 lines

Maybe we can combine both approaches? We'll happily port the current patch to Xenomai 3.1
and give the monitor thread approach a try.

I'd like to see how you implemented that signal-like context switching in your approach. That is where I shied away from, rather using a thread switch plus a (not too nice, granted) mcontext get/set for other threads.

The key is ipipe_trap_hook().
__ipipe_notify_trap() passes a struct pt_regs of the trapping task to ipipe_trap_hook(). In ipipe_trap_hook() a signal frame is placed on the stack and
program counter, stack pointer, etc. in pt_regs are set accordingly. After ipipe_trap_hook() returns, the entry logic in arch/arm/ or arch/x86/ will restore
from the modified pt_regs and the trapping task executes the signal handler.

Quoted 8 lines
Quoted 4 lines

My current understanding is that a real-time domain signal mechanism will require changes in the pipeline patch underneath, thus first of all needs to be sold to the dovetail infrastructure (and then we can discuss if we backport that to ipipe anymore).

I'm pretty sure our approach has issues but so far no changes to ipipe are needed.

Then look at dovetail - ipipe will vanish. I'm pretty sure now your approach faces the same issue like we do /wrt not delivering fixed-up faults.

If that is the case, we should first of all ensure that the foundation is provided by the pipeline. Then we can sort out the API model and implementation details at Xenomai-level.

Jan

Signature · 2 lines
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux
#9 lore raw
Quoted 7 lines · Jan Kiszka

Then look at dovetail - ipipe will vanish. I'm pretty sure now your approach faces the same issue like we do /wrt not delivering fixed-up faults.

If that is the case, we should first of all ensure that the foundation is provided by the pipeline. Then we can sort out the API model and implementation details at Xenomai-level.

I've heard that with dovetail Xenomai no longer sees exceptions. So this is a set in stone concept?

Signature · 2 lines
Thanks,
//richard
author
#10 lore raw
Quoted 11 lines · Richard Weinberger
Quoted 7 lines · Jan Kiszka

Then look at dovetail - ipipe will vanish. I'm pretty sure now your approach faces the same issue like we do /wrt not delivering fixed-up faults.

If that is the case, we should first of all ensure that the foundation is provided by the pipeline. Then we can sort out the API model and implementation details at Xenomai-level.

I've heard that with dovetail Xenomai no longer sees exceptions. So this is a set in stone concept?

Xenomai can no longer declare exceptions resolved. It still sees them.

If we come up with a reasonable use case, I can't imagine that this is set in stone.

Jan

Signature · 2 lines
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux
#11 lore raw
Quoted 4 lines · Jan Kiszka
Quoted 2 lines

I've heard that with dovetail Xenomai no longer sees exceptions. So this is a set in stone concept?

Xenomai can no longer declare exceptions resolved. It still sees them.

Understood.

Quoted 2 lines

If we come up with a reasonable use case, I can't imagine that this is set in stone.

I can understand that dovetail doesn't want to offer something super generic as ipipe_trap_hook().
But maybe it can offer enough to implement signal delivery for exceptions?

The signal approach also allows us to use the very same signal handlers in userspace
for realtime as provided by Xenomai and PREEMPT_RT.

Signature · 2 lines
Thanks,
//richard
#12 lore raw
Quoted 6 lines · Richard Weinberger via Xenomai
Quoted 4 lines · Jan Kiszka
Quoted 2 lines

I've heard that with dovetail Xenomai no longer sees exceptions. So this is a set in stone concept?

Xenomai can no longer declare exceptions resolved. It still sees them.

Quoted 7 lines

Understood.

Quoted 2 lines

If we come up with a reasonable use case, I can't imagine that this is set in stone.

I can understand that dovetail doesn't want to offer something super generic as ipipe_trap_hook().

Just like the I-pipe, Dovetail implements a way to tell the companion core that "something happened", including a fault for the current task, by calling handlers with weak bindings the core can override. For faults, this would be oob_trap_notify().

Quoted 5 lines

But maybe it can offer enough to implement signal delivery for exceptions?

The signal approach also allows us to use the very same signal handlers in userspace
for realtime as provided by Xenomai and PREEMPT_RT.

You may want to post a proposal of such a feature for a recent Dovetail release, so that we can discuss it, and figure out whether it can be easily maintained for all the supported architectures in the long run. This is the key issue to me.

Signature · 1 line
Philippe.
#13 lore raw
Quoted 13 lines · Philippe Gerum
Quoted 11 lines · Richard Weinberger via Xenomai
Quoted 4 lines · Jan Kiszka
Quoted 2 lines

I've heard that with dovetail Xenomai no longer sees exceptions. So this is a set in stone concept?

Xenomai can no longer declare exceptions resolved. It still sees them.

Understood.

Quoted 2 lines

If we come up with a reasonable use case, I can't imagine that this is set in stone.

Quoted 7 lines
Quoted 2 lines

I can understand that dovetail doesn't want to offer something super generic as ipipe_trap_hook().

Just like the I-pipe, Dovetail implements a way to tell the companion core that "something happened", including a fault for the current task, by calling handlers with weak bindings the core can override. For faults, this would be oob_trap_notify().

oob_trap_notify() is a Dovetail service calling the following weak hooks which should be provided by the core actually: handle_oob_trap_entry() then handle_oob_trap_exit().

Signature · 1 line
Philippe.
#14 lore raw
In reply to Philippe Gerum
Quoted 10 lines · Philippe Gerum
Quoted 5 lines

But maybe it can offer enough to implement signal delivery for exceptions?

The signal approach also allows us to use the very same signal handlers in userspace
for realtime as provided by Xenomai and PREEMPT_RT.

You may want to post a proposal of such a feature for a recent Dovetail release, so that we can discuss it, and figure out whether it can be easily maintained for all the supported architectures in the long run. This is the key issue to me.

True. Time to explore dovetail in more detail. :-)

Signature · 2 lines
Thanks,
//richard