Skip to contentSkip to conversation

rtapp monitor update

PATCH In review Nam Cao 3 participants 17 messages Jun 19 – Jul 23

Subsystems (3)
lore
Share & tools
lore

Nam Cao

author
#0 lore raw

v2 cover rtapp monitor update

A couple of minor improvements to the rtapp monitor:

- Making the monitor more informative to user by changing the context of the tracepoint into the monitored task itself, not the IPI wakeup path.

- and update the allow list regarding clock_nanosleep syscall.

- Stop monitoring the kernel threads to simplify the monitors.

- Add a new rtapp/wakeup monitor to give complement the rtapp/sleep monitor.

v2..v1 https://lore.kernel.org/lkml/[email protected]/
- Use clearer waker/wakee terminologies
- Fix build issue
- Add new patch "rv/rtapp/sleep: Stop monitoring kernel threads"
- Require RV_PER_TASK_MONITORS >= 3

Nam Cao (4):
rv/rtapp/sleep: Make the error more informative for user rv/rtapp/sleep: Update nanosleep rule rv/rtapp/sleep: Stop monitoring kernel threads rv/rtapp: Add wakeup monitor

 Documentation/trace/rv/monitor_rtapp.rst      |  61 ++++---
 kernel/trace/rv/Kconfig                       |   1 +
 kernel/trace/rv/Makefile                      |   1 +
 kernel/trace/rv/monitors/rtapp/Kconfig        |   2 +-
 kernel/trace/rv/monitors/sleep/Kconfig        |   1 -
 kernel/trace/rv/monitors/sleep/sleep.c        |  59 ++-----
 kernel/trace/rv/monitors/sleep/sleep.h        | 142 +++++++---------
 kernel/trace/rv/monitors/wakeup/Kconfig       |  16 ++
 kernel/trace/rv/monitors/wakeup/wakeup.c      | 153 ++++++++++++++++++
 kernel/trace/rv/monitors/wakeup/wakeup.h      |  92 +++++++++++
 .../trace/rv/monitors/wakeup/wakeup_trace.h   |  14 ++
 kernel/trace/rv/rv_trace.h                    |   1 +
 tools/verification/models/rtapp/sleep.ltl     |  11 +-
 tools/verification/models/rtapp/wakeup.ltl    |   5 +
 14 files changed, 396 insertions(+), 163 deletions(-)
 create mode 100644 kernel/trace/rv/monitors/wakeup/Kconfig
 create mode 100644 kernel/trace/rv/monitors/wakeup/wakeup.c
 create mode 100644 kernel/trace/rv/monitors/wakeup/wakeup.h
 create mode 100644 kernel/trace/rv/monitors/wakeup/wakeup_trace.h
 create mode 100644 tools/verification/models/rtapp/wakeup.ltl
Signature · 1 line
2.47.3
author
#1 lore raw

PATCH v2 1/4 rv/rtapp/sleep: Make the error more informative for user

The rtapp/sleep monitor detects real-time tasks which go to sleep in an real-time-unsafe manner. If this happen, the monitor triggers a trace event in the sched_wakeup tracepoint's handler.

However, the invoking context of that trace event is not the most informative, because of the stack trace of that event is the wakeup's code path which is not very helpful:

74.669317: rv:error_sleep: condvar[254]: violation detected

    ltl_validate+0x345 ([kernel.kallsyms])
    handle_sched_wakeup+0x34 ([kernel.kallsyms])
    ttwu_do_activate+0xff ([kernel.kallsyms])
    sched_ttwu_pending+0x104 ([kernel.kallsyms])
    __flush_smp_call_function_queue+0x15b ([kernel.kallsyms])
    __sysvec_call_function_single+0x18 ([kernel.kallsyms])
    sysvec_call_function_single+0x66 ([kernel.kallsyms])
    asm_sysvec_call_function_single+0x1a ([kernel.kallsyms])
    pv_native_safe_halt+0xf ([kernel.kallsyms])
    default_idle+0x9 ([kernel.kallsyms])
    default_idle_call+0x33 ([kernel.kallsyms])
    do_idle+0x234 ([kernel.kallsyms])
    cpu_startup_entry+0x24 ([kernel.kallsyms])
    start_secondary+0xf8 ([kernel.kallsyms])
    common_startup_64+0x13e ([kernel.kallsyms])

What would be much more valuable is the stack trace of the task itself.

Instead of using the sched_wakeup tracepoint, use the sched_exit tracepoint. This makes the event happen in the task's context, making the stack trace far more informative for user:

rv:error_sleep: condvar[254]: violation detected

    ltl_validate+0x345 ([kernel.kallsyms])
    handle_sched_exit+0x39 ([kernel.kallsyms])
    __schedule+0x80f ([kernel.kallsyms])
    schedule+0x22 ([kernel.kallsyms])
    futex_do_wait+0x33 ([kernel.kallsyms])
    __futex_wait+0x8c ([kernel.kallsyms])
    futex_wait+0x73 ([kernel.kallsyms])
    do_futex+0xc6 ([kernel.kallsyms])
    __x64_sys_futex+0x121 ([kernel.kallsyms])
    do_syscall_64+0xf3 ([kernel.kallsyms])
    entry_SYSCALL_64_after_hwframe+0x77 ([kernel.kallsyms])
    __futex_abstimed_wait_common64+0xc6 (inlined)
    __futex_abstimed_wait_common+0xc6 (/usr/lib/x86_64-linux-gnu/libc.so.6)
---
 Documentation/trace/rv/monitor_rtapp.rst  |  2 +-
 kernel/trace/rv/monitors/sleep/sleep.c    | 10 +++++-----
 kernel/trace/rv/monitors/sleep/sleep.h    | 14 +++++++-------
 tools/verification/models/rtapp/sleep.ltl |  2 +-
 4 files changed, 14 insertions(+), 14 deletions(-)
Documentation/trace/rv/monitor_rtapp.rst +1 -1
index c8104eda924a..01656bf7080a 100644
@@ -95,7 +95,7 @@ The monitor's specification is::
   RULE = always ((RT and SLEEP) imply (RT_FRIENDLY_SLEEP or ALLOWLIST))
 
   RT_FRIENDLY_SLEEP = (RT_VALID_SLEEP_REASON or KERNEL_THREAD)
-                  and ((not WAKE) until RT_FRIENDLY_WAKE)
+                  and ((not SCHEDULE_IN) until RT_FRIENDLY_WAKE)
 
   RT_VALID_SLEEP_REASON = FUTEX_WAIT
                        or RT_FRIENDLY_NANOSLEEP
kernel/trace/rv/monitors/sleep/sleep.c +5 -5
index 8dfe5ec13e19..d6b677fab8f8 100644
@@ -36,7 +36,7 @@ static void ltl_atoms_fetch(struct task_struct *task, struct ltl_monitor *mon)
 static void ltl_atoms_init(struct task_struct *task, struct ltl_monitor *mon, bool task_creation)
 {
 	ltl_atom_set(mon, LTL_SLEEP, false);
-	ltl_atom_set(mon, LTL_WAKE, false);
+	ltl_atom_set(mon, LTL_SCHEDULE_IN, false);
 	ltl_atom_set(mon, LTL_ABORT_SLEEP, false);
 	ltl_atom_set(mon, LTL_WOKEN_BY_HARDIRQ, false);
 	ltl_atom_set(mon, LTL_WOKEN_BY_NMI, false);
@@ -92,9 +92,9 @@ static void handle_sched_set_state(void *data, struct task_struct *task, int sta
 		ltl_atom_pulse(task, LTL_ABORT_SLEEP, true);
 }
 
-static void handle_sched_wakeup(void *data, struct task_struct *task)
+static void handle_sched_exit(void *data, bool is_switch)
 {
-	ltl_atom_pulse(task, LTL_WAKE, true);
+	ltl_atom_pulse(current, LTL_SCHEDULE_IN, true);
 }
 
 static void handle_sched_waking(void *data, struct task_struct *task)
@@ -200,7 +200,7 @@ static int enable_sleep(void)
 		return retval;
 
 	rv_attach_trace_probe("rtapp_sleep", sched_waking, handle_sched_waking);
-	rv_attach_trace_probe("rtapp_sleep", sched_wakeup, handle_sched_wakeup);
+	rv_attach_trace_probe("rtapp_sleep", sched_exit_tp, handle_sched_exit);
 	rv_attach_trace_probe("rtapp_sleep", sched_set_state_tp, handle_sched_set_state);
 	rv_attach_trace_probe("rtapp_sleep", contention_begin, handle_contention_begin);
 	rv_attach_trace_probe("rtapp_sleep", contention_end, handle_contention_end);
@@ -213,7 +213,7 @@ static int enable_sleep(void)
 static void disable_sleep(void)
 {
 	rv_detach_trace_probe("rtapp_sleep", sched_waking, handle_sched_waking);
-	rv_detach_trace_probe("rtapp_sleep", sched_wakeup, handle_sched_wakeup);
+	rv_detach_trace_probe("rtapp_sleep", sched_exit_tp, handle_sched_exit);
 	rv_detach_trace_probe("rtapp_sleep", sched_set_state_tp, handle_sched_set_state);
 	rv_detach_trace_probe("rtapp_sleep", contention_begin, handle_contention_begin);
 	rv_detach_trace_probe("rtapp_sleep", contention_end, handle_contention_end);
kernel/trace/rv/monitors/sleep/sleep.h +7 -7
index 95dc2727c059..403dc2852c52 100644
@@ -24,10 +24,10 @@ enum ltl_atom {
 	LTL_NANOSLEEP_CLOCK_TAI,
 	LTL_NANOSLEEP_TIMER_ABSTIME,
 	LTL_RT,
+	LTL_SCHEDULE_IN,
 	LTL_SLEEP,
 	LTL_TASK_IS_MIGRATION,
 	LTL_TASK_IS_RCU,
-	LTL_WAKE,
 	LTL_WOKEN_BY_EQUAL_OR_HIGHER_PRIO,
 	LTL_WOKEN_BY_HARDIRQ,
 	LTL_WOKEN_BY_NMI,
@@ -50,10 +50,10 @@ static const char *ltl_atom_str(enum ltl_atom atom)
 		"na_cl_ta",
 		"na_ti_ab",
 		"rt",
-		"sl",
+		"sch_in",
+		"sle",
 		"ta_mi",
 		"ta_rc",
-		"wak",
 		"wo_eq_hi_pr",
 		"wo_ha",
 		"wo_nm",
@@ -81,10 +81,10 @@ static void ltl_start(struct task_struct *task, struct ltl_monitor *mon)
 	bool woken_by_hardirq = test_bit(LTL_WOKEN_BY_HARDIRQ, mon->atoms);
 	bool woken_by_equal_or_higher_prio = test_bit(LTL_WOKEN_BY_EQUAL_OR_HIGHER_PRIO,
 	     mon->atoms);
-	bool wake = test_bit(LTL_WAKE, mon->atoms);
 	bool task_is_rcu = test_bit(LTL_TASK_IS_RCU, mon->atoms);
 	bool task_is_migration = test_bit(LTL_TASK_IS_MIGRATION, mon->atoms);
 	bool sleep = test_bit(LTL_SLEEP, mon->atoms);
+	bool schedule_in = test_bit(LTL_SCHEDULE_IN, mon->atoms);
 	bool rt = test_bit(LTL_RT, mon->atoms);
 	bool nanosleep_timer_abstime = test_bit(LTL_NANOSLEEP_TIMER_ABSTIME, mon->atoms);
 	bool nanosleep_clock_tai = test_bit(LTL_NANOSLEEP_CLOCK_TAI, mon->atoms);
@@ -104,7 +104,7 @@ static void ltl_start(struct task_struct *task, struct ltl_monitor *mon)
 	bool val35 = woken_by_nmi || val34;
 	bool val36 = woken_by_hardirq || val35;
 	bool val14 = woken_by_equal_or_higher_prio || val36;
-	bool val13 = !wake;
+	bool val13 = !schedule_in;
 	bool val26 = nanosleep_clock_monotonic || nanosleep_clock_tai;
 	bool val27 = nanosleep_timer_abstime && val26;
 	bool val18 = clock_nanosleep && val27;
@@ -132,10 +132,10 @@ ltl_possible_next_states(struct ltl_monitor *mon, unsigned int state, unsigned l
 	bool woken_by_hardirq = test_bit(LTL_WOKEN_BY_HARDIRQ, mon->atoms);
 	bool woken_by_equal_or_higher_prio = test_bit(LTL_WOKEN_BY_EQUAL_OR_HIGHER_PRIO,
 	     mon->atoms);
-	bool wake = test_bit(LTL_WAKE, mon->atoms);
 	bool task_is_rcu = test_bit(LTL_TASK_IS_RCU, mon->atoms);
 	bool task_is_migration = test_bit(LTL_TASK_IS_MIGRATION, mon->atoms);
 	bool sleep = test_bit(LTL_SLEEP, mon->atoms);
+	bool schedule_in = test_bit(LTL_SCHEDULE_IN, mon->atoms);
 	bool rt = test_bit(LTL_RT, mon->atoms);
 	bool nanosleep_timer_abstime = test_bit(LTL_NANOSLEEP_TIMER_ABSTIME, mon->atoms);
 	bool nanosleep_clock_tai = test_bit(LTL_NANOSLEEP_CLOCK_TAI, mon->atoms);
@@ -155,7 +155,7 @@ ltl_possible_next_states(struct ltl_monitor *mon, unsigned int state, unsigned l
 	bool val35 = woken_by_nmi || val34;
 	bool val36 = woken_by_hardirq || val35;
 	bool val14 = woken_by_equal_or_higher_prio || val36;
-	bool val13 = !wake;
+	bool val13 = !schedule_in;
 	bool val26 = nanosleep_clock_monotonic || nanosleep_clock_tai;
 	bool val27 = nanosleep_timer_abstime && val26;
 	bool val18 = clock_nanosleep && val27;
tools/verification/models/rtapp/sleep.ltl +1 -1
index 6f26c4810f78..464c84b9df87 100644
@@ -1,7 +1,7 @@
 RULE = always ((RT and SLEEP) imply (RT_FRIENDLY_SLEEP or ALLOWLIST))
 
 RT_FRIENDLY_SLEEP = (RT_VALID_SLEEP_REASON or KERNEL_THREAD)
-                and ((not WAKE) until RT_FRIENDLY_WAKE)
+                and ((not SCHEDULE_IN) until RT_FRIENDLY_WAKE)
 
 RT_VALID_SLEEP_REASON = FUTEX_WAIT
                      or RT_FRIENDLY_NANOSLEEP
Signature · 1 line
2.47.3
author
#4 lore raw

PATCH v2 2/4 rv/rtapp/sleep: Update nanosleep rule

In reply to Nam Cao

CLOCK_REALTIME is the only clock that often is misused in real-time applications. The other clocks either are safe for real-time uses (CLOCK_TAI, CLOCK_MONOTONIC, CLOCK_BOOTTIME) or are unlikely to be misused (CLOCK_AUX, CLOCK_PROCESS_CPUTIME_ID).

Update the monitor to only warn about CLOCK_REALTIME.

While at it, update the out-of-sync documentation.

---
 Documentation/trace/rv/monitor_rtapp.rst  | 17 +++++---
 kernel/trace/rv/monitors/sleep/sleep.c    | 12 ++----
 kernel/trace/rv/monitors/sleep/sleep.h    | 52 +++++++++++------------
 tools/verification/models/rtapp/sleep.ltl |  2 +-
 4 files changed, 39 insertions(+), 44 deletions(-)
Documentation/trace/rv/monitor_rtapp.rst +10 -7
index 01656bf7080a..570be67a8f3b 100644
@@ -51,12 +51,13 @@ The `sleep` monitor reports real-time threads sleeping in a manner that may
 cause undesirable latency. Real-time applications should only put a real-time
 thread to sleep for one of the following reasons:
 
-  - Cyclic work: real-time thread sleeps waiting for the next cycle. For this
-    case, only the `clock_nanosleep` syscall should be used with `TIMER_ABSTIME`
-    (to avoid time drift) and `CLOCK_MONOTONIC` (to avoid the clock being
-    changed). No other method is safe for real-time. For example, threads
-    waiting for timerfd can be woken by softirq which provides no real-time
-    guarantee.
+  - Cyclic work: real-time thread sleeps waiting for the next
+    cycle. For this case, only the `clock_nanosleep` syscall should be
+    used with `TIMER_ABSTIME` (to avoid time drift). Additionally,
+    `CLOCK_REALTIME` should not be used (to avoid the clock being
+    changed). No other method is safe for real-time. For example,
+    threads waiting for timerfd can be woken by softirq which provides
+    no real-time guarantee.
   - Real-time thread waiting for something to happen (e.g. another thread
     releasing shared resources, or a completion signal from another thread). In
     this case, only futexes (FUTEX_LOCK_PI, FUTEX_LOCK_PI2 or one of
@@ -99,14 +100,16 @@ The monitor's specification is::
 
   RT_VALID_SLEEP_REASON = FUTEX_WAIT
                        or RT_FRIENDLY_NANOSLEEP
+                       or EPOLL_WAIT
 
   RT_FRIENDLY_NANOSLEEP = CLOCK_NANOSLEEP
                       and NANOSLEEP_TIMER_ABSTIME
-                      and NANOSLEEP_CLOCK_MONOTONIC
+                      and not NANOSLEEP_CLOCK_REALTIME
 
   RT_FRIENDLY_WAKE = WOKEN_BY_EQUAL_OR_HIGHER_PRIO
                   or WOKEN_BY_HARDIRQ
                   or WOKEN_BY_NMI
+                  or ABORT_SLEEP
                   or KTHREAD_SHOULD_STOP
 
   ALLOWLIST = BLOCK_ON_RT_MUTEX
kernel/trace/rv/monitors/sleep/sleep.c +4 -8
index d6b677fab8f8..638be7d8747f 100644
@@ -44,8 +44,7 @@ static void ltl_atoms_init(struct task_struct *task, struct ltl_monitor *mon, bo
 
 	if (task_creation) {
 		ltl_atom_set(mon, LTL_KTHREAD_SHOULD_STOP, false);
-		ltl_atom_set(mon, LTL_NANOSLEEP_CLOCK_MONOTONIC, false);
-		ltl_atom_set(mon, LTL_NANOSLEEP_CLOCK_TAI, false);
+		ltl_atom_set(mon, LTL_NANOSLEEP_CLOCK_REALTIME, false);
 		ltl_atom_set(mon, LTL_NANOSLEEP_TIMER_ABSTIME, false);
 		ltl_atom_set(mon, LTL_CLOCK_NANOSLEEP, false);
 		ltl_atom_set(mon, LTL_FUTEX_WAIT, false);
@@ -60,8 +59,7 @@ static void ltl_atoms_init(struct task_struct *task, struct ltl_monitor *mon, bo
 		/* kernel tasks do not do syscall */
 		ltl_atom_set(mon, LTL_FUTEX_WAIT, false);
 		ltl_atom_set(mon, LTL_FUTEX_LOCK_PI, false);
-		ltl_atom_set(mon, LTL_NANOSLEEP_CLOCK_MONOTONIC, false);
-		ltl_atom_set(mon, LTL_NANOSLEEP_CLOCK_TAI, false);
+		ltl_atom_set(mon, LTL_NANOSLEEP_CLOCK_REALTIME, false);
 		ltl_atom_set(mon, LTL_NANOSLEEP_TIMER_ABSTIME, false);
 		ltl_atom_set(mon, LTL_CLOCK_NANOSLEEP, false);
 		ltl_atom_set(mon, LTL_EPOLL_WAIT, false);
@@ -136,8 +134,7 @@ static void handle_sys_enter(void *data, struct pt_regs *regs, long id)
 	case __NR_clock_nanosleep_time64:
 #endif
 		syscall_get_arguments(current, regs, args);
-		ltl_atom_set(mon, LTL_NANOSLEEP_CLOCK_MONOTONIC, args[0] == CLOCK_MONOTONIC);
-		ltl_atom_set(mon, LTL_NANOSLEEP_CLOCK_TAI, args[0] == CLOCK_TAI);
+		ltl_atom_set(mon, LTL_NANOSLEEP_CLOCK_REALTIME, args[0] == CLOCK_REALTIME);
 		ltl_atom_set(mon, LTL_NANOSLEEP_TIMER_ABSTIME, args[1] == TIMER_ABSTIME);
 		ltl_atom_update(current, LTL_CLOCK_NANOSLEEP, true);
 		break;
@@ -178,8 +175,7 @@ static void handle_sys_exit(void *data, struct pt_regs *regs, long ret)
 
 	ltl_atom_set(mon, LTL_FUTEX_LOCK_PI, false);
 	ltl_atom_set(mon, LTL_FUTEX_WAIT, false);
-	ltl_atom_set(mon, LTL_NANOSLEEP_CLOCK_MONOTONIC, false);
-	ltl_atom_set(mon, LTL_NANOSLEEP_CLOCK_TAI, false);
+	ltl_atom_set(mon, LTL_NANOSLEEP_CLOCK_REALTIME, false);
 	ltl_atom_set(mon, LTL_NANOSLEEP_TIMER_ABSTIME, false);
 	ltl_atom_set(mon, LTL_EPOLL_WAIT, false);
 	ltl_atom_update(current, LTL_CLOCK_NANOSLEEP, false);
kernel/trace/rv/monitors/sleep/sleep.h +24 -28

90 lines not on this page · raw message

tools/verification/models/rtapp/sleep.ltl +1 -1
index 464c84b9df87..5923e58d7810 100644
@@ -9,7 +9,7 @@ RT_VALID_SLEEP_REASON = FUTEX_WAIT
 
 RT_FRIENDLY_NANOSLEEP = CLOCK_NANOSLEEP
                     and NANOSLEEP_TIMER_ABSTIME
-                    and (NANOSLEEP_CLOCK_MONOTONIC or NANOSLEEP_CLOCK_TAI)
+                    and not NANOSLEEP_CLOCK_REALTIME
 
 RT_FRIENDLY_WAKE = WOKEN_BY_EQUAL_OR_HIGHER_PRIO
                 or WOKEN_BY_HARDIRQ
Signature · 1 line
2.47.3
author
#6 lore raw

PATCH v2 3/4 rv/rtapp/sleep: Stop monitoring kernel threads

In reply to Nam Cao

The rtapp/sleep monitor's primary purpose is detecting common mistakes with user-space real-time design. Monitoring real-time issues with kernel threads is a bonus.

However, accomodating kernel threads complicates the monitor due to the edge cases which is seen by the monitor as lower-priority task waking higher-priority task:

- kthread_stop() wakes up the task in order to stop it.

- The rcu thread and migration thread can be woken by any task.

- The ktimerd thread is woken near the end of irq_exit_rcu(), where the preempt counter is "broken" and falsely says this is task context. This requires the monitor to use the hardirq_context flag instead of the preempt counter.

Beside complicating the monitor, the final case also requires enabling CONFIG_TRACE_IRQFLAGS (so that "hardirq_context" can be used). This adds overhead to the kernel even when the monitor is not active. This may be an obstacle to enabling this monitor in distros' kernels.

Furthermore, kernel threads usually are started before the monitor is enabled. Consequently, the threads' states (i.o.w. the monitor's atomic propositions for the threads) are not fully known to the monitor. As a result, the kernel threads mostly cannot be monitored.

Overall, the downsides of accomodating kernel threads outweights the benefits. Thus, exclude kernel threads to simplify the monitor.

---
Cc: Sebastian Andrzej Siewior <[email protected]>

---
 Documentation/trace/rv/monitor_rtapp.rst  |  22 ++---
 kernel/trace/rv/monitors/sleep/Kconfig    |   1 -
 kernel/trace/rv/monitors/sleep/sleep.c    |  39 +-------
 kernel/trace/rv/monitors/sleep/sleep.h    | 104 +++++++++-------------
 tools/verification/models/rtapp/sleep.ltl |   7 +-
 5 files changed, 54 insertions(+), 119 deletions(-)
Documentation/trace/rv/monitor_rtapp.rst +6 -16
index 570be67a8f3b..502d3ea412eb 100644
@@ -93,9 +93,9 @@ assessment.
 
 The monitor's specification is::
 
-  RULE = always ((RT and SLEEP) imply (RT_FRIENDLY_SLEEP or ALLOWLIST))
+  RULE = always ((RT and SLEEP and USER_THREAD) imply (RT_FRIENDLY_SLEEP or ALLOWLIST))
 
-  RT_FRIENDLY_SLEEP = (RT_VALID_SLEEP_REASON or KERNEL_THREAD)
+  RT_FRIENDLY_SLEEP = RT_VALID_SLEEP_REASON
                   and ((not SCHEDULE_IN) until RT_FRIENDLY_WAKE)
 
   RT_VALID_SLEEP_REASON = FUTEX_WAIT
@@ -110,23 +110,13 @@ The monitor's specification is::
                   or WOKEN_BY_HARDIRQ
                   or WOKEN_BY_NMI
                   or ABORT_SLEEP
-                  or KTHREAD_SHOULD_STOP
 
   ALLOWLIST = BLOCK_ON_RT_MUTEX
            or FUTEX_LOCK_PI
-           or TASK_IS_RCU
-           or TASK_IS_MIGRATION
-
-Beside the scenarios described above, this specification also handle some
-special cases:
-
-  - `KERNEL_THREAD`: kernel tasks do not have any pattern that can be recognized
-    as valid real-time sleeping reasons. Therefore sleeping reason is not
-    checked for kernel tasks.
-  - `KTHREAD_SHOULD_STOP`: a non-real-time thread may stop a real-time kernel
-    thread by waking it and waiting for it to exit (`kthread_stop()`). This
-    wakeup is safe for real-time.
-  - `ALLOWLIST`: to handle known false positives with the kernel.
+
+Beside the scenarios described above, this specification also defines an allow list
+to handle some special cases:
+
   - `BLOCK_ON_RT_MUTEX` is included in the allowlist due to its implementation.
     In the release path of rt_mutex, a boosted task is de-boosted before waking
     the rt_mutex's waiter. Consequently, the monitor may see a real-time-unsafe
kernel/trace/rv/monitors/sleep/Kconfig +0 -1
index 6b7a122e7b47..d6ec3e9a91b6 100644
@@ -5,7 +5,6 @@ config RV_MON_SLEEP
 	select RV_LTL_MONITOR
 	depends on HAVE_SYSCALL_TRACEPOINTS
 	depends on RV_MON_RTAPP
-	select TRACE_IRQFLAGS
 	default y
 	select LTL_MON_EVENTS_ID
 	bool "sleep monitor"
kernel/trace/rv/monitors/sleep/sleep.c +2 -37
index 638be7d8747f..aa5a984853b5 100644
@@ -43,7 +43,6 @@ static void ltl_atoms_init(struct task_struct *task, struct ltl_monitor *mon, bo
 	ltl_atom_set(mon, LTL_WOKEN_BY_EQUAL_OR_HIGHER_PRIO, false);
 
 	if (task_creation) {
-		ltl_atom_set(mon, LTL_KTHREAD_SHOULD_STOP, false);
 		ltl_atom_set(mon, LTL_NANOSLEEP_CLOCK_REALTIME, false);
 		ltl_atom_set(mon, LTL_NANOSLEEP_TIMER_ABSTIME, false);
 		ltl_atom_set(mon, LTL_CLOCK_NANOSLEEP, false);
@@ -53,33 +52,7 @@ static void ltl_atoms_init(struct task_struct *task, struct ltl_monitor *mon, bo
 		ltl_atom_set(mon, LTL_BLOCK_ON_RT_MUTEX, false);
 	}
 
-	if (task->flags & PF_KTHREAD) {
-		ltl_atom_set(mon, LTL_KERNEL_THREAD, true);
-
-		/* kernel tasks do not do syscall */
-		ltl_atom_set(mon, LTL_FUTEX_WAIT, false);
-		ltl_atom_set(mon, LTL_FUTEX_LOCK_PI, false);
-		ltl_atom_set(mon, LTL_NANOSLEEP_CLOCK_REALTIME, false);
-		ltl_atom_set(mon, LTL_NANOSLEEP_TIMER_ABSTIME, false);
-		ltl_atom_set(mon, LTL_CLOCK_NANOSLEEP, false);
-		ltl_atom_set(mon, LTL_EPOLL_WAIT, false);
-
-		if (strstarts(task->comm, "migration/"))
-			ltl_atom_set(mon, LTL_TASK_IS_MIGRATION, true);
-		else
-			ltl_atom_set(mon, LTL_TASK_IS_MIGRATION, false);
-
-		if (strstarts(task->comm, "rcu"))
-			ltl_atom_set(mon, LTL_TASK_IS_RCU, true);
-		else
-			ltl_atom_set(mon, LTL_TASK_IS_RCU, false);
-	} else {
-		ltl_atom_set(mon, LTL_KTHREAD_SHOULD_STOP, false);
-		ltl_atom_set(mon, LTL_KERNEL_THREAD, false);
-		ltl_atom_set(mon, LTL_TASK_IS_RCU, false);
-		ltl_atom_set(mon, LTL_TASK_IS_MIGRATION, false);
-	}
-
+	ltl_atom_set(mon, LTL_USER_THREAD, !(task->flags & PF_KTHREAD));
 }
 
 static void handle_sched_set_state(void *data, struct task_struct *task, int state)
@@ -97,7 +70,7 @@ static void handle_sched_exit(void *data, bool is_switch)
 
 static void handle_sched_waking(void *data, struct task_struct *task)
 {
-	if (this_cpu_read(hardirq_context)) {
+	if (in_hardirq()) {
 		ltl_atom_pulse(task, LTL_WOKEN_BY_HARDIRQ, true);
 	} else if (in_task()) {
 		if (current->prio <= task->prio)
@@ -181,12 +154,6 @@ static void handle_sys_exit(void *data, struct pt_regs *regs, long ret)
 	ltl_atom_update(current, LTL_CLOCK_NANOSLEEP, false);
 }
 
-static void handle_kthread_stop(void *data, struct task_struct *task)
-{
-	/* FIXME: this could race with other tracepoint handlers */
-	ltl_atom_update(task, LTL_KTHREAD_SHOULD_STOP, true);
-}
-
 static int enable_sleep(void)
 {
 	int retval;
@@ -200,7 +167,6 @@ static int enable_sleep(void)
 	rv_attach_trace_probe("rtapp_sleep", sched_set_state_tp, handle_sched_set_state);
 	rv_attach_trace_probe("rtapp_sleep", contention_begin, handle_contention_begin);
 	rv_attach_trace_probe("rtapp_sleep", contention_end, handle_contention_end);
-	rv_attach_trace_probe("rtapp_sleep", sched_kthread_stop, handle_kthread_stop);
 	rv_attach_trace_probe("rtapp_sleep", sys_enter, handle_sys_enter);
 	rv_attach_trace_probe("rtapp_sleep", sys_exit, handle_sys_exit);
 	return 0;
@@ -213,7 +179,6 @@ static void disable_sleep(void)
 	rv_detach_trace_probe("rtapp_sleep", sched_set_state_tp, handle_sched_set_state);
 	rv_detach_trace_probe("rtapp_sleep", contention_begin, handle_contention_begin);
 	rv_detach_trace_probe("rtapp_sleep", contention_end, handle_contention_end);
-	rv_detach_trace_probe("rtapp_sleep", sched_kthread_stop, handle_kthread_stop);
 	rv_detach_trace_probe("rtapp_sleep", sys_enter, handle_sys_enter);
 	rv_detach_trace_probe("rtapp_sleep", sys_exit, handle_sys_exit);
 
kernel/trace/rv/monitors/sleep/sleep.h +44 -60

250 lines not on this page · raw message

tools/verification/models/rtapp/sleep.ltl +2 -5
index 5923e58d7810..4d78fdd204c0 100644
@@ -1,6 +1,6 @@
-RULE = always ((RT and SLEEP) imply (RT_FRIENDLY_SLEEP or ALLOWLIST))
+RULE = always ((RT and SLEEP and USER_THREAD) imply (RT_FRIENDLY_SLEEP or ALLOWLIST))
 
-RT_FRIENDLY_SLEEP = (RT_VALID_SLEEP_REASON or KERNEL_THREAD)
+RT_FRIENDLY_SLEEP = RT_VALID_SLEEP_REASON
                 and ((not SCHEDULE_IN) until RT_FRIENDLY_WAKE)
 
 RT_VALID_SLEEP_REASON = FUTEX_WAIT
@@ -15,9 +15,6 @@ RT_FRIENDLY_WAKE = WOKEN_BY_EQUAL_OR_HIGHER_PRIO
                 or WOKEN_BY_HARDIRQ
                 or WOKEN_BY_NMI
                 or ABORT_SLEEP
-                or KTHREAD_SHOULD_STOP
 
 ALLOWLIST = BLOCK_ON_RT_MUTEX
          or FUTEX_LOCK_PI
-         or TASK_IS_RCU
-         or TASK_IS_MIGRATION
Signature · 1 line
2.47.3
author
#8 lore raw

PATCH v2 4/4 rv/rtapp: Add wakeup monitor

In reply to Nam Cao

Add a wakeup monitor to detect a lower-priority task waking up a higher-priority task.

The rtapp/sleep monitor already detects this. However, that monitor triggers an error in the context of the wakee task and user only gets the stacktrace of that task. It is also extremely useful to get the stacktrace of the waker task, which this monitor offers. In other words, this monitor complements the rtapp/sleep monitor.

---
 Documentation/trace/rv/monitor_rtapp.rst      |  20 +++
 kernel/trace/rv/Kconfig                       |   1 +
 kernel/trace/rv/Makefile                      |   1 +
 kernel/trace/rv/monitors/rtapp/Kconfig        |   2 +-
 kernel/trace/rv/monitors/wakeup/Kconfig       |  16 ++
 kernel/trace/rv/monitors/wakeup/wakeup.c      | 153 ++++++++++++++++++
 kernel/trace/rv/monitors/wakeup/wakeup.h      |  92 +++++++++++
 .../trace/rv/monitors/wakeup/wakeup_trace.h   |  14 ++
 kernel/trace/rv/rv_trace.h                    |   1 +
 tools/verification/models/rtapp/wakeup.ltl    |   5 +
 10 files changed, 304 insertions(+), 1 deletion(-)
 create mode 100644 kernel/trace/rv/monitors/wakeup/Kconfig
 create mode 100644 kernel/trace/rv/monitors/wakeup/wakeup.c
 create mode 100644 kernel/trace/rv/monitors/wakeup/wakeup.h
 create mode 100644 kernel/trace/rv/monitors/wakeup/wakeup_trace.h
 create mode 100644 tools/verification/models/rtapp/wakeup.ltl
Documentation/trace/rv/monitor_rtapp.rst +20 -0
index 502d3ea412eb..238b59395ff5 100644
@@ -124,3 +124,23 @@ to handle some special cases:
     real-time-safe because preemption is disabled for the duration.
   - `FUTEX_LOCK_PI` is included in the allowlist for the same reason as
     `BLOCK_ON_RT_MUTEX`.
+
+Monitor wakeup
+++++++++++++++
+
+The `wakeup` monitor reports real-time threads being woken by lower-priority threads,
+which is a hint of priority inversion. Its specification is::
+
+  RULE = always (((RT and USER_THREAD) imply
+                (not (WOKEN_BY_LOWER_PRIO or WOKEN_BY_SOFTIRQ)) or ALLOWLIST))
+
+  ALLOWLIST = BLOCK_ON_RT_MUTEX
+           or FUTEX_LOCK_PI
+
+The `sleep` monitor already reports this type of problem. The difference is the
+context in which the problem is reported. While the `sleep` monitor reports the problem
+in the context of the wakee, this `wakeup` monitor reports the problem in the context of
+the waker. This monitor complement the `sleep` monitor, giving user better
+understanding of the issue. For instance, to debug a lower-priority task waking a
+higher-priority task scenario, user can enable both `wakeup` monitor and `sleep`
+monitor to get the stack traces of both tasks.
kernel/trace/rv/Kconfig +1 -0
index 3884b14df375..4d3a14a0bac2 100644
@@ -76,6 +76,7 @@ source "kernel/trace/rv/monitors/opid/Kconfig"
 source "kernel/trace/rv/monitors/rtapp/Kconfig"
 source "kernel/trace/rv/monitors/pagefault/Kconfig"
 source "kernel/trace/rv/monitors/sleep/Kconfig"
+source "kernel/trace/rv/monitors/wakeup/Kconfig"
 # Add new rtapp monitors here
 
 source "kernel/trace/rv/monitors/stall/Kconfig"
kernel/trace/rv/Makefile +1 -0
index 94498da35b37..c2c0e4142eb4 100644
@@ -20,6 +20,7 @@ obj-$(CONFIG_RV_MON_OPID) += monitors/opid/opid.o
 obj-$(CONFIG_RV_MON_STALL) += monitors/stall/stall.o
 obj-$(CONFIG_RV_MON_DEADLINE) += monitors/deadline/deadline.o
 obj-$(CONFIG_RV_MON_NOMISS) += monitors/nomiss/nomiss.o
+obj-$(CONFIG_RV_MON_WAKEUP) += monitors/wakeup/wakeup.o
 # Add new monitors here
 obj-$(CONFIG_RV_REACTORS) += rv_reactors.o
 obj-$(CONFIG_RV_REACT_PRINTK) += reactor_printk.o
kernel/trace/rv/monitors/rtapp/Kconfig +1 -1
index 1ce9370a9ba8..1fcd7a400ded 100644
@@ -1,6 +1,6 @@
 config RV_MON_RTAPP
 	depends on RV
-	depends on RV_PER_TASK_MONITORS >= 2
+	depends on RV_PER_TASK_MONITORS >= 3
 	bool "rtapp monitor"
 	help
 	  Collection of monitors to check for common problems with real-time
kernel/trace/rv/monitors/wakeup/Kconfig new file +16 -0
new file mode 100644
index 000000000000..ec3a5c06a8c4
@@ -0,0 +1,16 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+config RV_MON_WAKEUP
+	depends on RV
+	depends on RV_MON_RTAPP
+	depends on HAVE_SYSCALL_TRACEPOINTS
+	default y
+	select LTL_MON_EVENTS_ID
+	bool "wakeup monitor"
+	help
+	  This monitor detects a lower-priority task waking up a
+	  higher-priority task. The RV_MON_SLEEP monitor already
+	  detects this case, but this monitor detects in the context
+	  of the waker task instead. This and RV_MON_SLEEP can be
+	  enabled together to get the stacktrace of both the waker
+	  task and the wakee task.
kernel/trace/rv/monitors/wakeup/wakeup.c new file +153 -0

153 lines not on this page · raw message

kernel/trace/rv/monitors/wakeup/wakeup.h new file +92 -0

92 lines not on this page · raw message

kernel/trace/rv/monitors/wakeup/wakeup_trace.h new file +14 -0
new file mode 100644
index 000000000000..7e056183f920
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * Snippet to be included in rv_trace.h
+ */
+
+#ifdef CONFIG_RV_MON_WAKEUP
+DEFINE_EVENT(event_ltl_monitor_id, event_wakeup,
+	     TP_PROTO(struct task_struct *task, char *states, char *atoms, char *next),
+	     TP_ARGS(task, states, atoms, next));
+DEFINE_EVENT(error_ltl_monitor_id, error_wakeup,
+	     TP_PROTO(struct task_struct *task),
+	     TP_ARGS(task));
+#endif /* CONFIG_RV_MON_WAKEUP */
kernel/trace/rv/rv_trace.h +1 -0
index 9622c269789c..2f8a932432c9 100644
@@ -241,6 +241,7 @@ DECLARE_EVENT_CLASS(error_ltl_monitor_id,
 );
 #include <monitors/pagefault/pagefault_trace.h>
 #include <monitors/sleep/sleep_trace.h>
+#include <monitors/wakeup/wakeup_trace.h>
 // Add new monitors based on CONFIG_LTL_MON_EVENTS_ID here
 #endif /* CONFIG_LTL_MON_EVENTS_ID */
 
tools/verification/models/rtapp/wakeup.ltl new file +5 -0
new file mode 100644
index 000000000000..a5d63ca0811a
@@ -0,0 +1,5 @@
+RULE = always (((RT and USER_THREAD) imply
+		(not (WOKEN_BY_LOWER_PRIO or WOKEN_BY_SOFTIRQ)) or ALLOWLIST))
+
+ALLOWLIST = BLOCK_ON_RT_MUTEX
+         or FUTEX_LOCK_PI
Signature · 1 line
2.47.3
maintainer Reviewed-by on v2 1/4
#2 lore raw

Re: [PATCH v2 1/4] rv/rtapp/sleep: Make the error more informative for user

In reply to Nam Cao
Quoted 6 lines · Nam Cao
kernel/trace/rv/monitors/sleep/sleep.c +1 -1
-static void handle_sched_wakeup(void *data, struct task_struct *task)
+static void handle_sched_exit(void *data, bool is_switch)

 {

kernel/trace/rv/monitors/sleep/sleep.c +1 -1
-	ltl_atom_pulse(task, LTL_WAKE, true);
+	ltl_atom_pulse(current, LTL_SCHEDULE_IN, true);

 }

Well, this triggers also in a spurious schedule: current -> current, can that be a problem? Otherwise you may either use sched_switch over next or pulse only if is_switch.

It probably isn't a big deal though, since there's no SLEEP prior.. But we may want to avoid the noise.

Looks good overall.

Thanks,
Gabriele

maintainer Reviewed-by on v2 2/4
#5 lore raw

Re: [PATCH v2 2/4] rv/rtapp/sleep: Update nanosleep rule

In reply to Nam Cao
Quoted 10 lines · Nam Cao

CLOCK_REALTIME is the only clock that often is misused in real-time applications. The other clocks either are safe for real-time uses (CLOCK_TAI, CLOCK_MONOTONIC, CLOCK_BOOTTIME) or are unlikely to be misused (CLOCK_AUX, CLOCK_PROCESS_CPUTIME_ID).

Update the monitor to only warn about CLOCK_REALTIME.

While at it, update the out-of-sync documentation.

Looks good, thanks.

Quoted 235 lines

---
 Documentation/trace/rv/monitor_rtapp.rst  | 17 +++++--- kernel/trace/rv/monitors/sleep/sleep.c    | 12 ++---- kernel/trace/rv/monitors/sleep/sleep.h    | 52 +++++++++++------------ tools/verification/models/rtapp/sleep.ltl |  2 +-

 4 files changed, 39 insertions(+), 44 deletions(-)
Documentation/trace/rv/monitor_rtapp.rst +0 -0
index 01656bf7080a..570be67a8f3b 100644
@@ -51,12 +51,13 @@ The `sleep` monitor reports real-time threads sleeping in

a manner that may
 cause undesirable latency. Real-time applications should only put a real-time thread to sleep for one of the following reasons:

-  - Cyclic work: real-time thread sleeps waiting for the next cycle. For this -    case, only the clock_nanosleep syscall should be used with TIMER_ABSTIME
-    (to avoid time drift) and CLOCK_MONOTONIC (to avoid the clock being -    changed). No other method is safe for real-time. For example, threads -    waiting for timerfd can be woken by softirq which provides no real-time -    guarantee.
+  - Cyclic work: real-time thread sleeps waiting for the next
+    cycle. For this case, only the clock_nanosleep syscall should be +    used with TIMER_ABSTIME (to avoid time drift). Additionally, +    CLOCK_REALTIME should not be used (to avoid the clock being +    changed). No other method is safe for real-time. For example, +    threads waiting for timerfd can be woken by softirq which provides +    no real-time guarantee.
   - Real-time thread waiting for something to happen (e.g. another thread releasing shared resources, or a completion signal from another thread). In
     this case, only futexes (FUTEX_LOCK_PI, FUTEX_LOCK_PI2 or one of

Documentation/trace/rv/monitor_rtapp.rst +0 -0
@@ -99,14 +100,16 @@ The monitor's specification is::


   RT_VALID_SLEEP_REASON = FUTEX_WAIT
                        or RT_FRIENDLY_NANOSLEEP +                       or EPOLL_WAIT

   RT_FRIENDLY_NANOSLEEP = CLOCK_NANOSLEEP
                       and NANOSLEEP_TIMER_ABSTIME -                      and NANOSLEEP_CLOCK_MONOTONIC +                      and not NANOSLEEP_CLOCK_REALTIME
   RT_FRIENDLY_WAKE = WOKEN_BY_EQUAL_OR_HIGHER_PRIO or WOKEN_BY_HARDIRQ
                   or WOKEN_BY_NMI
+                  or ABORT_SLEEP
                   or KTHREAD_SHOULD_STOP

   ALLOWLIST = BLOCK_ON_RT_MUTEX

kernel/trace/rv/monitors/sleep/sleep.c +0 -0
index d6b677fab8f8..638be7d8747f 100644
@@ -44,8 +44,7 @@ static void ltl_atoms_init(struct task_struct *task, struct

ltl_monitor *mon, bo

  if (task_creation) {
  ltl_atom_set(mon, LTL_KTHREAD_SHOULD_STOP, false);

kernel/trace/rv/monitors/sleep/sleep.c +1 -2
-		ltl_atom_set(mon, LTL_NANOSLEEP_CLOCK_MONOTONIC, false);
-		ltl_atom_set(mon, LTL_NANOSLEEP_CLOCK_TAI, false);
+		ltl_atom_set(mon, LTL_NANOSLEEP_CLOCK_REALTIME, false);

  ltl_atom_set(mon, LTL_NANOSLEEP_TIMER_ABSTIME, false); ltl_atom_set(mon, LTL_CLOCK_NANOSLEEP, false); ltl_atom_set(mon, LTL_FUTEX_WAIT, false);

kernel/trace/rv/monitors/sleep/sleep.c +0 -0
@@ -60,8 +59,7 @@ static void ltl_atoms_init(struct task_struct *task, struct

ltl_monitor *mon, bo
  /* kernel tasks do not do syscall */ ltl_atom_set(mon, LTL_FUTEX_WAIT, false); ltl_atom_set(mon, LTL_FUTEX_LOCK_PI, false);

kernel/trace/rv/monitors/sleep/sleep.c +1 -2
-		ltl_atom_set(mon, LTL_NANOSLEEP_CLOCK_MONOTONIC, false);
-		ltl_atom_set(mon, LTL_NANOSLEEP_CLOCK_TAI, false);
+		ltl_atom_set(mon, LTL_NANOSLEEP_CLOCK_REALTIME, false);

  ltl_atom_set(mon, LTL_NANOSLEEP_TIMER_ABSTIME, false); ltl_atom_set(mon, LTL_CLOCK_NANOSLEEP, false); ltl_atom_set(mon, LTL_EPOLL_WAIT, false);

kernel/trace/rv/monitors/sleep/sleep.c +0 -0
@@ -136,8 +134,7 @@ static void handle_sys_enter(void *data, struct pt_regs
 *regs, long id)

  case __NR_clock_nanosleep_time64:
 #endif
  syscall_get_arguments(current, regs, args);

kernel/trace/rv/monitors/sleep/sleep.c +0 -1
-		ltl_atom_set(mon, LTL_NANOSLEEP_CLOCK_MONOTONIC, args[0] ==

CLOCK_MONOTONIC);

kernel/trace/rv/monitors/sleep/sleep.c +0 -1
-		ltl_atom_set(mon, LTL_NANOSLEEP_CLOCK_TAI, args[0] ==

CLOCK_TAI);

kernel/trace/rv/monitors/sleep/sleep.c +1 -0
+		ltl_atom_set(mon, LTL_NANOSLEEP_CLOCK_REALTIME, args[0] ==

CLOCK_REALTIME);
  ltl_atom_set(mon, LTL_NANOSLEEP_TIMER_ABSTIME, args[1] == TIMER_ABSTIME);
  ltl_atom_update(current, LTL_CLOCK_NANOSLEEP, true); break;

kernel/trace/rv/monitors/sleep/sleep.c +0 -0
@@ -178,8 +175,7 @@ static void handle_sys_exit(void *data, struct pt_regs
 *regs, long ret)


  ltl_atom_set(mon, LTL_FUTEX_LOCK_PI, false); ltl_atom_set(mon, LTL_FUTEX_WAIT, false);

kernel/trace/rv/monitors/sleep/sleep.c +1 -2
-	ltl_atom_set(mon, LTL_NANOSLEEP_CLOCK_MONOTONIC, false);
-	ltl_atom_set(mon, LTL_NANOSLEEP_CLOCK_TAI, false);
+	ltl_atom_set(mon, LTL_NANOSLEEP_CLOCK_REALTIME, false);

  ltl_atom_set(mon, LTL_NANOSLEEP_TIMER_ABSTIME, false); ltl_atom_set(mon, LTL_EPOLL_WAIT, false); ltl_atom_update(current, LTL_CLOCK_NANOSLEEP, false);

kernel/trace/rv/monitors/sleep/sleep.h +2 -6
index 403dc2852c52..2fe2ec7edae8 100644
@@ -20,8 +20,7 @@ enum ltl_atom {
  	LTL_FUTEX_WAIT,
  	LTL_KERNEL_THREAD,
  	LTL_KTHREAD_SHOULD_STOP,
-	LTL_NANOSLEEP_CLOCK_MONOTONIC,
-	LTL_NANOSLEEP_CLOCK_TAI,
+	LTL_NANOSLEEP_CLOCK_REALTIME,
  	LTL_NANOSLEEP_TIMER_ABSTIME,
  	LTL_RT,
  	LTL_SCHEDULE_IN,
@@ -46,8 +45,7 @@ static const char *ltl_atom_str(enum ltl_atom atom)
  		"fu_wa",
  		"ker_th",
  		"kth_sh_st",
-		"na_cl_mo",
-		"na_cl_ta",
+		"na_cl_re",
  		"na_ti_ab",
  		"rt",
  		"sch_in",
@@ -87,8 +85,7 @@ static void ltl_start(struct task_struct *task, struct
 ltl_monitor *mon)
  	bool schedule_in = test_bit(LTL_SCHEDULE_IN, mon->atoms);
  	bool rt = test_bit(LTL_RT, mon->atoms);
  	bool nanosleep_timer_abstime = test_bit(LTL_NANOSLEEP_TIMER_ABSTIME,
 mon->atoms);
-	bool nanosleep_clock_tai = test_bit(LTL_NANOSLEEP_CLOCK_TAI, mon-
 >atoms);
-	bool nanosleep_clock_monotonic =

test_bit(LTL_NANOSLEEP_CLOCK_MONOTONIC, mon->atoms);

kernel/trace/rv/monitors/sleep/sleep.h +1 -0
+	bool nanosleep_clock_realtime =

test_bit(LTL_NANOSLEEP_CLOCK_REALTIME, mon->atoms);
  bool kthread_should_stop = test_bit(LTL_KTHREAD_SHOULD_STOP, mon-

Quoted 1 line

atoms);

  bool kernel_thread = test_bit(LTL_KERNEL_THREAD, mon->atoms); bool futex_wait = test_bit(LTL_FUTEX_WAIT, mon->atoms);

kernel/trace/rv/monitors/sleep/sleep.h +10 -10
@@ -97,17 +94,17 @@ static void ltl_start(struct task_struct *task, struct
 ltl_monitor *mon)
  	bool clock_nanosleep = test_bit(LTL_CLOCK_NANOSLEEP, mon->atoms);
  	bool block_on_rt_mutex = test_bit(LTL_BLOCK_ON_RT_MUTEX, mon->atoms);
  	bool abort_sleep = test_bit(LTL_ABORT_SLEEP, mon->atoms);
-	bool val42 = task_is_rcu || task_is_migration;
-	bool val43 = futex_lock_pi || val42;
-	bool val5 = block_on_rt_mutex || val43;
-	bool val34 = abort_sleep || kthread_should_stop;
-	bool val35 = woken_by_nmi || val34;
-	bool val36 = woken_by_hardirq || val35;
-	bool val14 = woken_by_equal_or_higher_prio || val36;
+	bool val41 = task_is_rcu || task_is_migration;
+	bool val42 = futex_lock_pi || val41;
+	bool val5 = block_on_rt_mutex || val42;
+	bool val33 = abort_sleep || kthread_should_stop;
+	bool val34 = woken_by_nmi || val33;
+	bool val35 = woken_by_hardirq || val34;
+	bool val14 = woken_by_equal_or_higher_prio || val35;
  	bool val13 = !schedule_in;
-	bool val26 = nanosleep_clock_monotonic || nanosleep_clock_tai;
-	bool val27 = nanosleep_timer_abstime && val26;
-	bool val18 = clock_nanosleep && val27;
+	bool val25 = !nanosleep_clock_realtime;
+	bool val26 = nanosleep_timer_abstime && val25;
+	bool val18 = clock_nanosleep && val26;
  	bool val20 = val18 || epoll_wait;
  	bool val9 = futex_wait || val20;

  bool val11 = val9 || kernel_thread;

kernel/trace/rv/monitors/sleep/sleep.h +0 -0
@@ -138,8 +135,7 @@ ltl_possible_next_states(struct ltl_monitor *mon, unsigned

int state, unsigned l
  bool schedule_in = test_bit(LTL_SCHEDULE_IN, mon->atoms); bool rt = test_bit(LTL_RT, mon->atoms);
  bool nanosleep_timer_abstime = test_bit(LTL_NANOSLEEP_TIMER_ABSTIME, mon->atoms);

kernel/trace/rv/monitors/sleep/sleep.h +0 -1
-	bool nanosleep_clock_tai = test_bit(LTL_NANOSLEEP_CLOCK_TAI, mon-
Quoted 1 line

atoms);

kernel/trace/rv/monitors/sleep/sleep.h +0 -1
-	bool nanosleep_clock_monotonic =

test_bit(LTL_NANOSLEEP_CLOCK_MONOTONIC, mon->atoms);

kernel/trace/rv/monitors/sleep/sleep.h +1 -0
+	bool nanosleep_clock_realtime =

test_bit(LTL_NANOSLEEP_CLOCK_REALTIME, mon->atoms);
  bool kthread_should_stop = test_bit(LTL_KTHREAD_SHOULD_STOP, mon-

Quoted 1 line

atoms);

  bool kernel_thread = test_bit(LTL_KERNEL_THREAD, mon->atoms); bool futex_wait = test_bit(LTL_FUTEX_WAIT, mon->atoms);

kernel/trace/rv/monitors/sleep/sleep.h +0 -0
@@ -148,17 +144,17 @@ ltl_possible_next_states(struct ltl_monitor *mon,

unsigned int state, unsigned l
  bool clock_nanosleep = test_bit(LTL_CLOCK_NANOSLEEP, mon->atoms); bool block_on_rt_mutex = test_bit(LTL_BLOCK_ON_RT_MUTEX, mon->atoms); bool abort_sleep = test_bit(LTL_ABORT_SLEEP, mon->atoms);

kernel/trace/rv/monitors/sleep/sleep.h +7 -7
-	bool val42 = task_is_rcu || task_is_migration;
-	bool val43 = futex_lock_pi || val42;
-	bool val5 = block_on_rt_mutex || val43;
-	bool val34 = abort_sleep || kthread_should_stop;
-	bool val35 = woken_by_nmi || val34;
-	bool val36 = woken_by_hardirq || val35;
-	bool val14 = woken_by_equal_or_higher_prio || val36;
+	bool val41 = task_is_rcu || task_is_migration;
+	bool val42 = futex_lock_pi || val41;
+	bool val5 = block_on_rt_mutex || val42;
+	bool val33 = abort_sleep || kthread_should_stop;
+	bool val34 = woken_by_nmi || val33;
+	bool val35 = woken_by_hardirq || val34;
+	bool val14 = woken_by_equal_or_higher_prio || val35;

  bool val13 = !schedule_in;

kernel/trace/rv/monitors/sleep/sleep.h +3 -3
-	bool val26 = nanosleep_clock_monotonic || nanosleep_clock_tai;
-	bool val27 = nanosleep_timer_abstime && val26;
-	bool val18 = clock_nanosleep && val27;
+	bool val25 = !nanosleep_clock_realtime;
+	bool val26 = nanosleep_timer_abstime && val25;
+	bool val18 = clock_nanosleep && val26;

  bool val20 = val18 || epoll_wait;
  bool val9 = futex_wait || val20;
  bool val11 = val9 || kernel_thread;

tools/verification/models/rtapp/sleep.ltl +0 -0
index 464c84b9df87..5923e58d7810 100644
@@ -9,7 +9,7 @@ RT_VALID_SLEEP_REASON = FUTEX_WAIT


 RT_FRIENDLY_NANOSLEEP = CLOCK_NANOSLEEP
                     and NANOSLEEP_TIMER_ABSTIME
-                    and (NANOSLEEP_CLOCK_MONOTONIC or NANOSLEEP_CLOCK_TAI) +                    and not NANOSLEEP_CLOCK_REALTIME

 RT_FRIENDLY_WAKE = WOKEN_BY_EQUAL_OR_HIGHER_PRIO
                 or WOKEN_BY_HARDIRQ

maintainer
#3 lore raw

Re: [PATCH v2 1/4] rv/rtapp/sleep: Make the error more informative for user

In reply to Gabriele Monaco
Quoted 10 lines · Gabriele Monaco
Quoted 6 lines · Nam Cao
kernel/trace/rv/monitors/sleep/sleep.c +1 -1
-static void handle_sched_wakeup(void *data, struct task_struct *task)
+static void handle_sched_exit(void *data, bool is_switch)

 {

kernel/trace/rv/monitors/sleep/sleep.c +1 -1
-	ltl_atom_pulse(task, LTL_WAKE, true);
+	ltl_atom_pulse(current, LTL_SCHEDULE_IN, true);

 }

Well, this triggers also in a spurious schedule: current -> current, can that be a problem? Otherwise you may either use sched_switch over next

sched_switch would obviously be in the wrong context.. Ignore that.

Quoted 11 lines

or pulse only if is_switch.

It probably isn't a big deal though, since there's no SLEEP prior.. But we may want to avoid the noise.

Looks good overall.

Thanks,
Gabriele

maintainer Reviewed-by on v2 3/4
#7 lore raw

Re: [PATCH v2 3/4] rv/rtapp/sleep: Stop monitoring kernel threads

In reply to Nam Cao
Quoted 17 lines · Nam Cao

The rtapp/sleep monitor's primary purpose is detecting common mistakes with user-space real-time design. Monitoring real-time issues with kernel threads is a bonus.

However, accomodating kernel threads complicates the monitor due to the edge cases which is seen by the monitor as lower-priority task waking higher-priority task:

  - kthread_stop() wakes up the task in order to stop it.

  - The rcu thread and migration thread can be woken by any task.

  - The ktimerd thread is woken near the end of irq_exit_rcu(), where the preempt counter is "broken" and falsely says this is task context. This requires the monitor to use the hardirq_context flag instead of the preempt counter.

Quoted 4 lines

Beside complicating the monitor, the final case also requires enabling CONFIG_TRACE_IRQFLAGS (so that "hardirq_context" can be used). This adds overhead to the kernel even when the monitor is not active. This may be an obstacle to enabling this monitor in distros' kernels.

Very good especially for this!

Quoted 9 lines

Furthermore, kernel threads usually are started before the monitor is enabled. Consequently, the threads' states (i.o.w. the monitor's atomic propositions for the threads) are not fully known to the monitor. As a result, the kernel threads mostly cannot be monitored.

Overall, the downsides of accomodating kernel threads outweights the benefits. Thus, exclude kernel threads to simplify the monitor.

Thanks,
Gabriele

Quoted 466 lines

---
Cc: Sebastian Andrzej Siewior <[email protected]>
---
 Documentation/trace/rv/monitor_rtapp.rst  |  22 ++---

 kernel/trace/rv/monitors/sleep/Kconfig    |   1 -
 kernel/trace/rv/monitors/sleep/sleep.c    |  39 +-------
 kernel/trace/rv/monitors/sleep/sleep.h    | 104 +++++++++-------------
 tools/verification/models/rtapp/sleep.ltl |   7 +-
 5 files changed, 54 insertions(+), 119 deletions(-)
Documentation/trace/rv/monitor_rtapp.rst +0 -0
index 570be67a8f3b..502d3ea412eb 100644
@@ -93,9 +93,9 @@ assessment.


 The monitor's specification is::

-  RULE = always ((RT and SLEEP) imply (RT_FRIENDLY_SLEEP or ALLOWLIST)) +  RULE = always ((RT and SLEEP and USER_THREAD) imply (RT_FRIENDLY_SLEEP or ALLOWLIST))

-  RT_FRIENDLY_SLEEP = (RT_VALID_SLEEP_REASON or KERNEL_THREAD) +  RT_FRIENDLY_SLEEP = RT_VALID_SLEEP_REASON
                   and ((not SCHEDULE_IN) until RT_FRIENDLY_WAKE)
   RT_VALID_SLEEP_REASON = FUTEX_WAIT

Documentation/trace/rv/monitor_rtapp.rst +0 -0
@@ -110,23 +110,13 @@ The monitor's specification is::

                   or WOKEN_BY_HARDIRQ
                   or WOKEN_BY_NMI
                   or ABORT_SLEEP
-                  or KTHREAD_SHOULD_STOP

   ALLOWLIST = BLOCK_ON_RT_MUTEX
            or FUTEX_LOCK_PI
-           or TASK_IS_RCU
-           or TASK_IS_MIGRATION

Documentation/trace/rv/monitor_rtapp.rst +0 -5
-
-Beside the scenarios described above, this specification also handle some
-special cases:
-
-  - `KERNEL_THREAD`: kernel tasks do not have any pattern that can be

recognized
-    as valid real-time sleeping reasons. Therefore sleeping reason is not -    checked for kernel tasks.
-  - KTHREAD_SHOULD_STOP: a non-real-time thread may stop a real-time kernel -    thread by waking it and waiting for it to exit (kthread_stop()). This -    wakeup is safe for real-time.
-  - ALLOWLIST: to handle known false positives with the kernel.

Documentation/trace/rv/monitor_rtapp.rst +2 -0
+
+Beside the scenarios described above, this specification also defines an

allow list

Documentation/trace/rv/monitor_rtapp.rst +2 -0
+to handle some special cases:
+

   - BLOCK_ON_RT_MUTEX is included in the allowlist due to its implementation.
     In the release path of rt_mutex, a boosted task is de-boosted before waking
     the rt_mutex's waiter. Consequently, the monitor may see a real-time- unsafe

kernel/trace/rv/monitors/sleep/Kconfig +0 -0
index 6b7a122e7b47..d6ec3e9a91b6 100644
@@ -5,7 +5,6 @@ config RV_MON_SLEEP

  select RV_LTL_MONITOR
  depends on HAVE_SYSCALL_TRACEPOINTS
  depends on RV_MON_RTAPP

kernel/trace/rv/monitors/sleep/Kconfig +0 -1
-	select TRACE_IRQFLAGS

  default y
  select LTL_MON_EVENTS_ID
  bool "sleep monitor"

kernel/trace/rv/monitors/sleep/sleep.c +0 -0
index 638be7d8747f..aa5a984853b5 100644
@@ -43,7 +43,6 @@ static void ltl_atoms_init(struct task_struct *task, struct

ltl_monitor *mon, bo
  ltl_atom_set(mon, LTL_WOKEN_BY_EQUAL_OR_HIGHER_PRIO, false);
  if (task_creation) {

kernel/trace/rv/monitors/sleep/sleep.c +0 -1
-		ltl_atom_set(mon, LTL_KTHREAD_SHOULD_STOP, false);

  ltl_atom_set(mon, LTL_NANOSLEEP_CLOCK_REALTIME, false); ltl_atom_set(mon, LTL_NANOSLEEP_TIMER_ABSTIME, false); ltl_atom_set(mon, LTL_CLOCK_NANOSLEEP, false);

kernel/trace/rv/monitors/sleep/sleep.c +0 -0
@@ -53,33 +52,7 @@ static void ltl_atoms_init(struct task_struct *task, struct

ltl_monitor *mon, bo
  ltl_atom_set(mon, LTL_BLOCK_ON_RT_MUTEX, false); }

kernel/trace/rv/monitors/sleep/sleep.c +1 -27
-	if (task->flags & PF_KTHREAD) {
-		ltl_atom_set(mon, LTL_KERNEL_THREAD, true);
-
-		/* kernel tasks do not do syscall */
-		ltl_atom_set(mon, LTL_FUTEX_WAIT, false);
-		ltl_atom_set(mon, LTL_FUTEX_LOCK_PI, false);
-		ltl_atom_set(mon, LTL_NANOSLEEP_CLOCK_REALTIME, false);
-		ltl_atom_set(mon, LTL_NANOSLEEP_TIMER_ABSTIME, false);
-		ltl_atom_set(mon, LTL_CLOCK_NANOSLEEP, false);
-		ltl_atom_set(mon, LTL_EPOLL_WAIT, false);
-
-		if (strstarts(task->comm, "migration/"))
-			ltl_atom_set(mon, LTL_TASK_IS_MIGRATION, true);
-		else
-			ltl_atom_set(mon, LTL_TASK_IS_MIGRATION, false);
-
-		if (strstarts(task->comm, "rcu"))
-			ltl_atom_set(mon, LTL_TASK_IS_RCU, true);
-		else
-			ltl_atom_set(mon, LTL_TASK_IS_RCU, false);
-	} else {
-		ltl_atom_set(mon, LTL_KTHREAD_SHOULD_STOP, false);
-		ltl_atom_set(mon, LTL_KERNEL_THREAD, false);
-		ltl_atom_set(mon, LTL_TASK_IS_RCU, false);
-		ltl_atom_set(mon, LTL_TASK_IS_MIGRATION, false);
-	}
-
+	ltl_atom_set(mon, LTL_USER_THREAD, !(task->flags & PF_KTHREAD));

 }

 static void handle_sched_set_state(void *data, struct task_struct *task, int state)

kernel/trace/rv/monitors/sleep/sleep.c +0 -0
@@ -97,7 +70,7 @@ static void handle_sched_exit(void *data, bool is_switch)


 static void handle_sched_waking(void *data, struct task_struct *task) {

kernel/trace/rv/monitors/sleep/sleep.c +1 -1
-	if (this_cpu_read(hardirq_context)) {
+	if (in_hardirq()) {

  ltl_atom_pulse(task, LTL_WOKEN_BY_HARDIRQ, true); } else if (in_task()) {
  if (current->prio <= task->prio)

kernel/trace/rv/monitors/sleep/sleep.c +0 -0
@@ -181,12 +154,6 @@ static void handle_sys_exit(void *data, struct pt_regs
 *regs, long ret)
  	ltl_atom_update(current, LTL_CLOCK_NANOSLEEP, false);
  }

kernel/trace/rv/monitors/sleep/sleep.c +0 -6
-static void handle_kthread_stop(void *data, struct task_struct *task)
-{
-	/* FIXME: this could race with other tracepoint handlers */
-	ltl_atom_update(task, LTL_KTHREAD_SHOULD_STOP, true);
-}
-

 static int enable_sleep(void)
 {
  int retval;

kernel/trace/rv/monitors/sleep/sleep.c +0 -1
@@ -200,7 +167,6 @@ static int enable_sleep(void)
  	rv_attach_trace_probe("rtapp_sleep", sched_set_state_tp,
 handle_sched_set_state);
  	rv_attach_trace_probe("rtapp_sleep", contention_begin,
 handle_contention_begin);
  	rv_attach_trace_probe("rtapp_sleep", contention_end,
 handle_contention_end);
-	rv_attach_trace_probe("rtapp_sleep", sched_kthread_stop,

handle_kthread_stop);
  rv_attach_trace_probe("rtapp_sleep", sys_enter, handle_sys_enter); rv_attach_trace_probe("rtapp_sleep", sys_exit, handle_sys_exit); return 0;

kernel/trace/rv/monitors/sleep/sleep.c +0 -1
@@ -213,7 +179,6 @@ static void disable_sleep(void)
  	rv_detach_trace_probe("rtapp_sleep", sched_set_state_tp,
 handle_sched_set_state);
  	rv_detach_trace_probe("rtapp_sleep", contention_begin,
 handle_contention_begin);
  	rv_detach_trace_probe("rtapp_sleep", contention_end,
 handle_contention_end);
-	rv_detach_trace_probe("rtapp_sleep", sched_kthread_stop,

handle_kthread_stop);
  rv_detach_trace_probe("rtapp_sleep", sys_enter, handle_sys_enter); rv_detach_trace_probe("rtapp_sleep", sys_exit, handle_sys_exit);

kernel/trace/rv/monitors/sleep/sleep.h +2 -8
index 2fe2ec7edae8..44e593f41e6a 100644
@@ -18,15 +18,12 @@ enum ltl_atom {
  	LTL_EPOLL_WAIT,
  	LTL_FUTEX_LOCK_PI,
  	LTL_FUTEX_WAIT,
-	LTL_KERNEL_THREAD,
-	LTL_KTHREAD_SHOULD_STOP,
  	LTL_NANOSLEEP_CLOCK_REALTIME,
  	LTL_NANOSLEEP_TIMER_ABSTIME,
  	LTL_RT,
  	LTL_SCHEDULE_IN,
  	LTL_SLEEP,
-	LTL_TASK_IS_MIGRATION,
-	LTL_TASK_IS_RCU,
+	LTL_USER_THREAD,
  	LTL_WOKEN_BY_EQUAL_OR_HIGHER_PRIO,
  	LTL_WOKEN_BY_HARDIRQ,
  	LTL_WOKEN_BY_NMI,
@@ -43,15 +40,12 @@ static const char *ltl_atom_str(enum ltl_atom atom)
  		"ep_wa",
  		"fu_lo_pi",
  		"fu_wa",
-		"ker_th",
-		"kth_sh_st",
  		"na_cl_re",
  		"na_ti_ab",
  		"rt",
  		"sch_in",
  		"sle",
-		"ta_mi",
-		"ta_rc",
+		"us_th",
  		"wo_eq_hi_pr",
  		"wo_ha",
  		"wo_nm",
@@ -79,46 +73,41 @@ static void ltl_start(struct task_struct *task, struct
 ltl_monitor *mon)
  	bool woken_by_hardirq = test_bit(LTL_WOKEN_BY_HARDIRQ, mon->atoms);

  bool woken_by_equal_or_higher_prio =
test_bit(LTL_WOKEN_BY_EQUAL_OR_HIGHER_PRIO,
       mon->atoms);

kernel/trace/rv/monitors/sleep/sleep.h +1 -2
-	bool task_is_rcu = test_bit(LTL_TASK_IS_RCU, mon->atoms);
-	bool task_is_migration = test_bit(LTL_TASK_IS_MIGRATION, mon->atoms);
+	bool user_thread = test_bit(LTL_USER_THREAD, mon->atoms);

  bool sleep = test_bit(LTL_SLEEP, mon->atoms);
  bool schedule_in = test_bit(LTL_SCHEDULE_IN, mon->atoms); bool rt = test_bit(LTL_RT, mon->atoms);
  bool nanosleep_timer_abstime = test_bit(LTL_NANOSLEEP_TIMER_ABSTIME, mon->atoms);
  bool nanosleep_clock_realtime = test_bit(LTL_NANOSLEEP_CLOCK_REALTIME, mon->atoms);

kernel/trace/rv/monitors/sleep/sleep.h +0 -1
-	bool kthread_should_stop = test_bit(LTL_KTHREAD_SHOULD_STOP, mon-
Quoted 1 line

atoms);

kernel/trace/rv/monitors/sleep/sleep.h +0 -1
-	bool kernel_thread = test_bit(LTL_KERNEL_THREAD, mon->atoms);

  bool futex_wait = test_bit(LTL_FUTEX_WAIT, mon->atoms); bool futex_lock_pi = test_bit(LTL_FUTEX_LOCK_PI, mon->atoms); bool epoll_wait = test_bit(LTL_EPOLL_WAIT, mon->atoms); bool clock_nanosleep = test_bit(LTL_CLOCK_NANOSLEEP, mon->atoms); bool block_on_rt_mutex = test_bit(LTL_BLOCK_ON_RT_MUTEX, mon->atoms); bool abort_sleep = test_bit(LTL_ABORT_SLEEP, mon->atoms);

kernel/trace/rv/monitors/sleep/sleep.h +4 -7
-	bool val41 = task_is_rcu || task_is_migration;
-	bool val42 = futex_lock_pi || val41;
-	bool val5 = block_on_rt_mutex || val42;
-	bool val33 = abort_sleep || kthread_should_stop;
-	bool val34 = woken_by_nmi || val33;
-	bool val35 = woken_by_hardirq || val34;
-	bool val14 = woken_by_equal_or_higher_prio || val35;
+	bool val7 = block_on_rt_mutex || futex_lock_pi;
+	bool val32 = woken_by_nmi || abort_sleep;
+	bool val33 = woken_by_hardirq || val32;
+	bool val14 = woken_by_equal_or_higher_prio || val33;

  bool val13 = !schedule_in;
  bool val25 = !nanosleep_clock_realtime; bool val26 = nanosleep_timer_abstime && val25; bool val18 = clock_nanosleep && val26; bool val20 = val18 || epoll_wait;

kernel/trace/rv/monitors/sleep/sleep.h +2 -2
-	bool val9 = futex_wait || val20;
-	bool val11 = val9 || kernel_thread;
+	bool val11 = futex_wait || val20;
+	bool val3 = !user_thread;

  bool val2 = !sleep;

kernel/trace/rv/monitors/sleep/sleep.h +1 -0
+	bool val4 = val2 || val3;

  bool val1 = !rt;

kernel/trace/rv/monitors/sleep/sleep.h +1 -1
-	bool val3 = val1 || val2;
+	bool val5 = val1 || val4;

kernel/trace/rv/monitors/sleep/sleep.h +1 -1
-	if (val3)
+	if (val5)

  __set_bit(S0, mon->states);
  if (val11 && val13)
  __set_bit(S1, mon->states);
  if (val11 && val14)
  __set_bit(S4, mon->states);

kernel/trace/rv/monitors/sleep/sleep.h +1 -1
-	if (val5)
+	if (val7)

  __set_bit(S5, mon->states);
 }

kernel/trace/rv/monitors/sleep/sleep.h +0 -0
@@ -129,130 +118,125 @@ ltl_possible_next_states(struct ltl_monitor *mon,

unsigned int state, unsigned l
  bool woken_by_hardirq = test_bit(LTL_WOKEN_BY_HARDIRQ, mon->atoms); bool woken_by_equal_or_higher_prio = test_bit(LTL_WOKEN_BY_EQUAL_OR_HIGHER_PRIO,
       mon->atoms);

kernel/trace/rv/monitors/sleep/sleep.h +1 -2
-	bool task_is_rcu = test_bit(LTL_TASK_IS_RCU, mon->atoms);
-	bool task_is_migration = test_bit(LTL_TASK_IS_MIGRATION, mon->atoms);
+	bool user_thread = test_bit(LTL_USER_THREAD, mon->atoms);

  bool sleep = test_bit(LTL_SLEEP, mon->atoms);
  bool schedule_in = test_bit(LTL_SCHEDULE_IN, mon->atoms); bool rt = test_bit(LTL_RT, mon->atoms);
  bool nanosleep_timer_abstime = test_bit(LTL_NANOSLEEP_TIMER_ABSTIME, mon->atoms);
  bool nanosleep_clock_realtime = test_bit(LTL_NANOSLEEP_CLOCK_REALTIME, mon->atoms);

kernel/trace/rv/monitors/sleep/sleep.h +0 -1
-	bool kthread_should_stop = test_bit(LTL_KTHREAD_SHOULD_STOP, mon-
Quoted 1 line

atoms);

kernel/trace/rv/monitors/sleep/sleep.h +0 -1
-	bool kernel_thread = test_bit(LTL_KERNEL_THREAD, mon->atoms);

  bool futex_wait = test_bit(LTL_FUTEX_WAIT, mon->atoms); bool futex_lock_pi = test_bit(LTL_FUTEX_LOCK_PI, mon->atoms); bool epoll_wait = test_bit(LTL_EPOLL_WAIT, mon->atoms); bool clock_nanosleep = test_bit(LTL_CLOCK_NANOSLEEP, mon->atoms); bool block_on_rt_mutex = test_bit(LTL_BLOCK_ON_RT_MUTEX, mon->atoms); bool abort_sleep = test_bit(LTL_ABORT_SLEEP, mon->atoms);

kernel/trace/rv/monitors/sleep/sleep.h +4 -7
-	bool val41 = task_is_rcu || task_is_migration;
-	bool val42 = futex_lock_pi || val41;
-	bool val5 = block_on_rt_mutex || val42;
-	bool val33 = abort_sleep || kthread_should_stop;
-	bool val34 = woken_by_nmi || val33;
-	bool val35 = woken_by_hardirq || val34;
-	bool val14 = woken_by_equal_or_higher_prio || val35;
+	bool val7 = block_on_rt_mutex || futex_lock_pi;
+	bool val32 = woken_by_nmi || abort_sleep;
+	bool val33 = woken_by_hardirq || val32;
+	bool val14 = woken_by_equal_or_higher_prio || val33;

  bool val13 = !schedule_in;
  bool val25 = !nanosleep_clock_realtime; bool val26 = nanosleep_timer_abstime && val25; bool val18 = clock_nanosleep && val26; bool val20 = val18 || epoll_wait;

kernel/trace/rv/monitors/sleep/sleep.h +2 -2
-	bool val9 = futex_wait || val20;
-	bool val11 = val9 || kernel_thread;
+	bool val11 = futex_wait || val20;
+	bool val3 = !user_thread;

  bool val2 = !sleep;

kernel/trace/rv/monitors/sleep/sleep.h +1 -0
+	bool val4 = val2 || val3;

  bool val1 = !rt;

kernel/trace/rv/monitors/sleep/sleep.h +1 -1
-	bool val3 = val1 || val2;
+	bool val5 = val1 || val4;


  switch (state) {
  case S0:

kernel/trace/rv/monitors/sleep/sleep.h +1 -1
-		if (val3)
+		if (val5)

  __set_bit(S0, next);
  if (val11 && val13)
  __set_bit(S1, next);
  if (val11 && val14)
  __set_bit(S4, next);

kernel/trace/rv/monitors/sleep/sleep.h +1 -1
-		if (val5)
+		if (val7)

  __set_bit(S5, next);
  break;
  case S1:
  if (val11 && val13)
  __set_bit(S1, next);

kernel/trace/rv/monitors/sleep/sleep.h +1 -1
-		if (val13 && val3)
+		if (val13 && val5)

  __set_bit(S2, next);

kernel/trace/rv/monitors/sleep/sleep.h +1 -1
-		if (val14 && val3)
+		if (val14 && val5)

  __set_bit(S3, next);
  if (val11 && val14)
  __set_bit(S4, next);

kernel/trace/rv/monitors/sleep/sleep.h +1 -1
-		if (val13 && val5)
+		if (val13 && val7)

  __set_bit(S6, next);

kernel/trace/rv/monitors/sleep/sleep.h +1 -1
-		if (val14 && val5)
+		if (val14 && val7)

  __set_bit(S7, next);
  break;
  case S2:
  if (val11 && val13)
  __set_bit(S1, next);

kernel/trace/rv/monitors/sleep/sleep.h +1 -1
-		if (val13 && val3)
+		if (val13 && val5)

  __set_bit(S2, next);

kernel/trace/rv/monitors/sleep/sleep.h +1 -1
-		if (val14 && val3)
+		if (val14 && val5)

  __set_bit(S3, next);
  if (val11 && val14)
  __set_bit(S4, next);

kernel/trace/rv/monitors/sleep/sleep.h +1 -1
-		if (val13 && val5)
+		if (val13 && val7)

  __set_bit(S6, next);

kernel/trace/rv/monitors/sleep/sleep.h +1 -1
-		if (val14 && val5)
+		if (val14 && val7)

  __set_bit(S7, next);
  break;
  case S3:

kernel/trace/rv/monitors/sleep/sleep.h +1 -1
-		if (val3)
+		if (val5)

  __set_bit(S0, next);
  if (val11 && val13)
  __set_bit(S1, next);
  if (val11 && val14)
  __set_bit(S4, next);

kernel/trace/rv/monitors/sleep/sleep.h +1 -1
-		if (val5)
+		if (val7)

  __set_bit(S5, next);
  break;
  case S4:

kernel/trace/rv/monitors/sleep/sleep.h +1 -1
-		if (val3)
+		if (val5)

  __set_bit(S0, next);
  if (val11 && val13)
  __set_bit(S1, next);
  if (val11 && val14)
  __set_bit(S4, next);

kernel/trace/rv/monitors/sleep/sleep.h +1 -1
-		if (val5)
+		if (val7)

  __set_bit(S5, next);
  break;
  case S5:

kernel/trace/rv/monitors/sleep/sleep.h +1 -1
-		if (val3)
+		if (val5)

  __set_bit(S0, next);
  if (val11 && val13)
  __set_bit(S1, next);
  if (val11 && val14)
  __set_bit(S4, next);

kernel/trace/rv/monitors/sleep/sleep.h +1 -1
-		if (val5)
+		if (val7)

  __set_bit(S5, next);
  break;
  case S6:
  if (val11 && val13)
  __set_bit(S1, next);

kernel/trace/rv/monitors/sleep/sleep.h +1 -1
-		if (val13 && val3)
+		if (val13 && val5)

  __set_bit(S2, next);

kernel/trace/rv/monitors/sleep/sleep.h +1 -1
-		if (val14 && val3)
+		if (val14 && val5)

  __set_bit(S3, next);
  if (val11 && val14)
  __set_bit(S4, next);

kernel/trace/rv/monitors/sleep/sleep.h +1 -1
-		if (val13 && val5)
+		if (val13 && val7)

  __set_bit(S6, next);

kernel/trace/rv/monitors/sleep/sleep.h +1 -1
-		if (val14 && val5)
+		if (val14 && val7)

  __set_bit(S7, next);
  break;
  case S7:

kernel/trace/rv/monitors/sleep/sleep.h +1 -1
-		if (val3)
+		if (val5)

  __set_bit(S0, next);
  if (val11 && val13)
  __set_bit(S1, next);
  if (val11 && val14)
  __set_bit(S4, next);

kernel/trace/rv/monitors/sleep/sleep.h +1 -1
-		if (val5)
+		if (val7)

  __set_bit(S5, next);
  break;
  }

tools/verification/models/rtapp/sleep.ltl +1 -1
index 5923e58d7810..4d78fdd204c0 100644
@@ -1,6 +1,6 @@
-RULE = always ((RT and SLEEP) imply (RT_FRIENDLY_SLEEP or ALLOWLIST))
+RULE = always ((RT and SLEEP and USER_THREAD) imply (RT_FRIENDLY_SLEEP or
 ALLOWLIST))

tools/verification/models/rtapp/sleep.ltl +1 -1
-RT_FRIENDLY_SLEEP = (RT_VALID_SLEEP_REASON or KERNEL_THREAD)
+RT_FRIENDLY_SLEEP = RT_VALID_SLEEP_REASON

                 and ((not SCHEDULE_IN) until RT_FRIENDLY_WAKE)
 RT_VALID_SLEEP_REASON = FUTEX_WAIT

tools/verification/models/rtapp/sleep.ltl +0 -0
@@ -15,9 +15,6 @@ RT_FRIENDLY_WAKE = WOKEN_BY_EQUAL_OR_HIGHER_PRIO

                 or WOKEN_BY_HARDIRQ
                 or WOKEN_BY_NMI
                 or ABORT_SLEEP
-                or KTHREAD_SHOULD_STOP

 ALLOWLIST = BLOCK_ON_RT_MUTEX
          or FUTEX_LOCK_PI
-         or TASK_IS_RCU
-         or TASK_IS_MIGRATION

maintainer
#9 lore raw

Re: [PATCH v2 4/4] rv/rtapp: Add wakeup monitor

In reply to Nam Cao
Quoted 10 lines · Nam Cao

Add a wakeup monitor to detect a lower-priority task waking up a higher-priority task.

The rtapp/sleep monitor already detects this. However, that monitor triggers an error in the context of the wakee task and user only gets the stacktrace of that task. It is also extremely useful to get the stacktrace of the waker task, which this monitor offers. In other words, this monitor complements the rtapp/sleep monitor.

This looks good, but if I understand it correctly, the same violation should be spotted by both monitors from two different perspectives, but sleep catches more things (e.g. tasks using wrong sleeping ways despite their wakeup):

  # perf stat -a -e rv:error_sleep -e rv:error_wakeup -- stress-ng --cpu 5 --cpu-load 90 --sched rr -t 5

Performance counter stats for 'system wide':

285 rv:error_sleep
20 rv:error_wakeup

Provided I don't really know what's happening down there (I just let the stressor run free), this discrepancy is expected, right?

Thanks,
Gabriele

Quoted 434 lines

---
 Documentation/trace/rv/monitor_rtapp.rst      |  20 +++ kernel/trace/rv/Kconfig                       |   1 + kernel/trace/rv/Makefile                      |   1 + kernel/trace/rv/monitors/rtapp/Kconfig        |   2 +- kernel/trace/rv/monitors/wakeup/Kconfig       |  16 ++ kernel/trace/rv/monitors/wakeup/wakeup.c      | 153 ++++++++++++++++++ kernel/trace/rv/monitors/wakeup/wakeup.h      |  92 +++++++++++ .../trace/rv/monitors/wakeup/wakeup_trace.h   |  14 ++ kernel/trace/rv/rv_trace.h                    |   1 + tools/verification/models/rtapp/wakeup.ltl    |   5 +

 10 files changed, 304 insertions(+), 1 deletion(-)

 create mode 100644 kernel/trace/rv/monitors/wakeup/Kconfig create mode 100644 kernel/trace/rv/monitors/wakeup/wakeup.c create mode 100644 kernel/trace/rv/monitors/wakeup/wakeup.h create mode 100644 kernel/trace/rv/monitors/wakeup/wakeup_trace.h create mode 100644 tools/verification/models/rtapp/wakeup.ltl

Documentation/trace/rv/monitor_rtapp.rst +0 -0
index 502d3ea412eb..238b59395ff5 100644
@@ -124,3 +124,23 @@ to handle some special cases:

     real-time-safe because preemption is disabled for the duration.
   - FUTEX_LOCK_PI is included in the allowlist for the same reason as BLOCK_ON_RT_MUTEX.

Documentation/trace/rv/monitor_rtapp.rst +5 -0
+
+Monitor wakeup
+++++++++++++++
+
+The `wakeup` monitor reports real-time threads being woken by lower-priority

threads,

Documentation/trace/rv/monitor_rtapp.rst +4 -0
+which is a hint of priority inversion. Its specification is::
+
+  RULE = always (((RT and USER_THREAD) imply
+                (not (WOKEN_BY_LOWER_PRIO or WOKEN_BY_SOFTIRQ)) or

ALLOWLIST))

Documentation/trace/rv/monitor_rtapp.rst +5 -0
+
+  ALLOWLIST = BLOCK_ON_RT_MUTEX
+           or FUTEX_LOCK_PI
+
+The `sleep` monitor already reports this type of problem. The difference is

the
+context in which the problem is reported. While the sleep monitor reports the problem
+in the context of the wakee, this wakeup monitor reports the problem in the context of

Documentation/trace/rv/monitor_rtapp.rst +2 -0
+the waker. This monitor complement the `sleep` monitor, giving user better
+understanding of the issue. For instance, to debug a lower-priority task

waking a
+higher-priority task scenario, user can enable both wakeup monitor and sleep

Documentation/trace/rv/monitor_rtapp.rst +1 -0
+monitor to get the stack traces of both tasks.
kernel/trace/rv/Kconfig +0 -0
index 3884b14df375..4d3a14a0bac2 100644
@@ -76,6 +76,7 @@ source "kernel/trace/rv/monitors/opid/Kconfig"

 source "kernel/trace/rv/monitors/rtapp/Kconfig" source "kernel/trace/rv/monitors/pagefault/Kconfig" source "kernel/trace/rv/monitors/sleep/Kconfig" +source "kernel/trace/rv/monitors/wakeup/Kconfig" # Add new rtapp monitors here

 source "kernel/trace/rv/monitors/stall/Kconfig"

kernel/trace/rv/Makefile +0 -0
index 94498da35b37..c2c0e4142eb4 100644
@@ -20,6 +20,7 @@ obj-$(CONFIG_RV_MON_OPID) += monitors/opid/opid.o

 obj-$(CONFIG_RV_MON_STALL) += monitors/stall/stall.o obj-$(CONFIG_RV_MON_DEADLINE) += monitors/deadline/deadline.o obj-$(CONFIG_RV_MON_NOMISS) += monitors/nomiss/nomiss.o +obj-$(CONFIG_RV_MON_WAKEUP) += monitors/wakeup/wakeup.o # Add new monitors here
 obj-$(CONFIG_RV_REACTORS) += rv_reactors.o obj-$(CONFIG_RV_REACT_PRINTK) += reactor_printk.o

kernel/trace/rv/monitors/rtapp/Kconfig +0 -0
index 1ce9370a9ba8..1fcd7a400ded 100644
@@ -1,6 +1,6 @@

 config RV_MON_RTAPP
  depends on RV

kernel/trace/rv/monitors/rtapp/Kconfig +1 -1
-	depends on RV_PER_TASK_MONITORS >= 2
+	depends on RV_PER_TASK_MONITORS >= 3

  bool "rtapp monitor"
  help
    Collection of monitors to check for common problems with real-time

kernel/trace/rv/monitors/wakeup/Kconfig new file +16 -0
new file mode 100644
index 000000000000..ec3a5c06a8c4
@@ -0,0 +1,16 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+config RV_MON_WAKEUP
+	depends on RV
+	depends on RV_MON_RTAPP
+	depends on HAVE_SYSCALL_TRACEPOINTS
+	default y
+	select LTL_MON_EVENTS_ID
+	bool "wakeup monitor"
+	help
+	  This monitor detects a lower-priority task waking up a
+	  higher-priority task. The RV_MON_SLEEP monitor already
+	  detects this case, but this monitor detects in the context
+	  of the waker task instead. This and RV_MON_SLEEP can be
+	  enabled together to get the stacktrace of both the waker
+	  task and the wakee task.
kernel/trace/rv/monitors/wakeup/wakeup.c new file +31 -0
new file mode 100644
index 000000000000..01b47416f24e
@@ -0,0 +1,153 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/ftrace.h>
+#include <linux/tracepoint.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/init.h>
+#include <linux/rv.h>
+#include <rv/instrumentation.h>
+
+#define MODULE_NAME "wakeup"
+
+#include <trace/events/syscalls.h>
+#include <trace/events/sched.h>
+#include <trace/events/lock.h>
+#include <uapi/linux/futex.h>
+
+#include <rv_trace.h>
+#include <monitors/rtapp/rtapp.h>
+
+
+#ifndef __NR_futex
+#define __NR_futex (-__COUNTER__)
+#endif
+#ifndef __NR_futex_time64
+#define __NR_futex_time64 (-__COUNTER__)
+#endif
+
+#include "wakeup.h"
+#include <rv/ltl_monitor.h>
+
+static void ltl_atoms_fetch(struct task_struct *task, struct ltl_monitor

*mon)

kernel/trace/rv/monitors/wakeup/wakeup.c new file +11 -0
+{
+	/*
+	 * This includes "actual" real-time tasks and also PI-boosted
+	 * tasks. A task being PI-boosted means it is blocking an "actual"
+	 * real-task, therefore it should also obey the monitor's rule,
+	 * otherwise the "actual" real-task may be delayed.
+	 */
+	ltl_atom_set(mon, LTL_RT, rt_or_dl_task(task));
+}
+
+static void ltl_atoms_init(struct task_struct *task, struct ltl_monitor *mon,

bool task_creation)

kernel/trace/rv/monitors/wakeup/wakeup.c new file +23 -0
+{
+	ltl_atom_set(mon, LTL_WOKEN_BY_LOWER_PRIO, false);
+	ltl_atom_set(mon, LTL_WOKEN_BY_SOFTIRQ, false);
+
+	if (task_creation) {
+		ltl_atom_set(mon, LTL_BLOCK_ON_RT_MUTEX, false);
+		ltl_atom_set(mon, LTL_FUTEX_LOCK_PI, false);
+	}
+
+	ltl_atom_set(mon, LTL_USER_THREAD, !(task->flags & PF_KTHREAD));
+}
+
+static void handle_sched_waking(void *data, struct task_struct *task)
+{
+	if (in_task()) {
+		if (current->prio > task->prio)
+			ltl_atom_pulse(task, LTL_WOKEN_BY_LOWER_PRIO, true);
+	} else if (in_serving_softirq()) {
+		ltl_atom_pulse(task, LTL_WOKEN_BY_SOFTIRQ, true);
+	}
+}
+
+static void handle_contention_begin(void *data, void *lock, unsigned int

flags)

kernel/trace/rv/monitors/wakeup/wakeup.c new file +46 -0
+{
+	if (flags & LCB_F_RT)
+		ltl_atom_update(current, LTL_BLOCK_ON_RT_MUTEX, true);
+}
+
+static void handle_contention_end(void *data, void *lock, int ret)
+{
+	ltl_atom_update(current, LTL_BLOCK_ON_RT_MUTEX, false);
+}
+
+static void handle_sys_enter(void *data, struct pt_regs *regs, long id)
+{
+	unsigned long args[6];
+	int op, cmd;
+
+	switch (id) {
+	case __NR_futex:
+	case __NR_futex_time64:
+		syscall_get_arguments(current, regs, args);
+		op = args[1];
+		cmd = op & FUTEX_CMD_MASK;
+
+		switch (cmd) {
+		case FUTEX_LOCK_PI:
+		case FUTEX_LOCK_PI2:
+			ltl_atom_update(current, LTL_FUTEX_LOCK_PI, true);
+			break;
+		}
+		break;
+	}
+}
+
+static void handle_sys_exit(void *data, struct pt_regs *regs, long ret)
+{
+	ltl_atom_update(current, LTL_FUTEX_LOCK_PI, false);
+}
+
+static int enable_wakeup(void)
+{
+	int retval;
+
+	retval = ltl_monitor_init();
+	if (retval)
+		return retval;
+
+	rv_attach_trace_probe("rtapp_wakeup", sched_waking,

handle_sched_waking);

kernel/trace/rv/monitors/wakeup/wakeup.c new file +1 -0
+	rv_attach_trace_probe("rtapp_wakeup", contention_begin,

handle_contention_begin);

kernel/trace/rv/monitors/wakeup/wakeup.c new file +1 -0
+	rv_attach_trace_probe("rtapp_wakeup", contention_end,

handle_contention_end);

kernel/trace/rv/monitors/wakeup/wakeup.c new file +9 -0
+	rv_attach_trace_probe("rtapp_wakeup", sys_enter, handle_sys_enter);
+	rv_attach_trace_probe("rtapp_wakeup", sys_exit, handle_sys_exit);
+
+	return 0;
+}
+
+static void disable_wakeup(void)
+{
+	rv_detach_trace_probe("rtapp_wakeup", sched_waking,

handle_sched_waking);

kernel/trace/rv/monitors/wakeup/wakeup.c new file +1 -0
+	rv_detach_trace_probe("rtapp_wakeup", contention_begin,

handle_contention_begin);

kernel/trace/rv/monitors/wakeup/wakeup.c new file +1 -0
+	rv_detach_trace_probe("rtapp_wakeup", contention_end,

handle_contention_end);

kernel/trace/rv/monitors/wakeup/wakeup.c new file +9 -0
+	rv_detach_trace_probe("rtapp_wakeup", sys_enter, handle_sys_enter);
+	rv_detach_trace_probe("rtapp_wakeup", sys_exit, handle_sys_exit);
+
+	ltl_monitor_destroy();
+}
+
+static struct rv_monitor rv_wakeup = {
+	.name = "wakeup",
+	.description = "Monitor that real-time tasks are not woken by lower-

priority tasks",

kernel/trace/rv/monitors/wakeup/wakeup.c new file +20 -0
+	.enable = enable_wakeup,
+	.disable = disable_wakeup,
+};
+
+static int __init register_wakeup(void)
+{
+	return rv_register_monitor(&rv_wakeup, &rv_rtapp);
+}
+
+static void __exit unregister_wakeup(void)
+{
+	rv_unregister_monitor(&rv_wakeup);
+}
+
+module_init(register_wakeup);
+module_exit(unregister_wakeup);
+
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Nam Cao <[email protected]>");
+MODULE_DESCRIPTION("Monitor that real-time tasks are not woken by lower-

priority tasks");

kernel/trace/rv/monitors/wakeup/wakeup.h new file +48 -0
new file mode 100644
index 000000000000..6f80da64e0e1
@@ -0,0 +1,92 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * C implementation of Buchi automaton, automatically generated by
+ * tools/verification/rvgen from the linear temporal logic specification.
+ * For further information, see kernel documentation:
+ *   Documentation/trace/rv/linear_temporal_logic.rst
+ */
+
+#include <linux/rv.h>
+
+#define MONITOR_NAME wakeup
+
+enum ltl_atom {
+	LTL_BLOCK_ON_RT_MUTEX,
+	LTL_FUTEX_LOCK_PI,
+	LTL_RT,
+	LTL_USER_THREAD,
+	LTL_WOKEN_BY_LOWER_PRIO,
+	LTL_WOKEN_BY_SOFTIRQ,
+	LTL_NUM_ATOM
+};
+static_assert(LTL_NUM_ATOM <= RV_MAX_LTL_ATOM);
+
+static const char *ltl_atom_str(enum ltl_atom atom)
+{
+	static const char *const names[] = {
+		"bl_on_rt_mu",
+		"fu_lo_pi",
+		"rt",
+		"us_th",
+		"wo_lo_pr",
+		"wo_so",
+	};
+
+	return names[atom];
+}
+
+enum ltl_buchi_state {
+	S0,
+	RV_NUM_BA_STATES
+};
+static_assert(RV_NUM_BA_STATES <= RV_MAX_BA_STATES);
+
+static void ltl_start(struct task_struct *task, struct ltl_monitor *mon)
+{
+	bool woken_by_softirq = test_bit(LTL_WOKEN_BY_SOFTIRQ, mon->atoms);
+	bool woken_by_lower_prio = test_bit(LTL_WOKEN_BY_LOWER_PRIO, mon-
Quoted 1 line

atoms);

kernel/trace/rv/monitors/wakeup/wakeup.h new file +20 -0
+	bool user_thread = test_bit(LTL_USER_THREAD, mon->atoms);
+	bool rt = test_bit(LTL_RT, mon->atoms);
+	bool futex_lock_pi = test_bit(LTL_FUTEX_LOCK_PI, mon->atoms);
+	bool block_on_rt_mutex = test_bit(LTL_BLOCK_ON_RT_MUTEX, mon->atoms);
+	bool val9 = block_on_rt_mutex || futex_lock_pi;
+	bool val6 = !woken_by_softirq;
+	bool val5 = !woken_by_lower_prio;
+	bool val8 = val5 && val6;
+	bool val10 = val8 || val9;
+	bool val3 = !user_thread;
+	bool val2 = !rt;
+	bool val4 = val2 || val3;
+	bool val11 = val4 || val10;
+
+	if (val11)
+		__set_bit(S0, mon->states);
+}
+
+static void
+ltl_possible_next_states(struct ltl_monitor *mon, unsigned int state,

unsigned long *next)

kernel/trace/rv/monitors/wakeup/wakeup.h new file +3 -0
+{
+	bool woken_by_softirq = test_bit(LTL_WOKEN_BY_SOFTIRQ, mon->atoms);
+	bool woken_by_lower_prio = test_bit(LTL_WOKEN_BY_LOWER_PRIO, mon-
Quoted 1 line

atoms);

kernel/trace/rv/monitors/wakeup/wakeup.h new file +21 -0
+	bool user_thread = test_bit(LTL_USER_THREAD, mon->atoms);
+	bool rt = test_bit(LTL_RT, mon->atoms);
+	bool futex_lock_pi = test_bit(LTL_FUTEX_LOCK_PI, mon->atoms);
+	bool block_on_rt_mutex = test_bit(LTL_BLOCK_ON_RT_MUTEX, mon->atoms);
+	bool val9 = block_on_rt_mutex || futex_lock_pi;
+	bool val6 = !woken_by_softirq;
+	bool val5 = !woken_by_lower_prio;
+	bool val8 = val5 && val6;
+	bool val10 = val8 || val9;
+	bool val3 = !user_thread;
+	bool val2 = !rt;
+	bool val4 = val2 || val3;
+	bool val11 = val4 || val10;
+
+	switch (state) {
+	case S0:
+		if (val11)
+			__set_bit(S0, next);
+		break;
+	}
+}
kernel/trace/rv/monitors/wakeup/wakeup_trace.h new file +9 -0
new file mode 100644
index 000000000000..7e056183f920
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * Snippet to be included in rv_trace.h
+ */
+
+#ifdef CONFIG_RV_MON_WAKEUP
+DEFINE_EVENT(event_ltl_monitor_id, event_wakeup,
+	     TP_PROTO(struct task_struct *task, char *states, char *atoms,

char *next),

kernel/trace/rv/monitors/wakeup/wakeup_trace.h new file +5 -0
+	     TP_ARGS(task, states, atoms, next));
+DEFINE_EVENT(error_ltl_monitor_id, error_wakeup,
+	     TP_PROTO(struct task_struct *task),
+	     TP_ARGS(task));
+#endif /* CONFIG_RV_MON_WAKEUP */
kernel/trace/rv/rv_trace.h +1 -0
index 9622c269789c..2f8a932432c9 100644
@@ -241,6 +241,7 @@ DECLARE_EVENT_CLASS(error_ltl_monitor_id,
  );
  #include <monitors/pagefault/pagefault_trace.h>
  #include <monitors/sleep/sleep_trace.h>
+#include <monitors/wakeup/wakeup_trace.h>
  // Add new monitors based on CONFIG_LTL_MON_EVENTS_ID here
  #endif /* CONFIG_LTL_MON_EVENTS_ID */

tools/verification/models/rtapp/wakeup.ltl new file +2 -0
new file mode 100644
index 000000000000..a5d63ca0811a
@@ -0,0 +1,5 @@
+RULE = always (((RT and USER_THREAD) imply
+		(not (WOKEN_BY_LOWER_PRIO or WOKEN_BY_SOFTIRQ)) or

ALLOWLIST))

tools/verification/models/rtapp/wakeup.ltl new file +3 -0
+
+ALLOWLIST = BLOCK_ON_RT_MUTEX
+         or FUTEX_LOCK_PI
author
#10 lore raw

Re: [PATCH v2 4/4] rv/rtapp: Add wakeup monitor

Quoted 6 lines · Gabriele Monaco

This looks good, but if I understand it correctly, the same violation should be spotted by both monitors from two different perspectives, but sleep catches more things (e.g. tasks using wrong sleeping ways despite their wakeup):

  # perf stat -a -e rv:error_sleep -e rv:error_wakeup -- stress-ng --cpu 5 --cpu-load 90 --sched rr -t 5
Quoted 7 lines

Performance counter stats for 'system wide':

285 rv:error_sleep
20 rv:error_wakeup

Provided I don't really know what's happening down there (I just let the stressor run free), this discrepancy is expected, right?

Thanks for testing it.

Beside monitoring low-prio task waking high-prio task, the rtapp/sleep monitor also checks that user uses the correct APIs for sleeping. So yes, this discrepancy is expected.

Nam

maintainer Reviewed-by
#11 lore raw

Re: [PATCH v2 4/4] rv/rtapp: Add wakeup monitor

Quoted 17 lines · Nam Cao
Quoted 16 lines · Gabriele Monaco

This looks good, but if I understand it correctly, the same violation should be
spotted by both monitors from two different perspectives, but sleep catches more
things (e.g. tasks using wrong sleeping ways despite their wakeup):

  # perf stat -a -e rv:error_sleep -e rv:error_wakeup -- stress-ng --cpu 5 -
-cpu-load 90 --sched rr -t 5

   Performance counter stats for 'system wide':

               285
rv:error_sleep
                20
rv:error_wakeup

Quoted 8 lines
Quoted 2 lines

Provided I don't really know what's happening down there (I just let the stressor run free), this discrepancy is expected, right?

Thanks for testing it.

Beside monitoring low-prio task waking high-prio task, the rtapp/sleep monitor also checks that user uses the correct APIs for sleeping. So yes, this discrepancy is expected.

Alright makes sense, thanks.

#12 lore raw

Re: [PATCH v2 4/4] rv/rtapp: Add wakeup monitor

In reply to Nam Cao

Hi Nam,

I've been looking into the RV subsystem, and while reading this patch I noticed a small inconsistency.

Quoted 11 lines · Nam Cao

Add a wakeup monitor to detect a lower-priority task waking up a higher-priority task.

The rtapp/sleep monitor already detects this. However, that monitor triggers an error in the context of the wakee task and user only gets the stacktrace of that task. It is also extremely useful to get the stacktrace of the waker task, which this monitor offers. In other words, this monitor complements the rtapp/sleep monitor.

---

[...]

Quoted 12 lines
Documentation/trace/rv/monitor_rtapp.rst +12 -0
+Monitor wakeup
+++++++++++++++
+
+The `wakeup` monitor reports real-time threads being woken by lower-priority threads,
+which is a hint of priority inversion. Its specification is::
+
+  RULE = always (((RT and USER_THREAD) imply
+                (not (WOKEN_BY_LOWER_PRIO or WOKEN_BY_SOFTIRQ)) or ALLOWLIST))
+
+  ALLOWLIST = BLOCK_ON_RT_MUTEX
+           or FUTEX_LOCK_PI
+

[...]

kernel/trace/rv/monitors/wakeup/wakeup.c line 59

Quoted 5 lines
kernel/trace/rv/monitors/wakeup/wakeup.c new file +5 -0
+static void handle_sched_waking(void *data, struct task_struct *task)
+{
+	if (in_task()) {
+		if (current->prio > task->prio)
+			ltl_atom_pulse(task, LTL_WOKEN_BY_LOWER_PRIO, true);

The implementation also reports wakeups from softirq context through WOKEN_BY_SOFTIRQ:

kernel/trace/rv/monitors/wakeup/wakeup.c line 62

Quoted 3 lines
kernel/trace/rv/monitors/wakeup/wakeup.c new file +3 -0
+	} else if (in_serving_softirq()) {
+		ltl_atom_pulse(task, LTL_WOKEN_BY_SOFTIRQ, true);
+	}

Should this description mention softirq wakeups as well?

The Kconfig help text, struct rv_monitor's .description, and MODULE_DESCRIPTION() seem to have the same omission.

I also have a small follow-up documentation patch based on this series, which I plan to send as an RFC shortly.

Thanks,
Chao

author
#13 lore raw

Re: [PATCH v2 4/4] rv/rtapp: Add wakeup monitor

Quoted 9 lines · Chao Liu

kernel/trace/rv/monitors/wakeup/wakeup.c line 59

Quoted 5 lines
kernel/trace/rv/monitors/wakeup/wakeup.c new file +5 -0
+static void handle_sched_waking(void *data, struct task_struct *task)
+{
+	if (in_task()) {
+		if (current->prio > task->prio)
+			ltl_atom_pulse(task, LTL_WOKEN_BY_LOWER_PRIO, true);

The implementation also reports wakeups from softirq context through WOKEN_BY_SOFTIRQ:

Quoted 5 lines

kernel/trace/rv/monitors/wakeup/wakeup.c line 62

Quoted 3 lines
kernel/trace/rv/monitors/wakeup/wakeup.c new file +3 -0
+	} else if (in_serving_softirq()) {
+		ltl_atom_pulse(task, LTL_WOKEN_BY_SOFTIRQ, true);
+	}

Should this description mention softirq wakeups as well?

Technically, on RT kernel, softirq is a SCHED_OTHER task. On !RT kernel, softirq can also be executed as a SCHED_OTHER task, which is the cause for concern.

But explicitly mentioning softirq does not hurt, I wouldn't oppose that.

Nam

#14 lore raw

Re: [PATCH v2 4/4] rv/rtapp: Add wakeup monitor

Quoted 16 lines · Nam Cao
Quoted 9 lines · Chao Liu

kernel/trace/rv/monitors/wakeup/wakeup.c line 59

Quoted 5 lines
kernel/trace/rv/monitors/wakeup/wakeup.c new file +5 -0
+static void handle_sched_waking(void *data, struct task_struct *task)
+{
+	if (in_task()) {
+		if (current->prio > task->prio)
+			ltl_atom_pulse(task, LTL_WOKEN_BY_LOWER_PRIO, true);

The implementation also reports wakeups from softirq context through WOKEN_BY_SOFTIRQ:

Quoted 5 lines

kernel/trace/rv/monitors/wakeup/wakeup.c line 62

Quoted 3 lines
kernel/trace/rv/monitors/wakeup/wakeup.c new file +3 -0
+	} else if (in_serving_softirq()) {
+		ltl_atom_pulse(task, LTL_WOKEN_BY_SOFTIRQ, true);
+	}

Should this description mention softirq wakeups as well?

Quoted 5 lines

Technically, on RT kernel, softirq is a SCHED_OTHER task. On !RT kernel, softirq can also be executed as a SCHED_OTHER task, which is the cause for concern.

But explicitly mentioning softirq does not hurt, I wouldn't oppose that.

Thanks for the clarification. I understand it better now.

Thanks,
Chao

Quoted 2 lines

Nam

maintainer
#15 lore raw

Re: [PATCH v2 4/4] rv/rtapp: Add wakeup monitor

In reply to Nam Cao
Quoted 17 lines · Nam Cao
Quoted 10 lines · Chao Liu
Quoted 6 lines
kernel/trace/rv/monitors/wakeup/wakeup.c new file +5 -0
+static void handle_sched_waking(void *data, struct task_struct *task)
+{
+	if (in_task()) {
+		if (current->prio > task->prio)
+			ltl_atom_pulse(task, LTL_WOKEN_BY_LOWER_PRIO,

true);

The implementation also reports wakeups from softirq context through WOKEN_BY_SOFTIRQ:

Quoted 5 lines

kernel/trace/rv/monitors/wakeup/wakeup.c line 62

Quoted 3 lines
kernel/trace/rv/monitors/wakeup/wakeup.c new file +3 -0
+	} else if (in_serving_softirq()) {
+		ltl_atom_pulse(task, LTL_WOKEN_BY_SOFTIRQ, true);
+	}

Should this description mention softirq wakeups as well?

Quoted 5 lines

Technically, on RT kernel, softirq is a SCHED_OTHER task. On !RT kernel, softirq can also be executed as a SCHED_OTHER task, which is the cause for concern.

But explicitly mentioning softirq does not hurt, I wouldn't oppose that.

Sounds good to me. So as far as I understand, this change is going to go through a separate patch right?
I'm considering this series final.

Thanks,
Gabriele

author
#16 lore raw

Re: [PATCH v2 4/4] rv/rtapp: Add wakeup monitor

Quoted 3 lines · Gabriele Monaco

Sounds good to me. So as far as I understand, this change is going to go through a separate patch right?
I'm considering this series final.

That's how I understand it.

Nam