LTL per-cpu monitor type and real-time scheduling monitor
Subsystems (4)
v2 cover LTL per-cpu monitor type and real-time scheduling monitor
Hi,
This series adds support for linear temporal logic per-cpu monitor type, analogous to deterministic automaton per-cpu monitor. Then a new per-cpu monitor is added which validates real-time scheduling.
Nam Cao (5):
rv/ltl: Prepare for other monitor types
rv/ltl: Support per-cpu monitors verification/rvgen/ltl: Support per-cpu monitor generation
sched: Add task enqueue/dequeue trace points
rv: Add rts monitor
Documentation/trace/rv/monitor_sched.rst | 19 +++ include/linux/rv.h | 4 + include/rv/ltl_monitor.h | 117 +++++++++----- include/trace/events/sched.h | 13 ++ kernel/sched/core.c | 8 +- kernel/trace/rv/Kconfig | 5 + kernel/trace/rv/Makefile | 1 + .../trace/rv/monitors/pagefault/pagefault.h | 2 + kernel/trace/rv/monitors/rts/Kconfig | 17 +++ kernel/trace/rv/monitors/rts/rts.c | 144 ++++++++++++++++++ kernel/trace/rv/monitors/rts/rts.h | 126 +++++++++++++++ kernel/trace/rv/monitors/rts/rts_trace.h | 15 ++ kernel/trace/rv/monitors/sleep/sleep.h | 2 + kernel/trace/rv/rv_trace.h | 47 ++++++ tools/verification/models/sched/rts.ltl | 5 + tools/verification/rvgen/rvgen/ltl2k.py | 48 +++++- .../rvgen/rvgen/templates/ltl2k/main.c | 9 +- .../rvgen/rvgen/templates/ltl2k/trace.h | 7 +- 18 files changed, 539 insertions(+), 50 deletions(-) create mode 100644 kernel/trace/rv/monitors/rts/Kconfig create mode 100644 kernel/trace/rv/monitors/rts/rts.c create mode 100644 kernel/trace/rv/monitors/rts/rts.h create mode 100644 kernel/trace/rv/monitors/rts/rts_trace.h create mode 100644 tools/verification/models/sched/rts.ltl
Signature · 1 line
2.39.5
PATCH v2 1/5 rv/ltl: Prepare for other monitor types
rv/ltl_monitor.h is the template file used by all LTL monitors. But only per-task monitor is supported.
Prepare to support for other monitor types:
- Change the monitored target type into an opaque type which will be defined differently depending on the monitor type. This type is only defined as struct task_struct * for now.
- Separate out the per-task-specific printf format and arguments, so that rv_cond_react() can be shared with other monitor types.
No functional change intended.
---
v2: define the macros for DA usage as well
--- include/linux/rv.h | 4 + include/rv/ltl_monitor.h | 85 +++++++++++-------- .../trace/rv/monitors/pagefault/pagefault.h | 2 + kernel/trace/rv/monitors/sleep/sleep.h | 2 + 4 files changed, 58 insertions(+), 35 deletions(-)
include/linux/rv.h +4 -0
@@ -13,6 +13,10 @@#define MAX_DA_NAME_LEN 32#define MAX_DA_RETRY_RACING_EVENTS 3 +#define RV_MON_GLOBAL 0 +#define RV_MON_PER_CPU 1 +#define RV_MON_PER_TASK 2 +#ifdef CONFIG_RV #include <linux/bitops.h> #include <linux/types.h>
include/rv/ltl_monitor.h +50 -35
178 lines not on this page · raw message
kernel/trace/rv/monitors/pagefault/pagefault.h +2 -0
@@ -11,6 +11,8 @@#define MONITOR_NAME pagefault +#define LTL_MONITOR_TYPE RV_MON_PER_TASK +enum ltl_atom {LTL_PAGEFAULT ,LTL_RT ,
kernel/trace/rv/monitors/sleep/sleep.h +2 -0
@@ -11,6 +11,8 @@#define MONITOR_NAME sleep +#define LTL_MONITOR_TYPE RV_MON_PER_TASK +enum ltl_atom {LTL_ABORT_SLEEP ,LTL_BLOCK_ON_RT_MUTEX ,
Signature · 1 line
2.39.5
PATCH v2 2/5 rv/ltl: Support per-cpu monitors
In reply to Nam CaoAdd support for per-cpu run-time verification linear temporal logic monitors. This is analogous to deterministic automaton per-cpu monitors.
---
v2: Rename "implicit" to "cpu"
--- include/rv/ltl_monitor.h | 32 ++++++++++++++++++++++++++ kernel/trace/rv/Kconfig | 4 ++++ kernel/trace/rv/rv_trace.h | 46 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 82 insertions(+)
include/rv/ltl_monitor.h +32 -0
@@ -23,12 +23,21 @@typedef struct task_struct *monitor_target ; +#elif LTL_MONITOR_TYPE ==RV_MON_PER_CPU + +#define TARGET_PRINT_FORMAT "%u" +#define TARGET_PRINT_ARGS (cpu) cpu + +typedef unsigned int monitor_target ; +#endif #ifdef CONFIG_RV_REACTORS #define RV_MONITOR_NAME CONCATENATE(rv_, MONITOR_NAME)static struct rv_monitor RV_MONITOR_NAME ; +static struct ltl_monitor *ltl_get_monitor (monitor_target target); +static void rv_cond_react (monitor_target target) {if (!rv_reacting_on () || !RV_MONITOR_NAME .react)
@@ -54,6 +63,13 @@ static struct ltl_monitor *ltl_get_monitor(monitor_target target) {return &target->rv[ltl_monitor_slot].ltl_mon; } +#elif LTL_MONITOR_TYPE ==RV_MON_PER_CPU +static struct ltl_monitor *ltl_get_monitor (unsigned int cpu) +{ + static DEFINE_PER_CPU(struct ltl_monitor, ltl_monitor); + +return per_cpu_ptr (<l_monitor, cpu); +}#endif static void ltl_target_init (monitor_target target,bool target_creation)
@@ -108,6 +124,22 @@ static void ltl_monitor_destroy(void)rv_put_task_monitor_slot (ltl_monitor_slot); ltl_monitor_slot = RV_PER_TASK_MONITOR_INIT; } + +#elif LTL_MONITOR_TYPE == RV_MON_PER_CPU + +static int ltl_monitor_init (void ) +{ +unsigned int cpu; + +for_each_possible_cpu (cpu) +ltl_target_init (cpu, false); +return 0 ; +} + +static void ltl_monitor_destroy (void ) +{ +} +#endif static void ltl_illegal_state (monitor_target target,struct ltl_monitor *mon)
kernel/trace/rv/Kconfig +4 -0
@@ -16,6 +16,10 @@ config DA_MON_EVENTS_ID select RV_MON_MAINTENANCE_EVENTS bool +config LTL_MON_EVENTS_CPU + select RV_MON_EVENTS + bool + config LTL_MON_EVENTS_ID select RV_MON_EVENTS bool
kernel/trace/rv/rv_trace.h +46 -0
54 lines not on this page · raw message
Signature · 1 line
2.39.5
Re: [PATCH v2 2/5] rv/ltl: Support per-cpu monitors
Quoted 7 lines · Nam Cao
Add support for per-cpu run-time verification linear temporal logic monitors. This is analogous to deterministic automaton per-cpu monitors.
Signed-off-by Nam Cao <[email protected]>---
v2: Rename "implicit" to "cpu"
Looks good to me.
Thanks,
Gabriele
Quoted 150 lines
---
include/rv/ltl_monitor.h | 32 ++++++++++++++++++++++++++ kernel/trace/rv/Kconfig | 4 ++++kernel/trace/rv/rv_trace.h| 46
++++++++++++++++++++++++++++++++++++++3 files changed, 82 insertions(+)include/rv/ltl_monitor.h +0
-0@@ -23,12 +23,21 @@
typedef structtask_struct*monitor_target;include/rv/ltl_monitor.h +7
-0+#elif LTL_MONITOR_TYPE ==RV_MON_PER_CPU + +#define TARGET_PRINT_FORMAT "%u" +#define TARGET_PRINT_ARGS (cpu) cpu + +typedef unsigned int monitor_target ; +#endif
#ifdefCONFIG_RV_REACTORS
#defineRV_MONITOR_NAMECONCATENATE(rv_, MONITOR_NAME)static structrv_monitorRV_MONITOR_NAME;include/rv/ltl_monitor.h +2
-0+static struct ltl_monitor *ltl_get_monitor (monitor_target target); +static void
rv_cond_react(monitor_target target){
if (!rv_reacting_on() || !RV_MONITOR_NAME.react)include/rv/ltl_monitor.h +7
-0@@ -54,6 +63,13 @@ static struct ltl_monitor *ltl_get_monitor(monitor_target target) { return &target->rv[ltl_monitor_slot].ltl_mon; } +#elif LTL_MONITOR_TYPE ==RV_MON_PER_CPU +static struct ltl_monitor *ltl_get_monitor (unsigned int cpu) +{ + static DEFINE_PER_CPU(struct ltl_monitor, ltl_monitor); + +return per_cpu_ptr (<l_monitor, cpu); +} #endif
static voidltl_target_init(monitor_target target, bool target_creation)include/rv/ltl_monitor.h +16
-0@@ -108,6 +124,22 @@ static void ltl_monitor_destroy(void) rv_put_task_monitor_slot(ltl_monitor_slot); ltl_monitor_slot = RV_PER_TASK_MONITOR_INIT; } + +#elif LTL_MONITOR_TYPE ==RV_MON_PER_CPU + +static int ltl_monitor_init (void ) +{ +unsigned int cpu; + +for_each_possible_cpu (cpu) +ltl_target_init (cpu, false); +return 0 ; +} + +static void ltl_monitor_destroy (void ) +{ +} + #endif
static voidltl_illegal_state(monitor_target target, struct ltl_monitor *mon)kernel/trace/rv/Kconfig +0
-0@@ -16,6 +16,10 @@ config DA_MON_EVENTS_IDselect
RV_MON_MAINTENANCE_EVENTS
boolkernel/trace/rv/Kconfig +4
-0+config LTL_MON_EVENTS_CPU + select RV_MON_EVENTS + bool +config
LTL_MON_EVENTS_ID
selectRV_MON_EVENTS
boolkernel/trace/rv/rv_trace.h +1
-0@@ -177,8 +177,54 @@ DECLARE_EVENT_CLASS(error_ltl_monitor_id, #include <monitors/pagefault/pagefault_trace.h> #include <monitors/sleep/sleep_trace.h> // Add new monitors based on CONFIG_LTL_MON_EVENTS_ID here + #endif /* CONFIG_LTL_MON_EVENTS_ID */kernel/trace/rv/rv_trace.h +4
-0+#ifdef CONFIG_LTL_MON_EVENTS_CPU +DECLARE_EVENT_CLASS(event_ltl_monitor_cpu, + + TP_PROTO(unsigned int cpu, char *states, char *atoms, char*next),
kernel/trace/rv/rv_trace.h +19
-0+ + TP_ARGS(cpu, states, atoms, next), + +TP_STRUCT__entry ( +__field (unsigned int ,cpu ) +__string (states, states) +__string (atoms, atoms) +__string (next, next) + ), + +TP_fast_assign ( + __entry->cpu = cpu; +__assign_str (states); +__assign_str (atoms); +__assign_str (next); + ), + + TP_printk("cpu%u: (%s) x (%s) -> (%s)", __entry->cpu, + __get_str(states), __get_str(atoms),
__get_str(next))kernel/trace/rv/rv_trace.h +22
-0+); + +DECLARE_EVENT_CLASS (error_ltl_monitor_cpu, + + TP_PROTO(unsigned int cpu), + +TP_ARGS (cpu), + +TP_STRUCT__entry ( + __field(unsigned int, cpu) + ), + +TP_fast_assign ( + __entry->cpu = cpu; + ), + +TP_printk ("cpu%u: violation detected" , __entry->cpu) +); +// Add new monitors based on CONFIG_LTL_MON_EVENTS_CPU here + +#endif /* CONFIG_LTL_MON_EVENTS_CPU */ +#ifdef
CONFIG_RV_MON_MAINTENANCE_EVENTS
/* Tracepoint useful for monitors development, currenly only used in DA */
TRACE_EVENT(rv_retries_error,
PATCH v2 3/5 verification/rvgen/ltl: Support per-cpu monitor generation
In reply to Nam CaoAdd support to generate per-cpu LTL monitors. Similar to generating per-cpu monitors from .dot files, the "-t per_cpu" parameter can be used to generate per-cpu monitors from .ltl files.
---
v2: Rename "implicit" to "cpu"
--- tools/verification/rvgen/rvgen/ltl2k.py | 48 ++++++++++++++++--- .../rvgen/rvgen/templates/ltl2k/main.c | 9 ++-- .../rvgen/rvgen/templates/ltl2k/trace.h | 7 ++- 3 files changed, 50 insertions(+), 14 deletions(-)
tools/verification/rvgen/rvgen/ltl2k.py +42 -6
@@ -57,9 +57,16 @@ class ltl2k(generator.Monitor): template_dir = "ltl2k" def __init__(self, file_path, MonitorType, extra_params={}): - if MonitorType != "per_task": - raise NotImplementedError("Only per_task monitor is supported for LTL") + if MonitorType == "per_task": + self._target_arg = "struct task_struct *task" + self._target = "task" + elif MonitorType == "per_cpu": + self._target_arg = "unsigned int cpu" + self._target = "cpu" + else: + raise NotImplementedError(f"LTL does not support monitor type {MonitorType}") super().__init__(extra_params) + self.monitor_type = MonitorType with open(file_path) as f: self.atoms, self.ba, self.ltl = ltl2ba.create_graph(f.read()) self.atoms_abbr = abbreviate_atoms(self.atoms)
@@ -67,6 +74,13 @@ class ltl2k(generator.Monitor): if not self.name: self.name = Path(file_path).stem + def _fill_monitor_type(self) -> str: + if self.monitor_type == "per_task": + return "#define LTL_MONITOR_TYPE RV_MON_PER_TASK" + if self.monitor_type == "per_cpu": + return "#define LTL_MONITOR_TYPE RV_MON_PER_CPU" + assert False + def _fill_states(self) -> str: buf = [ "enum ltl_buchi_state {",
@@ -174,7 +188,7 @@ class ltl2k(generator.Monitor): def _fill_start(self): buf = [ - "static void ltl_start(struct task_struct *task, struct ltl_monitor *mon)", + "static void ltl_start(%s, struct ltl_monitor *mon)" % self._target_arg, "{" ]
@@ -205,7 +219,7 @@ class ltl2k(generator.Monitor): buff = [] buff.append("static void handle_example_event(void *data, /* XXX: fill header */)") buff.append("{") - buff.append("\tltl_atom_update(task, LTL_%s, true/false);" % self.atoms[0]) + buff.append("\tltl_atom_update(%s, LTL_%s, true/false);" % (self._target, self.atoms[0])) buff.append("}") buff.append("") return '\n'.join(buff)
@@ -241,6 +255,9 @@ class ltl2k(generator.Monitor): "" ] + buf.append(self._fill_monitor_type()) + buf.append('') + buf.extend(self._fill_atoms()) buf.append('')
@@ -259,13 +276,32 @@ class ltl2k(generator.Monitor): return '\n'.join(buf) def fill_monitor_class_type(self): - return "LTL_MON_EVENTS_ID" + if self.monitor_type == "per_task": + return "LTL_MON_EVENTS_ID" + if self.monitor_type == "per_cpu": + return "LTL_MON_EVENTS_CPU" + assert False def fill_monitor_class(self): - return "ltl_monitor_id" + if self.monitor_type == "per_task": + return "ltl_monitor_id" + if self.monitor_type == "per_cpu": + return "ltl_monitor_cpu" + assert False + + def fill_tracepoint_args_skel(self, tp_type): + if tp_type == "event": + return \ + ("\tTP_PROTO(%s, char *states, char *atoms, char *next),\n" % self._target_arg) + \ + ("\tTP_ARGS(%s, states, atoms, next)" % self._target) + if tp_type == "error": + return \ + ("\tTP_PROTO(%s),\n" % self._target_arg) + \ + ("\tTP_ARGS(%s)" % self._target) def fill_main_c(self): main_c = super().fill_main_c() main_c = main_c.replace("%%ATOMS_INIT%%", self.fill_atoms_init()) + main_c = main_c.replace("%%TARGET_ARG%%", self._target_arg) return main_c
tools/verification/rvgen/rvgen/templates/ltl2k/main.c +5 -4
@@ -23,7 +23,7 @@#include "%%MODEL_NAME%%.h" #include <rv/ltl_monitor.h> -static void ltl_atoms_fetch (struct task_struct *task,struct ltl_monitor *mon) +static void ltl_atoms_fetch(%%TARGET_ARG%%, struct ltl_monitor *mon) {/* * This is called everytime the Buchi automaton is triggered.
@@ -36,13 +36,14 @@ static void ltl_atoms_fetch(struct task_struct *task, struct ltl_monitor *mon)*/ } -static void ltl_atoms_init (struct task_struct *task,struct ltl_monitor *mon,bool task_creation) +static void ltl_atoms_init(%%TARGET_ARG%%, struct ltl_monitor *mon, bool target_creation) {/* * This should initialize as many atomic propositions as possible. * -* @task_creation indicates whether the task is being created. This is -* false if the task is already running before the monitor is enabled. +* @target_creation indicates whether the monitored target is being +* created. This is false if the monitor target is already online before +* the monitor is enabled. */ %%ATOMS_INIT%% }
tools/verification/rvgen/rvgen/templates/ltl2k/trace.h +3 -4
@@ -6,9 +6,8 @@ #ifdef CONFIG_RV_MON_%%MODEL_NAME_UP%% DEFINE_EVENT(event_%%MONITOR_CLASS%%, event_%%MODEL_NAME%%, - TP_PROTO(struct task_struct *task, char *states, char *atoms, char *next), -TP_ARGS (task, states, atoms, next)); +%%TRACEPOINT_ARGS_SKEL_EVENT%%); + DEFINE_EVENT(error_%%MONITOR_CLASS%%, error_%%MODEL_NAME%%, - TP_PROTO(struct task_struct *task), -TP_ARGS (task)); +%%TRACEPOINT_ARGS_SKEL_ERROR%%);#endif /* CONFIG_RV_MON_%%MODEL_NAME_UP%% */
Signature · 1 line
2.39.5
Re: [PATCH v2 3/5] verification/rvgen/ltl: Support per-cpu monitor generation
Quoted 26 lines · Nam Cao
Add support to generate per-cpu LTL monitors. Similar to generating per-cpu monitors from .dot files, the "-t
per_cpu" parameter can be used to generate per-cpu monitors from .ltl files.Signed-off-by Nam Cao <[email protected]>---
v2: Rename "implicit" to "cpu"
---
-static voidltl_atoms_init(struct task_struct *task, struct ltl_monitor *mon, bool task_creation)
+static voidltl_atoms_init(%%TARGET_ARG%%, struct ltl_monitor *mon, bool target_creation)
{
/*
* This should initialize as many atomic propositions as possible.
*(quoted diff) +0
-1- * @task_creation indicates whether the task is beingcreated. This is
(quoted diff) +0
-1- * false if the task is already running before the monitoris enabled.
(quoted diff) +1
-0+ * @target_creation indicates whether the monitored targetis being
(quoted diff) +1
-0+ * created. This is false if the monitor target is alreadyonline before
tools/verification/rvgen/rvgen/templates/ltl2k/main.c static void ltl_atoms_fetch(struct task_struct *task, struct ltl_monitor *mon) · line 46
Quoted 1 line
tools/verification/rvgen/rvgen/templates/ltl2k/main.c +1
-0+* the monitor is enabled.
I get you're trying to be more type-agnostic, but I believe this /online/ is a bit imprecise, unless you register a hotplug handler and just initialise the online CPUs (much of an overkill I'd say). What about something like "this is false if the monitor exists already before the monitor is enabled"
Other than that it looks good to me.
Thanks,
Gabriele
Quoted 22 lines
*/
%%ATOMS_INIT%%
}tools/verification/rvgen/rvgen/templates/ltl2k/trace.h +0
-0@@ -6,9 +6,8 @@
#ifdef CONFIG_RV_MON_%%MODEL_NAME_UP%% DEFINE_EVENT(event_%%MONITOR_CLASS%%, event_%%MODEL_NAME%%,tools/verification/rvgen/rvgen/templates/ltl2k/trace.h +0
-1- TP_PROTO(struct task_struct *task, char *states, char*atoms, char *next),
tools/verification/rvgen/rvgen/templates/ltl2k/trace.h +2
-1- TP_ARGS(task, states, atoms, next)); +%%TRACEPOINT_ARGS_SKEL_EVENT%%); +DEFINE_EVENT(error_%%MONITOR_CLASS%%, error_%%MODEL_NAME%%,
tools/verification/rvgen/rvgen/templates/ltl2k/trace.h +1
-2- TP_PROTO(struct task_struct *task), - TP_ARGS(task)); +%%TRACEPOINT_ARGS_SKEL_ERROR%%);#endif /* CONFIG_RV_MON_%%MODEL_NAME_UP%% */
Re: [PATCH v2 3/5] verification/rvgen/ltl: Support per-cpu monitor generation
Quoted 5 lines · Gabriele Monaco
I get you're trying to be more type-agnostic, but I believe this /online/ is a bit imprecise, unless you register a hotplug handler and just initialise the online CPUs (much of an overkill I'd say). What about something like "this is false if the monitor exists already before the monitor is enabled"
Sorry, after re-reading this one day later, I am still not sure why you says "online" is imprecise. Due to hotplug, CPUs can become online and offline.
The current implementation ignore hotplug and initialize all possible CPUs as if they are all oneline. But if hotplug becomes important in the future, I may add a CPU hotplug handler.
Quoted 3 lines
Other than that it looks good to me.
Reviewed-by Gabriele Monaco <[email protected]>
Thanks!
Nam
Gabriele Monaco
maintainerRe: [PATCH v2 3/5] verification/rvgen/ltl: Support per-cpu monitor generation
Quoted 7 lines · Nam Cao
Quoted 5 lines · Gabriele Monaco
I get you're trying to be more type-agnostic, but I believe this /online/ is a bit imprecise, unless you register a hotplug handler and just initialise the online CPUs (much of an overkill I'd say). What about something like "this is false if the monitor exists already before the monitor is enabled"
Quoted 7 lines
Sorry, after re-reading this one day later, I am still not sure why you says "online" is imprecise. Due to hotplug, CPUs can become online and offline.
The current implementation ignore hotplug and initialize all possible CPUs as if they are all oneline. But if hotplug becomes important in the future, I may add a CPU hotplug handler.
Alright, I was probably a bit unclear with that, I don't mean the implementation needs changes, only the wording.
Quoted 2 lines
This is false if the monitor target is already online before the monitor is enabled.
becomes
Quoted 2 lines
This is false if the CPU is already online before the monitor is enabled.
which is not always true, when starting the monitor, you initialise all possible CPUs, also offline ones, which are still initialised with @target_creation as false.
Mind I don't say you should change the value passed to ltl_target_init nor change your logic, I only mean /online/ isn't the right word here.
Does this make more sense?
Thanks,
Gabriele
Re: [PATCH v2 3/5] verification/rvgen/ltl: Support per-cpu monitor generation
Quoted 4 lines · Gabriele Monaco
Mind I don't say you should change the value passed to
ltl_target_initnor change your logic, I only mean /online/ isn't the right word here.Does this make more sense?
Yes, thanks for the elaboration!
Nam
PATCH v2 4/5 sched: Add task enqueue/dequeue trace points
In reply to Nam CaoAdd trace points into enqueue_task() and dequeue_task(). They are useful to implement RV monitor which validates RT scheduling.
---
Cc: Ingo Molnar <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Juri Lelli <[email protected]>
Cc: Vincent Guittot <[email protected]>
Cc: Dietmar Eggemann <[email protected]>
Cc: Ben Segall <[email protected]>
Cc: Mel Gorman <[email protected]>
Cc: Valentin Schneider <[email protected]>
Cc: K Prateek Nayak <[email protected]>
---
v2: Move the tracepoints to cover all task enqueue/dequeue, not just RT
--- include/trace/events/sched.h | 13 +++++++++++++ kernel/sched/core.c | 8 +++++++- 2 files changed, 20 insertions(+), 1 deletion(-)
include/trace/events/sched.h +13 -0
@@ -898,6 +898,19 @@ DECLARE_TRACE(sched_set_need_resched, TP_PROTO(struct task_struct *tsk, int cpu, int tif), TP_ARGS(tsk, cpu, tif)); +/* +* The two trace points below may not work as expected for fair tasks due +* to delayed dequeue. See: +* https://lore.kernel.org/lkml/[email protected]/ +*/ +DECLARE_TRACE (enqueue_task, + TP_PROTO(int cpu, struct task_struct *task), +TP_ARGS (cpu, task)); + +DECLARE_TRACE (dequeue_task, + TP_PROTO(int cpu, struct task_struct *task), +TP_ARGS (cpu, task)); +#endif /* _TRACE_SCHED_H */ /* This part must be outside protection */
kernel/sched/core.c +7 -1
@@ -2077,6 +2077,8 @@ unsigned long get_wchan(struct task_struct *p)void enqueue_task (struct rq *rq,struct task_struct *p,int flags) { +trace_enqueue_task_tp (rq->cpu, p); +if (!(flags &ENQUEUE_NOCLOCK ))update_rq_clock (rq);
@@ -2119,7 +2121,11 @@ inline bool dequeue_task(struct rq *rq, struct task_struct *p, int flags)* and mark the task ->sched_delayed. */ uclamp_rq_dec (rq, p); -return p->sched_class->dequeue_task (rq, p, flags); +if (p->sched_class->dequeue_task (rq, p, flags)) { +trace_dequeue_task_tp (rq->cpu, p); +return true; + } +return false; }void activate_task (struct rq *rq,struct task_struct *p,int flags)
Signature · 1 line
2.39.5
Peter Zijlstra
maintainerRe: [PATCH v2 4/5] sched: Add task enqueue/dequeue trace points
Quoted 12 lines · Nam Cao
include/trace/events/sched.h +9
-0+/* +* The two trace points below may not work as expected for fair tasks due +* to delayed dequeue. See: +* https://lore.kernel.org/lkml/[email protected]/ +*/ +DECLARE_TRACE (dequeue_task, + TP_PROTO(int cpu, struct task_struct *task), +TP_ARGS (cpu, task)); +
include/trace/events/sched.h inline bool dequeue_task(struct rq *rq, struct task_struct *p, int flags) · line 2129
Quoted 10 lines
include/trace/events/sched.h +5
-1@@ -2119,7 +2121,11 @@ inline bool dequeue_task(struct rq *rq, struct task_struct *p, int flags)* and mark the task ->sched_delayed. */ uclamp_rq_dec (rq, p); -return p->sched_class->dequeue_task (rq, p, flags); +if (p->sched_class->dequeue_task (rq, p, flags)) { +trace_dequeue_task_tp (rq->cpu, p); +return true; + } +return false; }
Hurmpff.. that's not very nice.
How about something like:
dequeue_task():
... ret = p->sched_class->dequeue_task(rq, p, flags); if (trace_dequeue_task_p_enabled() && !(flags & DEQUEUE_SLEEP)) __trace_dequeue_task_tp(rq->cpu, p); return ret;
__block_task():
trace_dequeue_task_tp(rq->cpu, p); ...
Specifically, only DEQUEUE_SLEEP is allowed to fail, and DEQUEUE_SLEEP will eventually cause __block_task() to be called, either directly, or delayed.
Peter Zijlstra
maintainerRe: [PATCH v2 4/5] sched: Add task enqueue/dequeue trace points
Quoted 37 lines · Peter Zijlstra
Quoted 12 lines · Nam Cao
include/trace/events/sched.h +9
-0+/* +* The two trace points below may not work as expected for fair tasks due +* to delayed dequeue. See: +* https://lore.kernel.org/lkml/[email protected]/ +*/ +DECLARE_TRACE (dequeue_task, + TP_PROTO(int cpu, struct task_struct *task), +TP_ARGS (cpu, task)); +include/trace/events/sched.h inline bool dequeue_task(struct rq *rq, struct task_struct *p, int flags) · line 2129
Quoted 10 lines
include/trace/events/sched.h +5
-1@@ -2119,7 +2121,11 @@ inline bool dequeue_task(struct rq *rq, struct task_struct *p, int flags)* and mark the task ->sched_delayed. */ uclamp_rq_dec (rq, p); -return p->sched_class->dequeue_task (rq, p, flags); +if (p->sched_class->dequeue_task (rq, p, flags)) { +trace_dequeue_task_tp (rq->cpu, p); +return true; + } +return false; }Hurmpff.. that's not very nice.
How about something like:
dequeue_task():... ret = p->sched_class->dequeue_task(rq, p, flags); if (trace_dequeue_task_p_enabled() && !(flags & DEQUEUE_SLEEP)) __trace_dequeue_task_tp(rq->cpu, p); return ret;
Quoted 8 lines
__block_task():trace_dequeue_task_tp(rq->cpu, p); ...Specifically, only
DEQUEUE_SLEEPis allowed to fail, andDEQUEUE_SLEEPwill eventually cause__block_task()to be called, either directly, or delayed.
If you extend the tracepoint with the sleep state, you can probably remove the nr_running tracepoints. Esp. once we get this new throttle stuff sorted.
Re: [PATCH v2 4/5] sched: Add task enqueue/dequeue trace points
Quoted 43 lines · Peter Zijlstra
Quoted 42 lines · Peter Zijlstra
Quoted 12 lines · Nam Cao
include/trace/events/sched.h +9
-0+/* +* The two trace points below may not work as expected for fair tasks due +* to delayed dequeue. See: +* https://lore.kernel.org/lkml/[email protected]/ +*/ +DECLARE_TRACE (dequeue_task, + TP_PROTO(int cpu, struct task_struct *task), +TP_ARGS (cpu, task)); +include/trace/events/sched.h inline bool dequeue_task(struct rq *rq, struct task_struct *p, int flags) · line 2129
Quoted 10 lines
include/trace/events/sched.h +5
-1@@ -2119,7 +2121,11 @@ inline bool dequeue_task(struct rq *rq, struct task_struct *p, int flags)* and mark the task ->sched_delayed. */ uclamp_rq_dec (rq, p); -return p->sched_class->dequeue_task (rq, p, flags); +if (p->sched_class->dequeue_task (rq, p, flags)) { +trace_dequeue_task_tp (rq->cpu, p); +return true; + } +return false; }Hurmpff.. that's not very nice.
How about something like:
dequeue_task():... ret = p->sched_class->dequeue_task(rq, p, flags); if (trace_dequeue_task_p_enabled() && !(flags & DEQUEUE_SLEEP)) __trace_dequeue_task_tp(rq->cpu, p); return ret;
__block_task():trace_dequeue_task_tp(rq->cpu, p); ...
Quoted 7 lines
Quoted 3 lines
Specifically, only
DEQUEUE_SLEEPis allowed to fail, andDEQUEUE_SLEEPwill eventually cause__block_task()to be called, either directly, or delayed.If you extend the tracepoint with the sleep state, you can probably remove the
nr_runningtracepoints. Esp. once we get this new throttle stuff sorted.
Sorry, I'm a bit out of depth here. Can you elaborate?
By "sleep state" do you mean (flags & DEQUEUE_SLEEP)? The nr_running tracepoints are not hit if the task is throttled, while these new tracepoints are hit. How does the sleep state help with this difference?
Also +Cc Phil Auld <[email protected]>, who seems to care about the nr_running tracepoints.
Nam
K Prateek Nayak
reviewerRe: [PATCH v2 4/5] sched: Add task enqueue/dequeue trace points
Hello Nam,
Quoted 23 lines · Nam Cao
Quoted 15 lines
Quoted 15 lines
How about something like:
dequeue_task():... ret = p->sched_class->dequeue_task(rq, p, flags); if (trace_dequeue_task_p_enabled() && !(flags & DEQUEUE_SLEEP)) __trace_dequeue_task_tp(rq->cpu, p); return ret;
__block_task():trace_dequeue_task_tp(rq->cpu, p); ...Quoted 7 lines
Quoted 3 lines
Specifically, only
DEQUEUE_SLEEPis allowed to fail, andDEQUEUE_SLEEPwill eventually cause__block_task()to be called, either directly, or delayed.If you extend the tracepoint with the sleep state, you can probably remove the
nr_runningtracepoints. Esp. once we get this new throttle stuff sorted.
Quoted 5 lines
Sorry, I'm a bit out of depth here. Can you elaborate?
By "sleep state" do you mean (flags &
DEQUEUE_SLEEP)? Thenr_runningtracepoints are not hit if the task is throttled, while these new tracepoints are hit. How does the sleep state help with this difference?
Once we have per-task throttling being discussed in
https://lore.kernel.org/lkml/[email protected]/
throttled tasks will do a
dequeue_task_fair(rq, p, DEQUEUE_SLEEP | DEQUEUE_THROTTLE);
and remove themselves from the runqueue but they won't hit block_task().
To preserve current throttle behavior, I don't think per-task throttle should call dequeue_task() directly since it does a bunch more stuff with core-sched dequeue, psi, uclamp, etc or maybe it is fine to do that now with per-task throttling?
Peter, what do you think?
If we don't what to do all that stuff in the throttle path, adding to Peter's suggestion, perhaps we can have a wrapper like:
int __dequeue_task(rq, p, flags)
int ret = p->sched_class->dequeue_task(rq, p, flags);
if (trace_dequeue_task_p_enabled() &&
!((flags & (DEQUEUE_SLEEP | DEQUEUE_THROTTLE)) == DEQUEUE_SLEEP))
__trace_dequeue_task_tp(rq->cpu, p);
return ret;and then per-task throttle can just call __dequeue_task() instead. I'll let Peter chime in with his thoughts.
Quoted 5 lines
Also +Cc Phil Auld <[email protected]>, who seems to care about the
nr_runningtracepoints.Nam
Signature · 2 lines
Thanks and Regards, Prateek
Re: [PATCH v2 4/5] sched: Add task enqueue/dequeue trace points
In reply to Peter ZijlstraQuoted 37 lines · Peter Zijlstra
Quoted 12 lines · Nam Cao
include/trace/events/sched.h +9
-0+/* +* The two trace points below may not work as expected for fair tasks due +* to delayed dequeue. See: +* https://lore.kernel.org/lkml/[email protected]/ +*/ +DECLARE_TRACE (dequeue_task, + TP_PROTO(int cpu, struct task_struct *task), +TP_ARGS (cpu, task)); +include/trace/events/sched.h inline bool dequeue_task(struct rq *rq, struct task_struct *p, int flags) · line 2129
Quoted 10 lines
include/trace/events/sched.h +5
-1@@ -2119,7 +2121,11 @@ inline bool dequeue_task(struct rq *rq, struct task_struct *p, int flags)* and mark the task ->sched_delayed. */ uclamp_rq_dec (rq, p); -return p->sched_class->dequeue_task (rq, p, flags); +if (p->sched_class->dequeue_task (rq, p, flags)) { +trace_dequeue_task_tp (rq->cpu, p); +return true; + } +return false; }Hurmpff.. that's not very nice.
How about something like:
dequeue_task():... ret = p->sched_class->dequeue_task(rq, p, flags); if (trace_dequeue_task_p_enabled() && !(flags & DEQUEUE_SLEEP)) __trace_dequeue_task_tp(rq->cpu, p); return ret;
Quoted 8 lines
__block_task():trace_dequeue_task_tp(rq->cpu, p); ...Specifically, only
DEQUEUE_SLEEPis allowed to fail, andDEQUEUE_SLEEPwill eventually cause__block_task()to be called, either directly, or delayed.
Thanks for the suggestion, this makes sense.
From my understanding, it makes the tracepoints work correctly for fair tasks too, so I will get rid of the comment.
Nam
Peter Zijlstra
maintainerRe: [PATCH v2 4/5] sched: Add task enqueue/dequeue trace points
Quoted 43 lines · Nam Cao
Quoted 42 lines · Peter Zijlstra
Quoted 12 lines · Nam Cao
include/trace/events/sched.h +9
-0+/* +* The two trace points below may not work as expected for fair tasks due +* to delayed dequeue. See: +* https://lore.kernel.org/lkml/[email protected]/ +*/ +DECLARE_TRACE (dequeue_task, + TP_PROTO(int cpu, struct task_struct *task), +TP_ARGS (cpu, task)); +include/trace/events/sched.h inline bool dequeue_task(struct rq *rq, struct task_struct *p, int flags) · line 2129
Quoted 10 lines
include/trace/events/sched.h +5
-1@@ -2119,7 +2121,11 @@ inline bool dequeue_task(struct rq *rq, struct task_struct *p, int flags)* and mark the task ->sched_delayed. */ uclamp_rq_dec (rq, p); -return p->sched_class->dequeue_task (rq, p, flags); +if (p->sched_class->dequeue_task (rq, p, flags)) { +trace_dequeue_task_tp (rq->cpu, p); +return true; + } +return false; }Hurmpff.. that's not very nice.
How about something like:
dequeue_task():... ret = p->sched_class->dequeue_task(rq, p, flags); if (trace_dequeue_task_p_enabled() && !(flags & DEQUEUE_SLEEP)) __trace_dequeue_task_tp(rq->cpu, p); return ret;
__block_task():trace_dequeue_task_tp(rq->cpu, p); ...
Quoted 8 lines
Quoted 3 lines
Specifically, only
DEQUEUE_SLEEPis allowed to fail, andDEQUEUE_SLEEPwill eventually cause__block_task()to be called, either directly, or delayed.Thanks for the suggestion, this makes sense.
From my understanding, it makes the tracepoints work correctly for fair tasks too, so I will get rid of the comment.
Just so indeed :-)
PATCH v2 5/5 Add rts monitor
In reply to Nam CaoAdd "real-time scheduling" monitor, which validates that SCHED_RR and SCHED_FIFO tasks are scheduled before tasks with normal and extensible scheduling policies
---
Cc: Ingo Molnar <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Juri Lelli <[email protected]>
Cc: Vincent Guittot <[email protected]>
Cc: Dietmar Eggemann <[email protected]>
Cc: Ben Segall <[email protected]>
Cc: Mel Gorman <[email protected]>
Cc: Valentin Schneider <[email protected]>
Cc: K Prateek Nayak <[email protected]>
---
v2:
- use the new tracepoints
- move to be under the rtapp container monitor
- re-generate with the modified scripts
- fixup incorrect enqueued status
--- Documentation/trace/rv/monitor_sched.rst | 19 +++ kernel/trace/rv/Kconfig | 1 + kernel/trace/rv/Makefile | 1 + kernel/trace/rv/monitors/rts/Kconfig | 17 +++ kernel/trace/rv/monitors/rts/rts.c | 144 +++++++++++++++++++++++ kernel/trace/rv/monitors/rts/rts.h | 126 ++++++++++++++++++++ kernel/trace/rv/monitors/rts/rts_trace.h | 15 +++ kernel/trace/rv/rv_trace.h | 1 + tools/verification/models/sched/rts.ltl | 5 + 9 files changed, 329 insertions(+) create mode 100644 kernel/trace/rv/monitors/rts/Kconfig create mode 100644 kernel/trace/rv/monitors/rts/rts.c create mode 100644 kernel/trace/rv/monitors/rts/rts.h create mode 100644 kernel/trace/rv/monitors/rts/rts_trace.h create mode 100644 tools/verification/models/sched/rts.ltl
Documentation/trace/rv/monitor_sched.rst +19 -0
@@ -396,6 +396,25 @@ preemption is always disabled. On non- ``PREEMPT_RT`` kernels, the interrupts might invoke a softirq to set ``need_resched`` and wake up a task. This is another special case that is currently not supported by the monitor. +Monitor rts +----------- + +The real-time scheduling monitor validates that tasks with real-time scheduling +policies (`SCHED_FIFO` and `SCHED_RR`) are always scheduled before tasks with +normal and extensible scheduling policies (`SCHED_OTHER`, `SCHED_BATCH`, +`SCHED_IDLE`, `SCHED_EXT`): + +.. literalinclude:: ../../../tools/verification/models/sched/rts.ltl + +Note that this monitor may report errors if real-time throttling or fair +deadline server is enabled. These mechanisms prevent real-time tasks from +monopolying the CPU by giving tasks with normal and extensible scheduling +policies a chance to run. They give system administrators a chance to kill a +misbehaved real-time task. However, they violate the scheduling priorities and +may cause latency to well-behaved real-time tasks. Thus, if you see errors from +this monitor, consider disabling real-time throttling and the fair deadline +server. + References ----------
kernel/trace/rv/Kconfig +1 -0
@@ -67,6 +67,7 @@ source "kernel/trace/rv/monitors/opid/Kconfig" source "kernel/trace/rv/monitors/rtapp/Kconfig" source "kernel/trace/rv/monitors/pagefault/Kconfig" source "kernel/trace/rv/monitors/sleep/Kconfig" +source "kernel/trace/rv/monitors/rts/Kconfig" # Add new rtapp monitors here # Add new monitors here
kernel/trace/rv/Makefile +1 -0
@@ -17,6 +17,7 @@ obj-$(CONFIG_RV_MON_STS) += monitors/sts/sts.o obj-$(CONFIG_RV_MON_NRP) += monitors/nrp/nrp.o obj-$(CONFIG_RV_MON_SSSW) += monitors/sssw/sssw.o obj-$(CONFIG_RV_MON_OPID) += monitors/opid/opid.o +obj-$(CONFIG_RV_MON_RTS) += monitors/rts/rts.o # Add new monitors here obj-$(CONFIG_RV_REACTORS) += rv_reactors.o obj-$(CONFIG_RV_REACT_PRINTK) += reactor_printk.o
kernel/trace/rv/monitors/rts/Kconfig new file +17 -0
@@ -0,0 +1,17 @@ +# SPDX-License-Identifier: GPL-2.0-only +# +config RV_MON_RTS + depends on RV + select RV_LTL_MONITOR + depends on RV_MON_RTAPP + default y + select LTL_MON_EVENTS_CPU + bool "rts monitor" + help + Add support for RTS (real-time scheduling) monitor which validates + that real-time-priority tasks are scheduled before SCHED_OTHER tasks. + + This monitor may report an error if RT throttling or deadline server + is enabled. + + Say Y if you are debugging or testing a real-time system.
kernel/trace/rv/monitors/rts/rts.c new file +144 -0
144 lines not on this page · raw message
kernel/trace/rv/monitors/rts/rts.h new file +126 -0
126 lines not on this page · raw message
kernel/trace/rv/monitors/rts/rts_trace.h new file +15 -0
@@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +/* +* Snippet to be included in rv_trace.h +*/ + +#ifdef CONFIG_RV_MON_RTS +DEFINE_EVENT (event_ltl_monitor_cpu, event_rts, + TP_PROTO(unsigned int cpu, char *states, char *atoms, char *next), +TP_ARGS (cpu, states, atoms, next)); + +DEFINE_EVENT (error_ltl_monitor_cpu, error_rts, + TP_PROTO(unsigned int cpu), +TP_ARGS (cpu)); +#endif /* CONFIG_RV_MON_RTS */
kernel/trace/rv/rv_trace.h +1 -0
@@ -221,6 +221,7 @@ DECLARE_EVENT_CLASS(error_ltl_monitor_cpu,TP_printk ("cpu%u: violation detected" , __entry->cpu) ); +#include <monitors/rts/rts_trace.h> // Add new monitors based on CONFIG_LTL_MON_EVENTS_CPU here #endif /* CONFIG_LTL_MON_EVENTS_CPU */
tools/verification/models/sched/rts.ltl new file +5 -0
@@ -0,0 +1,5 @@ +RULE = always (RT_TASK_ENQUEUED imply SCHEDULE_RT_NEXT) + +SCHEDULE_RT_NEXT = (not SCHED_SWITCH) until (SCHED_SWITCH_RT or EXCEPTIONS) + +EXCEPTIONS = SCHED_SWITCH_DL or not RT_TASK_ENQUEUED
Signature · 1 line
2.39.5
kernel test robot
build botRe: [PATCH v2 5/5] rv: Add rts monitor
kernel test robot noticed the following build errors:
Hi Nam,
kernel test robot noticed the following build errors:
[auto build test ERROR on trace/for-next]
[also build test ERROR on linus/master next-20250806] [cannot apply to tip/sched/core v6.16]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Nam-Cao/rv-ltl-Prepare-for-other-monitor-types/20250806-160342 base: https://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace for-next patch link: https://lore.kernel.org/r/88fdbeb3f2ecf3a6259f3ee8636ae5b21fa6b72d.1754466623.git.namcao%40linutronix.de patch subject: [PATCH v2 5/5] rv: Add rts monitor config: sh-allmodconfig (https://download.01.org/0day-ci/archive/20250807/[email protected]/config) compiler: sh4-linux-gcc (GCC) 15.1.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250807/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
In file included from include/asm-generic/percpu.h:7,
from ./arch/sh/include/generated/asm/percpu.h:1, from include/linux/irqflags.h:19, from arch/sh/include/asm/cmpxchg-irq.h:5, from arch/sh/include/asm/cmpxchg.h:21, from arch/sh/include/asm/atomic.h:19, from include/linux/atomic.h:7,
from include/asm-generic/bitops/atomic.h:5, from arch/sh/include/asm/bitops.h:23, from include/linux/bitops.h:67,
from include/linux/kernel.h:23,
from include/linux/interrupt.h:6, from include/linux/trace_recursion.h:5, from include/linux/ftrace.h:10,
from kernel/trace/rv/monitors/rts/rts.c:2:
include/rv/ltl_monitor.h: In function 'ltl_get_monitor':
>> include/linux/percpu-defs.h:90:40: error: section attribute cannot be specified for local variables
90 | extern __PCPU_DUMMY_ATTRS char __pcpu_unique_##name; \
| ^~~~~~~~~~~~~~
include/linux/percpu-defs.h:113:9: note: in expansion of macro 'DEFINE_PER_CPU_SECTION'
113 | DEFINE_PER_CPU_SECTION(type, name, "")
| ^~~~~~~~~~~~~~~~~~~~~~
include/rv/ltl_monitor.h:69:16: note: in expansion of macro 'DEFINE_PER_CPU'
69 | static DEFINE_PER_CPU(struct ltl_monitor, ltl_monitor);
| ^~~~~~~~~~~~~~
include/linux/percpu-defs.h:91:33: error: section attribute cannot be specified for local variables
91 | __PCPU_DUMMY_ATTRS char __pcpu_unique_##name; \
| ^~~~~~~~~~~~~~
include/linux/percpu-defs.h:113:9: note: in expansion of macro 'DEFINE_PER_CPU_SECTION'
113 | DEFINE_PER_CPU_SECTION(type, name, "")
| ^~~~~~~~~~~~~~~~~~~~~~
include/rv/ltl_monitor.h:69:16: note: in expansion of macro 'DEFINE_PER_CPU'
69 | static DEFINE_PER_CPU(struct ltl_monitor, ltl_monitor);
| ^~~~~~~~~~~~~~
>> include/linux/percpu-defs.h:91:33: error: declaration of '__pcpu_unique_ltl_monitor' with no linkage follows extern declaration
91 | __PCPU_DUMMY_ATTRS char __pcpu_unique_##name; \
| ^~~~~~~~~~~~~~
include/linux/percpu-defs.h:113:9: note: in expansion of macro 'DEFINE_PER_CPU_SECTION'
113 | DEFINE_PER_CPU_SECTION(type, name, "")
| ^~~~~~~~~~~~~~~~~~~~~~
include/rv/ltl_monitor.h:69:16: note: in expansion of macro 'DEFINE_PER_CPU'
69 | static DEFINE_PER_CPU(struct ltl_monitor, ltl_monitor);
| ^~~~~~~~~~~~~~
include/linux/percpu-defs.h:90:40: note: previous declaration of '__pcpu_unique_ltl_monitor' with type 'char'
90 | extern __PCPU_DUMMY_ATTRS char __pcpu_unique_##name; \
| ^~~~~~~~~~~~~~
include/linux/percpu-defs.h:113:9: note: in expansion of macro 'DEFINE_PER_CPU_SECTION'
113 | DEFINE_PER_CPU_SECTION(type, name, "")
| ^~~~~~~~~~~~~~~~~~~~~~
include/rv/ltl_monitor.h:69:16: note: in expansion of macro 'DEFINE_PER_CPU'
69 | static DEFINE_PER_CPU(struct ltl_monitor, ltl_monitor);
| ^~~~~~~~~~~~~~
>> include/rv/ltl_monitor.h:69:51: error: section attribute cannot be specified for local variables
69 | static DEFINE_PER_CPU(struct ltl_monitor, ltl_monitor);
| ^~~~~~~~~~~
include/linux/percpu-defs.h:92:51: note: in definition of macro 'DEFINE_PER_CPU_SECTION'
92 | extern __PCPU_ATTRS(sec) __typeof__(type) name; \
| ^~~~
include/rv/ltl_monitor.h:69:16: note: in expansion of macro 'DEFINE_PER_CPU'
69 | static DEFINE_PER_CPU(struct ltl_monitor, ltl_monitor);
| ^~~~~~~~~~~~~~
>> include/rv/ltl_monitor.h:69:51: error: section attribute cannot be specified for local variables
69 | static DEFINE_PER_CPU(struct ltl_monitor, ltl_monitor);
| ^~~~~~~~~~~
include/linux/percpu-defs.h:93:51: note: in definition of macro 'DEFINE_PER_CPU_SECTION'
93 | __PCPU_ATTRS(sec) __weak __typeof__(type) name
| ^~~~
include/rv/ltl_monitor.h:69:16: note: in expansion of macro 'DEFINE_PER_CPU'
69 | static DEFINE_PER_CPU(struct ltl_monitor, ltl_monitor);
| ^~~~~~~~~~~~~~
>> include/rv/ltl_monitor.h:69:51: error: weak declaration of 'ltl_monitor' must be public
69 | static DEFINE_PER_CPU(struct ltl_monitor, ltl_monitor);
| ^~~~~~~~~~~
include/linux/percpu-defs.h:93:51: note: in definition of macro 'DEFINE_PER_CPU_SECTION'
93 | __PCPU_ATTRS(sec) __weak __typeof__(type) name
| ^~~~
include/rv/ltl_monitor.h:69:16: note: in expansion of macro 'DEFINE_PER_CPU'
69 | static DEFINE_PER_CPU(struct ltl_monitor, ltl_monitor);
| ^~~~~~~~~~~~~~
>> include/rv/ltl_monitor.h:69:51: error: declaration of 'ltl_monitor' with no linkage follows extern declaration
69 | static DEFINE_PER_CPU(struct ltl_monitor, ltl_monitor);
| ^~~~~~~~~~~
include/linux/percpu-defs.h:93:51: note: in definition of macro 'DEFINE_PER_CPU_SECTION'
93 | __PCPU_ATTRS(sec) __weak __typeof__(type) name
| ^~~~
include/rv/ltl_monitor.h:69:16: note: in expansion of macro 'DEFINE_PER_CPU'
69 | static DEFINE_PER_CPU(struct ltl_monitor, ltl_monitor);
| ^~~~~~~~~~~~~~
include/rv/ltl_monitor.h:69:51: note: previous declaration of 'ltl_monitor' with type 'struct ltl_monitor'
69 | static DEFINE_PER_CPU(struct ltl_monitor, ltl_monitor);
| ^~~~~~~~~~~
include/linux/percpu-defs.h:92:51: note: in definition of macro 'DEFINE_PER_CPU_SECTION'
92 | extern __PCPU_ATTRS(sec) __typeof__(type) name; \
| ^~~~
include/rv/ltl_monitor.h:69:16: note: in expansion of macro 'DEFINE_PER_CPU'
69 | static DEFINE_PER_CPU(struct ltl_monitor, ltl_monitor);
| ^~~~~~~~~~~~~~vim +69 include/rv/ltl_monitor.h
a9769a5b987838 Nam Cao 2025-07-09 61
59c45d176bc0f2 Nam Cao 2025-08-06 62 static struct ltl_monitor *ltl_get_monitor(monitor_target target)
a9769a5b987838 Nam Cao 2025-07-09 63 {
59c45d176bc0f2 Nam Cao 2025-08-06 64 return &target->rv[ltl_monitor_slot].ltl_mon;
a9769a5b987838 Nam Cao 2025-07-09 65 }
50fd6be3de4982 Nam Cao 2025-08-06 66 #elif LTL_MONITOR_TYPE == RV_MON_PER_CPU
50fd6be3de4982 Nam Cao 2025-08-06 67 static struct ltl_monitor *ltl_get_monitor(unsigned int cpu)
50fd6be3de4982 Nam Cao 2025-08-06 68 {
50fd6be3de4982 Nam Cao 2025-08-06 @69 static DEFINE_PER_CPU(struct ltl_monitor, ltl_monitor);
50fd6be3de4982 Nam Cao 2025-08-06 70
50fd6be3de4982 Nam Cao 2025-08-06 71 return per_cpu_ptr(<l_monitor, cpu);
50fd6be3de4982 Nam Cao 2025-08-06 72 }
59c45d176bc0f2 Nam Cao 2025-08-06 73 #endif
a9769a5b987838 Nam Cao 2025-07-09 74 Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Gabriele Monaco
maintainerRe: [PATCH v2 5/5] rv: Add rts monitor
In reply to Nam CaoQuoted 5 lines · Nam Cao
Add "real-time scheduling" monitor, which validates that
SCHED_RRandSCHED_FIFOtasks are scheduled before tasks with normal and extensible scheduling policiesSigned-off-by Nam Cao <[email protected]>
Quoted 9 lines
---
v2:
- use the new tracepoints
- move to be under the rtapp container monitor
- re-generate with the modified scripts
- fixup incorrect enqueued status
---
Documentation/trace/rv/monitor_sched.rst | 19 +++ tools/verification/models/sched/rts.ltl | 5 +
You moved it under rtapp, you probably want to move the LTL model and the descriptions there too.
Thanks,
Gabriele
Quoted 455 lines
Documentation/trace/rv/monitor_sched.rst +0
-0@@ -396,6 +396,25 @@ preemption is always disabled. On non-``PREEMPT_RT`` kernels, the interrupts
might invoke a softirq to set ``need_resched`` and wake up a task. This is
another special case that is currently not supported by the monitor.Documentation/trace/rv/monitor_sched.rst +4
-0+Monitor rts +----------- + +The real-time scheduling monitor validates that tasks with real-timescheduling
+policies (SCHED_FIFOandSCHED_RR) are always scheduled before tasks with
+normal and extensible scheduling policies (SCHED_OTHER,SCHED_BATCH,
+SCHED_IDLE,SCHED_EXT):Documentation/trace/rv/monitor_sched.rst +4
-0+ +.. literalinclude:: ../../../tools/verification/models/sched/rts.ltl + +Note that this monitor may report errors if real-time throttling orfair
+deadline server is enabled. These mechanisms prevent real-time tasks from
+monopolying the CPU by giving tasks with normal and extensible scheduling
+policies a chance to run. They give system administrators a chance to kill a
+misbehaved real-time task. However, they violate the scheduling priorities and
+may cause latency to well-behaved real-time tasks. Thus, if you see errors from
+this monitor, consider disabling real-time throttling and the fair deadlineDocumentation/trace/rv/monitor_sched.rst +2
-0+server. +References
----------kernel/trace/rv/Kconfig +0
-0@@ -67,6 +67,7 @@ source "kernel/trace/rv/monitors/opid/Kconfig"source "
kernel/trace/rv/monitors/rtapp/Kconfig" source "kernel/trace/rv/monitors/pagefault/Kconfig" source "kernel/trace/rv/monitors/sleep/Kconfig" +source "kernel/trace/rv/monitors/rts/Kconfig" # Add new rtapp monitors here
# Add new monitors herekernel/trace/rv/Makefile +0
-0@@ -17,6 +17,7 @@ obj-$(CONFIG_RV_MON_STS) += monitors/sts/sts.oobj-$(CONFIG_RV_MON_NRP) +=
monitors/nrp/nrp.oobj-$(CONFIG_RV_MON_SSSW) +=monitors/sssw/sssw.oobj-$(CONFIG_RV_MON_OPID) +=monitors/opid/opid.o+obj-$(CONFIG_RV_MON_RTS) +=monitors/rts/rts.o# Add new monitors here obj-$(CONFIG_RV_REACTORS) += rv_reactors.o obj-$(CONFIG_RV_REACT_PRINTK) += reactor_printk.okernel/trace/rv/monitors/rts/Kconfig new file +11
-0@@ -0,0 +1,17 @@ +# SPDX-License-Identifier: GPL-2.0-only +# +config RV_MON_RTS + depends on RV + select RV_LTL_MONITOR + depends on RV_MON_RTAPP + default y + select LTL_MON_EVENTS_CPU + bool "rts monitor" + help + Add support for RTS (real-time scheduling) monitor whichvalidates
kernel/trace/rv/monitors/rts/Kconfig new file +1
-0+ that real-time-priority tasks are scheduled before
SCHED_OTHERtasks.kernel/trace/rv/monitors/rts/Kconfig new file +2
-0+ + This monitor may report an error if RT throttling ordeadline server
kernel/trace/rv/monitors/rts/Kconfig new file +3
-0+ is enabled. + + Say Y if you are debugging or testing a real-time system.kernel/trace/rv/monitors/rts/rts.c new file +23
-0@@ -0,0 +1,144 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/ftrace.h> +#include <linux/tracepoint.h> +#include <linux/kernel.h> +#include <linux/module.h> +#include <linux/init.h> +#include <linux/rv.h> +#include <linux/sched/deadline.h> +#include <linux/sched/rt.h> +#include <rv/instrumentation.h> + +#define MODULE_NAME "rts" + +#include <trace/events/sched.h> +#include <rv_trace.h> +#include <monitors/rtapp/rtapp.h> + +#include "rts.h" +#include <rv/ltl_monitor.h> + +static DEFINE_PER_CPU(unsigned int, nr_queued); + +static void ltl_atoms_fetch(unsigned int cpu, struct ltl_monitor*mon)
kernel/trace/rv/monitors/rts/rts.c new file +4
-0+{ +} + +static void ltl_atoms_init(unsigned int cpu, struct ltl_monitor*mon,
kernel/trace/rv/monitors/rts/rts.c new file +8
-0+ bool target_creation) +{ +ltl_atom_set (mon,LTL_SCHED_SWITCH , false); +ltl_atom_set (mon,LTL_SCHED_SWITCH_DL , false); +ltl_atom_set (mon,LTL_SCHED_SWITCH_RT , false); + +/* +* This may not be accurate, there may be enqueued RT tasks. But that's
kernel/trace/rv/monitors/rts/rts.c new file +1
-0+* okay, the worst we get is a false negative. It will be accurate as
kernel/trace/rv/monitors/rts/rts.c new file +6
-0+* soon as the CPU no longer has any queued RT task. +*/ +ltl_atom_set (mon,LTL_RT_TASK_ENQUEUED , false); +} + +static void handle_enqueue_task(void *data, int cpu, struct
task_struct*task)kernel/trace/rv/monitors/rts/rts.c new file +11
-0+{ +unsigned int *queued =per_cpu_ptr (&nr_queued, cpu); + +if (!rt_task (task)) +return ; + + (*queued)++; +ltl_atom_update (cpu,LTL_RT_TASK_ENQUEUED , true); +} + +static void handle_dequeue_task(void *data, int cpu, struct
task_struct*task)kernel/trace/rv/monitors/rts/rts.c new file +8
-0+{ +unsigned int *queued =per_cpu_ptr (&nr_queued, cpu); + +if (!rt_task (task)) +return ; + +/* +* This may not be accurate for a short time after the monitor is
kernel/trace/rv/monitors/rts/rts.c new file +1
-0+* enabled, because there may be enqueued RT tasks which are not counted
kernel/trace/rv/monitors/rts/rts.c new file +1
-0+* torward nr_queued. But that's okay, the worst we get is a false
kernel/trace/rv/monitors/rts/rts.c new file +1
-0+* negative. nr_queued will be accurate as soon as the CPU no longer has
kernel/trace/rv/monitors/rts/rts.c new file +9
-0+* any queued RT task. +*/ +if (*queued) + (*queued)--; +if (!*queued) +ltl_atom_update (cpu,LTL_RT_TASK_ENQUEUED , false); +} + +static void handle_sched_switch(void *data, bool preempt, struct
task_struct*prev,kernel/trace/rv/monitors/rts/rts.c new file +1
-0+struct task_struct *next, unsignedint
prev_state)kernel/trace/rv/monitors/rts/rts.c new file +29
-0+{ +unsigned int cpu =smp_processor_id (); +struct ltl_monitor *mon =ltl_get_monitor (cpu); + +ltl_atom_set (mon,LTL_SCHED_SWITCH_RT ,rt_task (next)); +ltl_atom_set (mon,LTL_SCHED_SWITCH_DL ,dl_task (next)); +ltl_atom_update (cpu,LTL_SCHED_SWITCH , true); + +ltl_atom_set (mon,LTL_SCHED_SWITCH_RT , false); +ltl_atom_set (mon,LTL_SCHED_SWITCH_DL , false); +ltl_atom_update (cpu,LTL_SCHED_SWITCH , false); +} + +static int enable_rts (void ) +{ +unsigned int cpu; +int retval; + + retval =ltl_monitor_init (); +if (retval) +return retval; + +for_each_possible_cpu (cpu) { +unsigned int *queued =per_cpu_ptr (&nr_queued, cpu); + + *queued =0 ; + } + + rv_attach_trace_probe("rts", dequeue_task_tp,
handle_dequeue_task);kernel/trace/rv/monitors/rts/rts.c new file +1
-0+ rv_attach_trace_probe("rts", enqueue_task_tp,
handle_enqueue_task);kernel/trace/rv/monitors/rts/rts.c new file +1
-0+ rv_attach_trace_probe("rts", sched_switch,
handle_sched_switch);kernel/trace/rv/monitors/rts/rts.c new file +7
-0+ +return 0 ; +} + +static void disable_rts (void ) +{ + rv_detach_trace_probe("rts", sched_switch,
handle_sched_switch);kernel/trace/rv/monitors/rts/rts.c new file +1
-0+ rv_detach_trace_probe("rts", enqueue_task_tp,
handle_enqueue_task);kernel/trace/rv/monitors/rts/rts.c new file +1
-0+ rv_detach_trace_probe("rts", dequeue_task_tp,
handle_dequeue_task);kernel/trace/rv/monitors/rts/rts.c new file +10
-0+ +ltl_monitor_destroy (); +} + +/* +* This is the monitor register section. +*/ +static struct rv_monitor rv_rts = { + .name ="rts" , + .description = "Validate that real-time tasks are scheduledbefore lower-priority tasks",
kernel/trace/rv/monitors/rts/rts.c new file +20
-0+ .enable = enable_rts, + .disable = disable_rts, +}; + +static int __init register_rts (void ) +{ +return rv_register_monitor (&rv_rts, &rv_rtapp); +} + +static void __exit unregister_rts(void) +{ +rv_unregister_monitor (&rv_rts); +} + +module_init (register_rts); +module_exit (unregister_rts); + +MODULE_LICENSE ("GPL" ); +MODULE_AUTHOR ("Nam Cao <[email protected]>" ); +MODULE_DESCRIPTION("rts: Validate that real-time tasks are scheduledbefore lower-priority tasks");
kernel/trace/rv/monitors/rts/rts.h new file +5
-0@@ -0,0 +1,126 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +/* +* C implementation of Buchi automaton, automatically generated by +* tools/verification/rvgen from the linear temporal logic specification.
+ * For further information, see kernel documentation:
+ *Documentation/trace/rv/linear_temporal_logic.rst
+ */kernel/trace/rv/monitors/rts/rts.h new file +41
-0+ +#include <linux/rv.h> + +#define MONITOR_NAME rts + +#define LTL_MONITOR_TYPE RV_MON_PER_CPU + +enum ltl_atom { +LTL_RT_TASK_ENQUEUED , +LTL_SCHED_SWITCH , +LTL_SCHED_SWITCH_DL , +LTL_SCHED_SWITCH_RT , +LTL_NUM_ATOM +}; +static_assert (LTL_NUM_ATOM <=RV_MAX_LTL_ATOM ); + +static const char *ltl_atom_str (enum ltl_atom atom) +{ +static const char *const names[] = { +"rt_ta_en" , +"sc_sw" , +"sc_sw_dl" , +"sc_sw_rt" , + }; + +return names[atom]; +} + +enum ltl_buchi_state { +S0 , +S1 , +S2 , +S3 , +S4 , +RV_NUM_BA_STATES +}; +static_assert (RV_NUM_BA_STATES <=RV_MAX_BA_STATES ); + +static void ltl_start (unsigned int cpu,struct ltl_monitor *mon) +{ + bool sched_switch_rt = test_bit(LTL_SCHED_SWITCH_RT, mon-Quoted 1 line
atoms);
kernel/trace/rv/monitors/rts/rts.h new file +1
-0+ bool sched_switch_dl = test_bit(LTL_SCHED_SWITCH_DL, mon-Quoted 1 line
atoms);
kernel/trace/rv/monitors/rts/rts.h new file +2
-0+bool sched_switch =test_bit (LTL_SCHED_SWITCH , mon->atoms); + bool rt_task_enqueued = test_bit(LTL_RT_TASK_ENQUEUED, mon-Quoted 1 line
atoms);
kernel/trace/rv/monitors/rts/rts.h new file +16
-0+bool val13 = !rt_task_enqueued; +bool val8 = sched_switch_dl || val13; +bool val9 = sched_switch_rt || val8; +bool val6 = !sched_switch; +bool val1 = !rt_task_enqueued; + +if (val1) +__set_bit (S0 , mon->states); +if (val6) +__set_bit (S1 , mon->states); +if (val9) +__set_bit (S4 , mon->states); +} + +static void +ltl_possible_next_states(struct ltl_monitor *mon, unsigned intstate, unsigned long *next)
kernel/trace/rv/monitors/rts/rts.h new file +2
-0+{ + bool sched_switch_rt = test_bit(LTL_SCHED_SWITCH_RT, mon-Quoted 1 line
atoms);
kernel/trace/rv/monitors/rts/rts.h new file +1
-0+ bool sched_switch_dl = test_bit(LTL_SCHED_SWITCH_DL, mon-Quoted 1 line
atoms);
kernel/trace/rv/monitors/rts/rts.h new file +2
-0+bool sched_switch =test_bit (LTL_SCHED_SWITCH , mon->atoms); + bool rt_task_enqueued = test_bit(LTL_RT_TASK_ENQUEUED, mon-Quoted 1 line
atoms);
kernel/trace/rv/monitors/rts/rts.h new file +53
-0+bool val13 = !rt_task_enqueued; +bool val8 = sched_switch_dl || val13; +bool val9 = sched_switch_rt || val8; +bool val6 = !sched_switch; +bool val1 = !rt_task_enqueued; + +switch (state) { +case S0 : +if (val1) +__set_bit (S0 , next); +if (val6) +__set_bit (S1 , next); +if (val9) +__set_bit (S4 , next); +break ; +case S1 : +if (val6) +__set_bit (S1 , next); +if (val1 && val6) +__set_bit (S2 , next); +if (val1 && val9) +__set_bit (S3 , next); +if (val9) +__set_bit (S4 , next); +break ; +case S2 : +if (val6) +__set_bit (S1 , next); +if (val1 && val6) +__set_bit (S2 , next); +if (val1 && val9) +__set_bit (S3 , next); +if (val9) +__set_bit (S4 , next); +break ; +case S3 : +if (val1) +__set_bit (S0 , next); +if (val6) +__set_bit (S1 , next); +if (val9) +__set_bit (S4 , next); +break ; +case S4 : +if (val1) +__set_bit (S0 , next); +if (val6) +__set_bit (S1 , next); +if (val9) +__set_bit (S4 , next); +break ; + } +}kernel/trace/rv/monitors/rts/rts_trace.h new file +9
-0@@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +/* +* Snippet to be included in rv_trace.h +*/ + +#ifdef CONFIG_RV_MON_RTS +DEFINE_EVENT(event_ltl_monitor_cpu, event_rts, + TP_PROTO(unsigned int cpu, char *states, char *atoms, char*next),
kernel/trace/rv/monitors/rts/rts_trace.h new file +6
-0+ TP_ARGS(cpu, states, atoms, next)); + +DEFINE_EVENT (error_ltl_monitor_cpu, error_rts, + TP_PROTO(unsigned int cpu), +TP_ARGS (cpu)); +#endif /* CONFIG_RV_MON_RTS */ kernel/trace/rv/rv_trace.h +0
-0@@ -221,6 +221,7 @@ DECLARE_EVENT_CLASS(error_ltl_monitor_cpu,
TP_printk("cpu%u: violation detected", __entry->cpu));
+#include <monitors/rts/rts_trace.h>
// Add new monitors based onCONFIG_LTL_MON_EVENTS_CPUhere
#endif /*CONFIG_LTL_MON_EVENTS_CPU*/tools/verification/models/sched/rts.ltl new file +3
-0@@ -0,0 +1,5 @@ +RULE = always (RT_TASK_ENQUEUED imply SCHEDULE_RT_NEXT) + +SCHEDULE_RT_NEXT = (not SCHED_SWITCH) until (SCHED_SWITCH_RT orEXCEPTIONS)
tools/verification/models/sched/rts.ltl new file +2
-0+ +EXCEPTIONS = SCHED_SWITCH_DL or not RT_TASK_ENQUEUED
Re: [PATCH v2 5/5] rv: Add rts monitor
Quoted 16 lines · Gabriele Monaco
Quoted 5 lines · Nam Cao
Add "real-time scheduling" monitor, which validates that
SCHED_RRandSCHED_FIFOtasks are scheduled before tasks with normal and extensible scheduling policiesSigned-off-by Nam Cao <[email protected]>Quoted 9 lines
---
v2:
- use the new tracepoints
- move to be under the rtapp container monitor
- re-generate with the modified scripts
- fixup incorrect enqueued status
---
Documentation/trace/rv/monitor_sched.rst | 19 +++ tools/verification/models/sched/rts.ltl | 5 +
Quoted 2 lines
You moved it under rtapp, you probably want to move the LTL model and the descriptions there too.
Ah, missed this one, thanks for pointing it out.
Nam
Gabriele Monaco
maintainerRe: [PATCH v2 5/5] rv: Add rts monitor
In reply to Nam CaoQuoted 6 lines · Nam Cao
Add "real-time scheduling" monitor, which validates that
SCHED_RRandSCHED_FIFOtasks are scheduled before tasks with normal and extensible scheduling policiesSigned-off-by Nam Cao <[email protected]>---
The monitor shows a violation also in case of priority inversion boosting, e.g.:
stress-ng --prio-inv 2
It seems perfectly reasonable from the monitor description but it's actually a behaviour meant to improve real time response. Is the user seeing this type of violation supposed to make sure all locks held by RT tasks are never shared by fair tasks? If that's the case I'd mention it in the description.
Also very rarely I see failures while cleaning up the monitor, not sure exactly what caused it but I could reproduce it with something like:
for i in $(seq 100); do timeout -s INT 2 rv mon rts -r printk; done
Running the monitor without stopping for the same amount of time doesn't seem to show violations (until I terminate it).
"rv" here is the tool under tools/verifications/rv, also the rv package on fedora works, but debian/ubuntu may still be shipping an outdated version, if they ship one at all.
Thanks,
Gabriele
Quoted 487 lines
Cc Ingo Molnar <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Juri Lelli <[email protected]>Cc Vincent Guittot <[email protected]>Cc Dietmar Eggemann <[email protected]>Cc Ben Segall <[email protected]>Cc Mel Gorman <[email protected]>Cc Valentin Schneider <[email protected]>Cc K Prateek Nayak <[email protected]>---
v2:
- use the new tracepoints
- move to be under the rtapp container monitor
- re-generate with the modified scripts
- fixup incorrect enqueued status
---
Documentation/trace/rv/monitor_sched.rst | 19 +++ kernel/trace/rv/Kconfig | 1 + kernel/trace/rv/Makefile | 1 + kernel/trace/rv/monitors/rts/Kconfig | 17 +++ kernel/trace/rv/monitors/rts/rts.c | 144
+++++++++++++++++++++++
kernel/trace/rv/monitors/rts/rts.h | 126 ++++++++++++++++++++kernel/trace/rv/monitors/rts/rts_trace.h| 15 +++ kernel/trace/rv/rv_trace.h | 1 + tools/verification/models/sched/rts.ltl | 5 +9 files changed, 329 insertions(+)create mode 100644
kernel/trace/rv/monitors/rts/Kconfigcreate mode 100644kernel/trace/rv/monitors/rts/rts.ccreate mode 100644kernel/trace/rv/monitors/rts/rts.hcreate mode 100644kernel/trace/rv/monitors/rts/rts_trace.hcreate mode 100644tools/verification/models/sched/rts.ltlDocumentation/trace/rv/monitor_sched.rst +0
-0@@ -396,6 +396,25 @@ preemption is always disabled. On non-``PREEMPT_RT`` kernels, the interrupts
might invoke a softirq to set ``need_resched`` and wake up a task. This is
another special case that is currently not supported by the monitor.Documentation/trace/rv/monitor_sched.rst +4
-0+Monitor rts +----------- + +The real-time scheduling monitor validates that tasks with real-timescheduling
+policies (SCHED_FIFOandSCHED_RR) are always scheduled before tasks with
+normal and extensible scheduling policies (SCHED_OTHER,SCHED_BATCH,
+SCHED_IDLE,SCHED_EXT):Documentation/trace/rv/monitor_sched.rst +4
-0+ +.. literalinclude:: ../../../tools/verification/models/sched/rts.ltl + +Note that this monitor may report errors if real-time throttling orfair
+deadline server is enabled. These mechanisms prevent real-time tasks from
+monopolying the CPU by giving tasks with normal and extensible scheduling
+policies a chance to run. They give system administrators a chance to kill a
+misbehaved real-time task. However, they violate the scheduling priorities and
+may cause latency to well-behaved real-time tasks. Thus, if you see errors from
+this monitor, consider disabling real-time throttling and the fair deadlineDocumentation/trace/rv/monitor_sched.rst +2
-0+server. +References
----------kernel/trace/rv/Kconfig +0
-0@@ -67,6 +67,7 @@ source "kernel/trace/rv/monitors/opid/Kconfig"source "
kernel/trace/rv/monitors/rtapp/Kconfig" source "kernel/trace/rv/monitors/pagefault/Kconfig" source "kernel/trace/rv/monitors/sleep/Kconfig" +source "kernel/trace/rv/monitors/rts/Kconfig" # Add new rtapp monitors here
# Add new monitors herekernel/trace/rv/Makefile +0
-0@@ -17,6 +17,7 @@ obj-$(CONFIG_RV_MON_STS) += monitors/sts/sts.oobj-$(CONFIG_RV_MON_NRP) +=
monitors/nrp/nrp.oobj-$(CONFIG_RV_MON_SSSW) +=monitors/sssw/sssw.oobj-$(CONFIG_RV_MON_OPID) +=monitors/opid/opid.o+obj-$(CONFIG_RV_MON_RTS) +=monitors/rts/rts.o# Add new monitors here obj-$(CONFIG_RV_REACTORS) += rv_reactors.o obj-$(CONFIG_RV_REACT_PRINTK) += reactor_printk.okernel/trace/rv/monitors/rts/Kconfig new file +11
-0@@ -0,0 +1,17 @@ +# SPDX-License-Identifier: GPL-2.0-only +# +config RV_MON_RTS + depends on RV + select RV_LTL_MONITOR + depends on RV_MON_RTAPP + default y + select LTL_MON_EVENTS_CPU + bool "rts monitor" + help + Add support for RTS (real-time scheduling) monitor whichvalidates
kernel/trace/rv/monitors/rts/Kconfig new file +1
-0+ that real-time-priority tasks are scheduled before
SCHED_OTHERtasks.kernel/trace/rv/monitors/rts/Kconfig new file +2
-0+ + This monitor may report an error if RT throttling ordeadline server
kernel/trace/rv/monitors/rts/Kconfig new file +3
-0+ is enabled. + + Say Y if you are debugging or testing a real-time system.kernel/trace/rv/monitors/rts/rts.c new file +23
-0@@ -0,0 +1,144 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/ftrace.h> +#include <linux/tracepoint.h> +#include <linux/kernel.h> +#include <linux/module.h> +#include <linux/init.h> +#include <linux/rv.h> +#include <linux/sched/deadline.h> +#include <linux/sched/rt.h> +#include <rv/instrumentation.h> + +#define MODULE_NAME "rts" + +#include <trace/events/sched.h> +#include <rv_trace.h> +#include <monitors/rtapp/rtapp.h> + +#include "rts.h" +#include <rv/ltl_monitor.h> + +static DEFINE_PER_CPU(unsigned int, nr_queued); + +static void ltl_atoms_fetch(unsigned int cpu, struct ltl_monitor*mon)
kernel/trace/rv/monitors/rts/rts.c new file +4
-0+{ +} + +static void ltl_atoms_init(unsigned int cpu, struct ltl_monitor*mon,
kernel/trace/rv/monitors/rts/rts.c new file +8
-0+ bool target_creation) +{ +ltl_atom_set (mon,LTL_SCHED_SWITCH , false); +ltl_atom_set (mon,LTL_SCHED_SWITCH_DL , false); +ltl_atom_set (mon,LTL_SCHED_SWITCH_RT , false); + +/* +* This may not be accurate, there may be enqueued RT tasks. But that's
kernel/trace/rv/monitors/rts/rts.c new file +1
-0+* okay, the worst we get is a false negative. It will be accurate as
kernel/trace/rv/monitors/rts/rts.c new file +6
-0+* soon as the CPU no longer has any queued RT task. +*/ +ltl_atom_set (mon,LTL_RT_TASK_ENQUEUED , false); +} + +static void handle_enqueue_task(void *data, int cpu, struct
task_struct*task)kernel/trace/rv/monitors/rts/rts.c new file +11
-0+{ +unsigned int *queued =per_cpu_ptr (&nr_queued, cpu); + +if (!rt_task (task)) +return ; + + (*queued)++; +ltl_atom_update (cpu,LTL_RT_TASK_ENQUEUED , true); +} + +static void handle_dequeue_task(void *data, int cpu, struct
task_struct*task)kernel/trace/rv/monitors/rts/rts.c new file +8
-0+{ +unsigned int *queued =per_cpu_ptr (&nr_queued, cpu); + +if (!rt_task (task)) +return ; + +/* +* This may not be accurate for a short time after the monitor is
kernel/trace/rv/monitors/rts/rts.c new file +1
-0+* enabled, because there may be enqueued RT tasks which are not counted
kernel/trace/rv/monitors/rts/rts.c new file +1
-0+* torward nr_queued. But that's okay, the worst we get is a false
kernel/trace/rv/monitors/rts/rts.c new file +1
-0+* negative. nr_queued will be accurate as soon as the CPU no longer has
kernel/trace/rv/monitors/rts/rts.c new file +9
-0+* any queued RT task. +*/ +if (*queued) + (*queued)--; +if (!*queued) +ltl_atom_update (cpu,LTL_RT_TASK_ENQUEUED , false); +} + +static void handle_sched_switch(void *data, bool preempt, struct
task_struct*prev,kernel/trace/rv/monitors/rts/rts.c new file +1
-0+struct task_struct *next, unsignedint
prev_state)kernel/trace/rv/monitors/rts/rts.c new file +29
-0+{ +unsigned int cpu =smp_processor_id (); +struct ltl_monitor *mon =ltl_get_monitor (cpu); + +ltl_atom_set (mon,LTL_SCHED_SWITCH_RT ,rt_task (next)); +ltl_atom_set (mon,LTL_SCHED_SWITCH_DL ,dl_task (next)); +ltl_atom_update (cpu,LTL_SCHED_SWITCH , true); + +ltl_atom_set (mon,LTL_SCHED_SWITCH_RT , false); +ltl_atom_set (mon,LTL_SCHED_SWITCH_DL , false); +ltl_atom_update (cpu,LTL_SCHED_SWITCH , false); +} + +static int enable_rts (void ) +{ +unsigned int cpu; +int retval; + + retval =ltl_monitor_init (); +if (retval) +return retval; + +for_each_possible_cpu (cpu) { +unsigned int *queued =per_cpu_ptr (&nr_queued, cpu); + + *queued =0 ; + } + + rv_attach_trace_probe("rts", dequeue_task_tp,
handle_dequeue_task);kernel/trace/rv/monitors/rts/rts.c new file +1
-0+ rv_attach_trace_probe("rts", enqueue_task_tp,
handle_enqueue_task);kernel/trace/rv/monitors/rts/rts.c new file +1
-0+ rv_attach_trace_probe("rts", sched_switch,
handle_sched_switch);kernel/trace/rv/monitors/rts/rts.c new file +7
-0+ +return 0 ; +} + +static void disable_rts (void ) +{ + rv_detach_trace_probe("rts", sched_switch,
handle_sched_switch);kernel/trace/rv/monitors/rts/rts.c new file +1
-0+ rv_detach_trace_probe("rts", enqueue_task_tp,
handle_enqueue_task);kernel/trace/rv/monitors/rts/rts.c new file +1
-0+ rv_detach_trace_probe("rts", dequeue_task_tp,
handle_dequeue_task);kernel/trace/rv/monitors/rts/rts.c new file +10
-0+ +ltl_monitor_destroy (); +} + +/* +* This is the monitor register section. +*/ +static struct rv_monitor rv_rts = { + .name ="rts" , + .description = "Validate that real-time tasks are scheduledbefore lower-priority tasks",
kernel/trace/rv/monitors/rts/rts.c new file +20
-0+ .enable = enable_rts, + .disable = disable_rts, +}; + +static int __init register_rts (void ) +{ +return rv_register_monitor (&rv_rts, &rv_rtapp); +} + +static void __exit unregister_rts(void) +{ +rv_unregister_monitor (&rv_rts); +} + +module_init (register_rts); +module_exit (unregister_rts); + +MODULE_LICENSE ("GPL" ); +MODULE_AUTHOR ("Nam Cao <[email protected]>" ); +MODULE_DESCRIPTION("rts: Validate that real-time tasks are scheduledbefore lower-priority tasks");
kernel/trace/rv/monitors/rts/rts.h new file +5
-0@@ -0,0 +1,126 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +/* +* C implementation of Buchi automaton, automatically generated by +* tools/verification/rvgen from the linear temporal logic specification.
+ * For further information, see kernel documentation:
+ *Documentation/trace/rv/linear_temporal_logic.rst
+ */kernel/trace/rv/monitors/rts/rts.h new file +41
-0+ +#include <linux/rv.h> + +#define MONITOR_NAME rts + +#define LTL_MONITOR_TYPE RV_MON_PER_CPU + +enum ltl_atom { +LTL_RT_TASK_ENQUEUED , +LTL_SCHED_SWITCH , +LTL_SCHED_SWITCH_DL , +LTL_SCHED_SWITCH_RT , +LTL_NUM_ATOM +}; +static_assert (LTL_NUM_ATOM <=RV_MAX_LTL_ATOM ); + +static const char *ltl_atom_str (enum ltl_atom atom) +{ +static const char *const names[] = { +"rt_ta_en" , +"sc_sw" , +"sc_sw_dl" , +"sc_sw_rt" , + }; + +return names[atom]; +} + +enum ltl_buchi_state { +S0 , +S1 , +S2 , +S3 , +S4 , +RV_NUM_BA_STATES +}; +static_assert (RV_NUM_BA_STATES <=RV_MAX_BA_STATES ); + +static void ltl_start (unsigned int cpu,struct ltl_monitor *mon) +{ + bool sched_switch_rt = test_bit(LTL_SCHED_SWITCH_RT, mon-Quoted 1 line
atoms);
kernel/trace/rv/monitors/rts/rts.h new file +1
-0+ bool sched_switch_dl = test_bit(LTL_SCHED_SWITCH_DL, mon-Quoted 1 line
atoms);
kernel/trace/rv/monitors/rts/rts.h new file +2
-0+bool sched_switch =test_bit (LTL_SCHED_SWITCH , mon->atoms); + bool rt_task_enqueued = test_bit(LTL_RT_TASK_ENQUEUED, mon-Quoted 1 line
atoms);
kernel/trace/rv/monitors/rts/rts.h new file +16
-0+bool val13 = !rt_task_enqueued; +bool val8 = sched_switch_dl || val13; +bool val9 = sched_switch_rt || val8; +bool val6 = !sched_switch; +bool val1 = !rt_task_enqueued; + +if (val1) +__set_bit (S0 , mon->states); +if (val6) +__set_bit (S1 , mon->states); +if (val9) +__set_bit (S4 , mon->states); +} + +static void +ltl_possible_next_states(struct ltl_monitor *mon, unsigned intstate, unsigned long *next)
kernel/trace/rv/monitors/rts/rts.h new file +2
-0+{ + bool sched_switch_rt = test_bit(LTL_SCHED_SWITCH_RT, mon-Quoted 1 line
atoms);
kernel/trace/rv/monitors/rts/rts.h new file +1
-0+ bool sched_switch_dl = test_bit(LTL_SCHED_SWITCH_DL, mon-Quoted 1 line
atoms);
kernel/trace/rv/monitors/rts/rts.h new file +2
-0+bool sched_switch =test_bit (LTL_SCHED_SWITCH , mon->atoms); + bool rt_task_enqueued = test_bit(LTL_RT_TASK_ENQUEUED, mon-Quoted 1 line
atoms);
kernel/trace/rv/monitors/rts/rts.h new file +53
-0+bool val13 = !rt_task_enqueued; +bool val8 = sched_switch_dl || val13; +bool val9 = sched_switch_rt || val8; +bool val6 = !sched_switch; +bool val1 = !rt_task_enqueued; + +switch (state) { +case S0 : +if (val1) +__set_bit (S0 , next); +if (val6) +__set_bit (S1 , next); +if (val9) +__set_bit (S4 , next); +break ; +case S1 : +if (val6) +__set_bit (S1 , next); +if (val1 && val6) +__set_bit (S2 , next); +if (val1 && val9) +__set_bit (S3 , next); +if (val9) +__set_bit (S4 , next); +break ; +case S2 : +if (val6) +__set_bit (S1 , next); +if (val1 && val6) +__set_bit (S2 , next); +if (val1 && val9) +__set_bit (S3 , next); +if (val9) +__set_bit (S4 , next); +break ; +case S3 : +if (val1) +__set_bit (S0 , next); +if (val6) +__set_bit (S1 , next); +if (val9) +__set_bit (S4 , next); +break ; +case S4 : +if (val1) +__set_bit (S0 , next); +if (val6) +__set_bit (S1 , next); +if (val9) +__set_bit (S4 , next); +break ; + } +}kernel/trace/rv/monitors/rts/rts_trace.h new file +9
-0@@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +/* +* Snippet to be included in rv_trace.h +*/ + +#ifdef CONFIG_RV_MON_RTS +DEFINE_EVENT(event_ltl_monitor_cpu, event_rts, + TP_PROTO(unsigned int cpu, char *states, char *atoms, char*next),
kernel/trace/rv/monitors/rts/rts_trace.h new file +6
-0+ TP_ARGS(cpu, states, atoms, next)); + +DEFINE_EVENT (error_ltl_monitor_cpu, error_rts, + TP_PROTO(unsigned int cpu), +TP_ARGS (cpu)); +#endif /* CONFIG_RV_MON_RTS */ kernel/trace/rv/rv_trace.h +0
-0@@ -221,6 +221,7 @@ DECLARE_EVENT_CLASS(error_ltl_monitor_cpu,
TP_printk("cpu%u: violation detected", __entry->cpu));
+#include <monitors/rts/rts_trace.h>
// Add new monitors based onCONFIG_LTL_MON_EVENTS_CPUhere
#endif /*CONFIG_LTL_MON_EVENTS_CPU*/tools/verification/models/sched/rts.ltl new file +3
-0@@ -0,0 +1,5 @@ +RULE = always (RT_TASK_ENQUEUED imply SCHEDULE_RT_NEXT) + +SCHEDULE_RT_NEXT = (not SCHED_SWITCH) until (SCHED_SWITCH_RT orEXCEPTIONS)
tools/verification/models/sched/rts.ltl new file +2
-0+ +EXCEPTIONS = SCHED_SWITCH_DL or not RT_TASK_ENQUEUED
Re: [PATCH v2 5/5] rv: Add rts monitor
Quoted 5 lines · Gabriele Monaco
Quoted 4 lines · Nam Cao
Add "real-time scheduling" monitor, which validates that
SCHED_RRandSCHED_FIFOtasks are scheduled before tasks with normal and extensible scheduling policies
Quoted 7 lines
Quoted 2 lines
Signed-off-by Nam Cao <[email protected]>---
The monitor shows a violation also in case of priority inversion boosting, e.g.:
stress-ng --prio-inv 2
This looks like something that would trigger the fair deadline server or RT throttling. Can you please try disabling both of them:
echo 0 | tee /sys/kernel/debug/sched/fair_server/cpu*/runtime sysctl -w kernel.sched_rt_runtime_us=-1
and see if the problem persists?
Quoted 5 lines
It seems perfectly reasonable from the monitor description but it's actually a behaviour meant to improve real time response. Is the user seeing this type of violation supposed to make sure all locks held by RT tasks are never shared by fair tasks? If that's the case I'd mention it in the description.
Boosted fair tasks are treated as RT tasks ;)
Quoted 11 lines
Also very rarely I see failures while cleaning up the monitor, not sure exactly what caused it but I could reproduce it with something like:
for i in $(seq 100); do timeout -s INT 2 rv mon rts -r printk; doneRunning the monitor without stopping for the same amount of time doesn't seem to show violations (until I terminate it).
"rv" here is the tool under
tools/verifications/rv, also the rv package on fedora works, but debian/ubuntu may still be shipping an outdated version, if they ship one at all.
This one is strange, I cannot reproduce this issue. Did you run only that command, or did you have other things running as well?
And does the problem still appears after disabling the fair deadline server and RT throttling?
Thanks for trying it out.
nam
Re: [PATCH v2 5/5] rv: Add rts monitor
Quoted 6 lines · Nam Cao
Quoted 4 lines · Gabriele Monaco
The monitor shows a violation also in case of priority inversion boosting, e.g.:
stress-ng --prio-inv 2
Quoted 7 lines
This looks like something that would trigger the fair deadline server or RT throttling. Can you please try disabling both of them:
echo 0 | tee
/sys/kernel/debug/sched/fair_server/cpu*/runtimesysctl -w kernel.sched_rt_runtime_us=-1and see if the problem persists?
My bad, that was the fair server's doing, ignore what I said.
Quoted 16 lines
Quoted 6 lines
It seems perfectly reasonable from the monitor description but it's actually a behaviour meant to improve real time response. Is the user seeing this type of violation supposed to make sure all locks held by RT tasks are never shared by fair tasks? If that's the case I'd mention it in the description.
Boosted fair tasks are treated as RT tasks ;)
Quoted 7 lines
Also very rarely I see failures while cleaning up the monitor, not sure exactly what caused it but I could reproduce it with something like:
for i in $(seq 100); do timeout -s INT 2 rv mon rts -r printk; done
Quoted 8 lines
Quoted 2 lines
Running the monitor without stopping for the same amount of time doesn't seem to show violations (until I terminate it).
This one is strange, I cannot reproduce this issue. Did you run only that command, or did you have other things running as well?
And does the problem still appears after disabling the fair deadline server and RT throttling?
Also here, I don't seem to reproduce it with both disabled.. Sorry for that, looks good for me then.
Thanks,
Gabriele
Peter Zijlstra
maintainerRe: [PATCH v2 5/5] rv: Add rts monitor
In reply to Nam CaoQuoted 3 lines · Nam Cao
Add "real-time scheduling" monitor, which validates that
SCHED_RRandSCHED_FIFOtasks are scheduled before tasks with normal and extensible scheduling policies
The actual monitor seems to know about deadline too. Surely changelog and document need updating?
While there, you should probably extend to cover at least STOP class, but possibly also IDLE class, no? Validating: IDLE < EXT < FAIR < FIFO/RR < DL < STOP.
Re: [PATCH v2 5/5] rv: Add rts monitor
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Nam Cao
Add "real-time scheduling" monitor, which validates that
SCHED_RRandSCHED_FIFOtasks are scheduled before tasks with normal and extensible scheduling policiesThe actual monitor seems to know about deadline too. Surely changelog and document need updating?
Yes.
Quoted 3 lines
While there, you should probably extend to cover at least STOP class, but possibly also IDLE class, no? Validating: IDLE < EXT < FAIR < FIFO/RR < DL < STOP.
Right, I forgot that idle and stop exists. Thanks for the reminder.
Nam