Add monitors to validate task switch
Subsystems (4)
Gabriele Monaco
maintainerv4 cover Add monitors to validate task switch
This series adds three monitors to the sched collection, extends and replaces previously existing monitors:
{tss,snroc} => sts:
Not only prove that switches occur in scheduling context and scheduling needs interrupt disabled but also that each call to the scheduler disables interrupts to (optionally) switch.
nrp (NEW):
* preemption requires need resched which is cleared by any switch (includes a non optimal workaround for /nested/ preemptions)
sssw (NEW):
* suspension requires setting the task to sleepable and, after the switch occurs, the task requires a wakeup to come back to runnable
opid (NEW):
* waking and need-resched operations occur with interrupts and preemption disabled or in IRQ without explicitly disabling preemption
Also include some minor cleanup patches (1-9) tracepoints (11) and preparatory fixes (10) covering some corner cases:
The series is currently based on the next tree.
Patch 1 fixes the behaviour of the rv tool with -s and idle tasks.
Patch 2 allows the rv tool to gracefully terminate with SIGTERM
Patch 3 adds da_handle_start_run_event_ also to per-task monitors
Patch 4 removes a trailing whitespace from the rv tracepoint string
Patch 5 returns the registration error in all DA monitor instead of 0
Patch 6 fixes an out-of-bound memory access in DA tracepoints
Patch 7 adjusts monitors to have minimised Kconfig dependencies
Patch 8 properly orders nested monitors in the RV Kconfig file
Patch 9 adjusts dot2c not to create lines over 100 columns
Patch 10 detects race conditions when rv monitors run concurrently and retries applying the events
Patch 11 adds the need_resched and removes unused arguments from schedule entry/exit tracepoints
Patch 12 adds the sts monitor to replace tss and sncid
Patch 13 adds the nrp and sssw monitors
Patch 14 adds the opid monitor
NOTES
The nrp and sssw monitors include workarounds for racy conditions:
* A sleeping task requires to set the state to sleepable, but in case of a task sleeping on an rtlock, the set sleepable and wakeup events race and we don't always see the right order:
5d..2. 107.488369: event: 639: sleepable x set_sleepable -> sleepable 4d..5. 107.488369: event: 639: sleepable x wakeup -> running (final) 5d..3. 107.488385: error: 639: switch_suspend not expected in the state running
wakeup() set_state()
state=RUNNING
trace_set_state()
trace_wakeup()
state=SLEEPING
I added a special event (switch_block) but there may be a better way. Taking a pi_lock in rtlock_slowlock_locked when setting the state to TASK_RTLOCK_WAIT avoids this race, although this is likely not something we want to do.
* I consider preemption any scheduling with preempt==true and assume this can happen only if need resched is set.
In practice, however, we may see a preemption where the flag is not set. This can happen in one specific condition:
need_resched
preempt_schedule()
preempt_schedule_irq()
__schedule()
!need_resched
__schedule() We start a standard preemption (e.g. from preempt_enable when the flag is set), an interrupts occurs before we schedule and, on its exit path, it schedules, which clears the need_resched flag.
When the preempted task runs again, we continue the standard preemption started earlier, although the flag is no longer set.
I added a workaround to allow the model not to fail in this condition, by allowing a preemption without need_resched if an interrupt is received. This might catch false negatives too.
Changes since V3 [1]:
* Fix condition for lines shorter than 100 columns in dot2c.
* Fix Kconfig tooltip for container monitors in rvgen.
* Improve condition not to skip pid-0 in userspace tool.
* Rearrange monitors to reduce needed tracepoints and arguments.
* Separately track errors when DAs run out of retries due to races.
* Fix issue in opid with multiple handlers in the same interrupt.
* Cleanup patches by removing, squashing and reordering.
Changes since RFC2:
* Arrange commits to prevent failed build while bisecting.
* Avoid dot2k generated files to reach the column limit. (Nam Cao)
* Rearrange and simplify da_monitor retry on racing events.
* Improve nrp monitor to handle /nested/ preemption on IRQ.
* Added minor patches (6-10).
* Cleanup and rearrange order.
Changes since RFC [2]:
* Remove wakeup tracepoint in try_to_block_task and use a different flavour of sched_set_state
* Split the large srs monitor in two separate monitors for preemption and sleep. These no longer have a concept of running task, they just enforce the requirements for the different types of sched out.
* Restore the snroc monitor to describe the relationship between generic sched out and sched in.
* Add opid monitor.
* Fix some build errors and cleanup.
[1] - https://lore.kernel.org/lkml/[email protected]
[2] - https://lore.kernel.org/lkml/[email protected]
To: Ingo Molnar <[email protected]>
To: Peter Zijlstra <[email protected]>
To: Steven Rostedt <[email protected]>
To: Nam Cao <[email protected]>
Cc: Tomas Glozar <[email protected]>
Cc: Juri Lelli <[email protected]>
Cc: Clark Williams <[email protected]>
Cc: John Kacur <[email protected]>
Gabriele Monaco (14):
tools/rv: Do not skip idle in trace
tools/rv: Stop gracefully also on SIGTERM
rv: Add da_handle_start_run_event_ to per-task monitors
rv: Remove trailing whitespace from tracepoint string
rv: Return init error when registering monitors
rv: Use strings in da monitors tracepoints
rv: Adjust monitor dependencies
verification/rvgen: Organise Kconfig entries for nested monitors tools/dot2c: Fix generated files going over 100 column limit
rv: Retry when da monitor detects race conditions
sched: Adapt sched tracepoints for RV task model
rv: Replace tss and sncid monitors with more complete sts
rv: Add nrp and sssw per-task monitors
rv: Add opid per-cpu monitor
Documentation/trace/rv/monitor_sched.rst | 307 +++++++++++++++---
include/linux/rv.h | 3 +-
include/linux/sched.h | 7 +-
include/rv/da_monitor.h | 129 +++++---
include/trace/events/sched.h | 12 +-
kernel/sched/core.c | 13 +-
kernel/trace/rv/Kconfig | 16 +-
kernel/trace/rv/Makefile | 6 +-
kernel/trace/rv/monitors/{tss => nrp}/Kconfig | 12 +-
kernel/trace/rv/monitors/nrp/nrp.c | 138 ++++++++
kernel/trace/rv/monitors/nrp/nrp.h | 75 +++++
kernel/trace/rv/monitors/nrp/nrp_trace.h | 15 +
kernel/trace/rv/monitors/opid/Kconfig | 19 ++
kernel/trace/rv/monitors/opid/opid.c | 169 ++++++++++
kernel/trace/rv/monitors/opid/opid.h | 104 ++++++
.../sncid_trace.h => opid/opid_trace.h} | 8 +-
kernel/trace/rv/monitors/sched/Kconfig | 1 +
kernel/trace/rv/monitors/sched/sched.c | 3 +-
kernel/trace/rv/monitors/sco/sco.c | 7 +-
kernel/trace/rv/monitors/scpd/Kconfig | 2 +-
kernel/trace/rv/monitors/scpd/scpd.c | 7 +-
kernel/trace/rv/monitors/sncid/sncid.c | 96 ------
kernel/trace/rv/monitors/sncid/sncid.h | 49 ---
kernel/trace/rv/monitors/snep/Kconfig | 2 +-
kernel/trace/rv/monitors/snep/snep.c | 7 +-
kernel/trace/rv/monitors/snep/snep.h | 14 +-
kernel/trace/rv/monitors/snroc/snroc.c | 3 +-
.../trace/rv/monitors/{sncid => sssw}/Kconfig | 10 +-
kernel/trace/rv/monitors/sssw/sssw.c | 116 +++++++
kernel/trace/rv/monitors/sssw/sssw.h | 105 ++++++
kernel/trace/rv/monitors/sssw/sssw_trace.h | 15 +
kernel/trace/rv/monitors/sts/Kconfig | 19 ++
kernel/trace/rv/monitors/sts/sts.c | 156 +++++++++
kernel/trace/rv/monitors/sts/sts.h | 117 +++++++
.../{tss/tss_trace.h => sts/sts_trace.h} | 8 +-
kernel/trace/rv/monitors/tss/tss.c | 91 ------
kernel/trace/rv/monitors/tss/tss.h | 47 ---
kernel/trace/rv/monitors/wip/Kconfig | 2 +-
kernel/trace/rv/monitors/wip/wip.c | 3 +-
kernel/trace/rv/monitors/wwnr/wwnr.c | 3 +-
kernel/trace/rv/rv_trace.h | 114 ++++---
tools/verification/models/sched/nrp.dot | 29 ++
tools/verification/models/sched/opid.dot | 35 ++
tools/verification/models/sched/sncid.dot | 18 -
tools/verification/models/sched/sssw.dot | 30 ++
tools/verification/models/sched/sts.dot | 38 +++
tools/verification/models/sched/tss.dot | 18 -
tools/verification/rv/src/in_kernel.c | 4 +-
tools/verification/rv/src/rv.c | 1 +
tools/verification/rvgen/rvgen/container.py | 10 +
tools/verification/rvgen/rvgen/dot2c.py | 20 +-
tools/verification/rvgen/rvgen/generator.py | 16 +-
52 files changed, 1723 insertions(+), 526 deletions(-)
rename kernel/trace/rv/monitors/{tss => nrp}/Kconfig (51%)
create mode 100644 kernel/trace/rv/monitors/nrp/nrp.c
create mode 100644 kernel/trace/rv/monitors/nrp/nrp.h
create mode 100644 kernel/trace/rv/monitors/nrp/nrp_trace.h
create mode 100644 kernel/trace/rv/monitors/opid/Kconfig
create mode 100644 kernel/trace/rv/monitors/opid/opid.c
create mode 100644 kernel/trace/rv/monitors/opid/opid.h
rename kernel/trace/rv/monitors/{sncid/sncid_trace.h => opid/opid_trace.h} (66%)
delete mode 100644 kernel/trace/rv/monitors/sncid/sncid.c
delete mode 100644 kernel/trace/rv/monitors/sncid/sncid.h
rename kernel/trace/rv/monitors/{sncid => sssw}/Kconfig (58%)
create mode 100644 kernel/trace/rv/monitors/sssw/sssw.c
create mode 100644 kernel/trace/rv/monitors/sssw/sssw.h
create mode 100644 kernel/trace/rv/monitors/sssw/sssw_trace.h
create mode 100644 kernel/trace/rv/monitors/sts/Kconfig
create mode 100644 kernel/trace/rv/monitors/sts/sts.c
create mode 100644 kernel/trace/rv/monitors/sts/sts.h
rename kernel/trace/rv/monitors/{tss/tss_trace.h => sts/sts_trace.h} (67%)
delete mode 100644 kernel/trace/rv/monitors/tss/tss.c
delete mode 100644 kernel/trace/rv/monitors/tss/tss.h
create mode 100644 tools/verification/models/sched/nrp.dot
create mode 100644 tools/verification/models/sched/opid.dot
delete mode 100644 tools/verification/models/sched/sncid.dot
create mode 100644 tools/verification/models/sched/sssw.dot
create mode 100644 tools/verification/models/sched/sts.dot
delete mode 100644 tools/verification/models/sched/tss.dotbase-commit: d086c886ceb9f59dea6c3a9dae7eb89e780a20c9
Signature · 1 line
2.50.1
Gabriele Monaco
maintainerPATCH v4 1/14 tools/rv: Do not skip idle in trace
Currently, the userspace RV tool skips trace events triggered by the RV tool itself, this can be changed by passing the parameter -s, which sets the variable config_my_pid to 0 (instead of the tool's PID). This has the side effect of skipping events generated by idle (PID 0).
Set config_my_pid to -1 (an invalid pid) to avoid skipping idle.
--- tools/verification/rv/src/in_kernel.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
tools/verification/rv/src/in_kernel.c +2 -2
@@ -431,7 +431,7 @@ ikm_event_handler(struct trace_seq *s, struct tep_record *record,if (config_has_id && (config_my_pid == id))return 0 ; -else if (config_my_pid && (config_my_pid == pid)) +else if (config_my_pid == pid)return 0 ; tep_print_event(trace_event->tep, s, record, "%16s-%-8d [%.3d] ",
@@ -734,7 +734,7 @@ static int parse_arguments(char *monitor_name, int argc, char **argv) config_reactor = optarg;break ;case 's' : - config_my_pid =0 ; + config_my_pid =-1 ;break ;case 't' : config_trace =1 ;
Signature · 1 line
2.50.1
Gabriele Monaco
maintainerPATCH v4 2/14 tools/rv: Stop gracefully also on SIGTERM
In reply to Gabriele MonacoCurrently the userspace RV tool starts a monitor and waits for the user to press Ctrl-C (SIGINT) to terminate and stop the monitor. This doesn't account for a scenario where a user starts RV in background and simply kills it (SIGTERM unless the user specifies differently). E.g.:
# rv mon wip & # kill %
Would terminate RV without stopping the monitor and next RV executions won't start correctly.
Register the signal handler used for SIGINT also to SIGTERM.
--- tools/verification/rv/src/rv.c | 1 + 1 file changed, 1 insertion(+)
tools/verification/rv/src/rv.c +1 -0
@@ -191,6 +191,7 @@ int main(int argc, char **argv)* and exit. */ signal (SIGINT , stop_rv); +signal (SIGTERM , stop_rv);rv_mon (argc -1 , &argv[1 ]); }
Signature · 1 line
2.50.1
Gabriele Monaco
maintainerPATCH v4 3/14 Add da_handle_start_run_event_ to per-task monitors
In reply to Gabriele MonacoThe RV da_monitor API allows to start monitors in two ways:da_handle_start_event_NAME and da_handle_start_run_event_NAME. The former is used when the event is followed by the initial state of the module, so we ignore the event but we know the monitor is in the initial state and can start monitoring, the latter can be used if the event can only occur in the initial state, so we do handle the event as if the monitor was in the initial state.
This latter API is defined for implicit monitors but not per-task ones.
Define da_handle_start_run_event_NAME macro also for per-task monitors.
--- include/rv/da_monitor.h | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)
include/rv/da_monitor.h +24 -0
@@ -487,6 +487,30 @@ da_handle_start_event_##name(struct task_struct *tsk, enum events_##name event) __da_handle_event_##name(da_mon, tsk, event); \ \ return 1; \ +} \ + \ +/* \ +* da_handle_start_run_event_##name - start monitoring and handle event \ +* \ +* This function is used to notify the monitor that the system is in the \ +* initial state, so the monitor can start monitoring and handling event. \ +*/ \ +static inline bool \ +da_handle_start_run_event_##name(struct task_struct *tsk, enum events_##name event) \ +{ \ + struct da_monitor *da_mon; \ + \ + if (!da_monitor_enabled_##name()) \ + return 0; \ + \ + da_mon = da_get_monitor_##name(tsk); \ + \ + if (unlikely(!da_monitoring_##name(da_mon))) \ + da_monitor_start_##name(da_mon); \ + \ + __da_handle_event_##name(da_mon, tsk, event); \ + \ + return 1; \ }/*
Signature · 1 line
2.50.1
Gabriele Monaco
maintainerPATCH v4 4/14 Remove trailing whitespace from tracepoint string
In reply to Gabriele MonacoRV event tracepoints print a line with the format:
"event_xyz: S0 x event -> S1 "
"event_xyz: S1 x event -> S0 (final)"While printing an event leading to a non-final state, the line has a trailing white space (visible above before the closing ").
Adapt the format string not to print the trailing whitespace if we are not printing "(final)".
--- kernel/trace/rv/rv_trace.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
kernel/trace/rv/rv_trace.h +4 -4
@@ -29,11 +29,11 @@ DECLARE_EVENT_CLASS(event_da_monitor, __entry->final_state = final_state; ), -TP_printk ("%s x %s -> %s %s" , +TP_printk ("%s x %s -> %s%s" , __entry->state, __entry->event, __entry->next_state, - __entry->final_state ?"(final)" :"" ) + __entry->final_state ?" (final)" :"" ) ); DECLARE_EVENT_CLASS(error_da_monitor,
@@ -90,12 +90,12 @@ DECLARE_EVENT_CLASS(event_da_monitor_id, __entry->final_state = final_state; ), -TP_printk ("%d: %s x %s -> %s %s" , +TP_printk ("%d: %s x %s -> %s%s" , __entry->id, __entry->state, __entry->event, __entry->next_state, - __entry->final_state ?"(final)" :"" ) + __entry->final_state ?" (final)" :"" ) ); DECLARE_EVENT_CLASS(error_da_monitor_id,
Signature · 1 line
2.50.1
Gabriele Monaco
maintainerPATCH v4 5/14 Return init error when registering monitors
In reply to Gabriele MonacoMonitors generated with dot2k have their registration function (the one called during monitor initialisation) return always 0, even if the registration failed on RV side.
This can hide potential errors.
Return the value returned by the RV register function.
--- kernel/trace/rv/monitors/sched/sched.c | 3 +-- kernel/trace/rv/monitors/sco/sco.c | 3 +-- kernel/trace/rv/monitors/scpd/scpd.c | 3 +-- kernel/trace/rv/monitors/sncid/sncid.c | 3 +-- kernel/trace/rv/monitors/snep/snep.c | 3 +-- kernel/trace/rv/monitors/snroc/snroc.c | 3 +-- kernel/trace/rv/monitors/tss/tss.c | 3 +-- kernel/trace/rv/monitors/wip/wip.c | 3 +-- kernel/trace/rv/monitors/wwnr/wwnr.c | 3 +-- 9 files changed, 9 insertions(+), 18 deletions(-)
kernel/trace/rv/monitors/sched/sched.c +1 -2
@@ -21,8 +21,7 @@ struct rv_monitor rv_sched = {static int __init register_sched (void ) { -rv_register_monitor (&rv_sched,NULL ); -return 0 ; +return rv_register_monitor (&rv_sched,NULL ); } static void __exit unregister_sched(void)
kernel/trace/rv/monitors/sco/sco.c +1 -2
@@ -71,8 +71,7 @@ static struct rv_monitor rv_sco = {static int __init register_sco (void ) { -rv_register_monitor (&rv_sco, &rv_sched); -return 0 ; +return rv_register_monitor (&rv_sco, &rv_sched); } static void __exit unregister_sco(void)
kernel/trace/rv/monitors/scpd/scpd.c +1 -2
@@ -79,8 +79,7 @@ static struct rv_monitor rv_scpd = {static int __init register_scpd (void ) { -rv_register_monitor (&rv_scpd, &rv_sched); -return 0 ; +return rv_register_monitor (&rv_scpd, &rv_sched); } static void __exit unregister_scpd(void)
kernel/trace/rv/monitors/sncid/sncid.c +1 -2
@@ -79,8 +79,7 @@ static struct rv_monitor rv_sncid = {static int __init register_sncid (void ) { -rv_register_monitor (&rv_sncid, &rv_sched); -return 0 ; +return rv_register_monitor (&rv_sncid, &rv_sched); } static void __exit unregister_sncid(void)
kernel/trace/rv/monitors/snep/snep.c +1 -2
@@ -79,8 +79,7 @@ static struct rv_monitor rv_snep = {static int __init register_snep (void ) { -rv_register_monitor (&rv_snep, &rv_sched); -return 0 ; +return rv_register_monitor (&rv_snep, &rv_sched); } static void __exit unregister_snep(void)
kernel/trace/rv/monitors/snroc/snroc.c +1 -2
@@ -68,8 +68,7 @@ static struct rv_monitor rv_snroc = {static int __init register_snroc (void ) { -rv_register_monitor (&rv_snroc, &rv_sched); -return 0 ; +return rv_register_monitor (&rv_snroc, &rv_sched); } static void __exit unregister_snroc(void)
kernel/trace/rv/monitors/tss/tss.c +1 -2
@@ -74,8 +74,7 @@ static struct rv_monitor rv_tss = {static int __init register_tss (void ) { -rv_register_monitor (&rv_tss, &rv_sched); -return 0 ; +return rv_register_monitor (&rv_tss, &rv_sched); } static void __exit unregister_tss(void)
kernel/trace/rv/monitors/wip/wip.c +1 -2
@@ -71,8 +71,7 @@ static struct rv_monitor rv_wip = {static int __init register_wip (void ) { -rv_register_monitor (&rv_wip,NULL ); -return 0 ; +return rv_register_monitor (&rv_wip,NULL ); } static void __exit unregister_wip(void)
kernel/trace/rv/monitors/wwnr/wwnr.c +1 -2
@@ -70,8 +70,7 @@ static struct rv_monitor rv_wwnr = {static int __init register_wwnr (void ) { -rv_register_monitor (&rv_wwnr,NULL ); -return 0 ; +return rv_register_monitor (&rv_wwnr,NULL ); } static void __exit unregister_wwnr(void)
Signature · 1 line
2.50.1
Gabriele Monaco
maintainerPATCH v4 6/14 Use strings in da monitors tracepoints
In reply to Gabriele MonacoUsing DA monitors tracepoints with KASAN enabled triggers the following warning:
BUG: KASAN: global-out-of-bounds in do_trace_event_raw_event_event_da_monitor+0xd6/0x1a0 Read of size 32 at addr ffffffffaada8980 by task ... Call Trace: <TASK> [...] do_trace_event_raw_event_event_da_monitor+0xd6/0x1a0 ? __pfx_do_trace_event_raw_event_event_da_monitor+0x10/0x10 ? trace_event_sncid+0x83/0x200 trace_event_sncid+0x163/0x200 [...] The buggy address belongs to the variable: automaton_snep+0x4e0/0x5e0
This is caused by the tracepoints reading 32 bytes __array instead of __string from the automata definition. Such strings are literals and reading 32 bytes ends up in out of bound memory accesses (e.g. the next automaton's data in this case).
The error is harmless as, while printing the string, we stop at the null terminator, but it should still be fixed.
Use the __string facilities while defining the tracepoints to avoid reading out of bound memory.
--- kernel/trace/rv/rv_trace.h | 76 +++++++++++++++++++------------------- 1 file changed, 38 insertions(+), 38 deletions(-)
kernel/trace/rv/rv_trace.h +38 -38
@@ -16,23 +16,23 @@ DECLARE_EVENT_CLASS(event_da_monitor, TP_ARGS(state, event, next_state, final_state),TP_STRUCT__entry ( -__array (char ,state ,MAX_DA_NAME_LEN ) -__array ( char, event,MAX_DA_NAME_LEN ) -__array ( char, next_state,MAX_DA_NAME_LEN ) -__field ( bool, final_state ) +__string (state ,state ) +__string ( event, event ) +__string ( next_state, next_state ) +__field ( bool, final_state ) ),TP_fast_assign ( - memcpy(__entry->state, state, MAX_DA_NAME_LEN); -memcpy (__entry->event, event,MAX_DA_NAME_LEN ); -memcpy (__entry->next_state, next_state,MAX_DA_NAME_LEN ); - __entry->final_state = final_state; + __assign_str(state); +__assign_str (event); +__assign_str (next_state); + __entry->final_state = final_state; ),TP_printk ("%s x %s -> %s%s" , - __entry->state, - __entry->event, - __entry->next_state, +__get_str (state), +__get_str (event), +__get_str (next_state), __entry->final_state ?" (final)" :"" ) );
@@ -43,18 +43,18 @@ DECLARE_EVENT_CLASS(error_da_monitor,TP_ARGS (state, event),TP_STRUCT__entry ( - __array( char, state, MAX_DA_NAME_LEN ) -__array ( char, event,MAX_DA_NAME_LEN ) + __string( state, state ) +__string ( event, event ) ),TP_fast_assign ( - memcpy(__entry->state, state, MAX_DA_NAME_LEN); -memcpy (__entry->event, event,MAX_DA_NAME_LEN ); + __assign_str(state); +__assign_str (event); ),TP_printk ("event %s not expected in the state %s" , - __entry->event, - __entry->state) +__get_str (event), +__get_str (state)) );#include <monitors/wip/wip_trace.h>
@@ -75,26 +75,26 @@ DECLARE_EVENT_CLASS(event_da_monitor_id, TP_ARGS(id, state, event, next_state, final_state),TP_STRUCT__entry ( -__field (int ,id ) -__array ( char, state,MAX_DA_NAME_LEN ) -__array ( char, event,MAX_DA_NAME_LEN ) -__array ( char, next_state,MAX_DA_NAME_LEN ) -__field ( bool, final_state ) +__field (int ,id ) +__string ( state, state ) +__string ( event, event ) +__string ( next_state, next_state ) +__field ( bool, final_state ) ),TP_fast_assign ( - memcpy(__entry->state, state, MAX_DA_NAME_LEN); -memcpy (__entry->event, event,MAX_DA_NAME_LEN ); -memcpy (__entry->next_state, next_state,MAX_DA_NAME_LEN ); - __entry->id = id; - __entry->final_state = final_state; + __assign_str(state); +__assign_str (event); +__assign_str (next_state); + __entry->id = id; + __entry->final_state = final_state; ),TP_printk ("%d: %s x %s -> %s%s" , __entry->id, - __entry->state, - __entry->event, - __entry->next_state, +__get_str (state), +__get_str (event), +__get_str (next_state), __entry->final_state ?" (final)" :"" ) );
@@ -105,21 +105,21 @@ DECLARE_EVENT_CLASS(error_da_monitor_id, TP_ARGS(id, state, event),TP_STRUCT__entry ( -__field (int ,id ) -__array ( char, state,MAX_DA_NAME_LEN ) -__array ( char, event,MAX_DA_NAME_LEN ) +__field (int ,id ) +__string ( state, state ) +__string ( event, event ) ),TP_fast_assign ( - memcpy(__entry->state, state, MAX_DA_NAME_LEN); -memcpy (__entry->event, event,MAX_DA_NAME_LEN ); - __entry->id = id; + __assign_str(state); +__assign_str (event); + __entry->id = id; ),TP_printk ("%d: event %s not expected in the state %s" , __entry->id, - __entry->event, - __entry->state) +__get_str (event), +__get_str (state)) );#include <monitors/wwnr/wwnr_trace.h>
Signature · 1 line
2.50.1
Gabriele Monaco
maintainerPATCH v4 7/14 Adjust monitor dependencies
In reply to Gabriele MonacoRV monitors relying on the preemptirqs tracepoints are set as dependent on PREEMPT_TRACER and IRQSOFF_TRACER. In fact, those configurations do enable the tracepoints but are not the minimal configurations enabling them, which are TRACE_PREEMPT_TOGGLE and TRACE_IRQFLAGS (not selectable manually).
Set TRACE_PREEMPT_TOGGLE and TRACE_IRQFLAGS as dependencies for monitors.
--- kernel/trace/rv/monitors/scpd/Kconfig | 2 +- kernel/trace/rv/monitors/sncid/Kconfig | 2 +- kernel/trace/rv/monitors/snep/Kconfig | 2 +- kernel/trace/rv/monitors/wip/Kconfig | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-)
kernel/trace/rv/monitors/scpd/Kconfig +1 -1
@@ -2,7 +2,7 @@ # config RV_MON_SCPD depends on RV - depends on PREEMPT_TRACER + depends on TRACE_PREEMPT_TOGGLE depends on RV_MON_SCHED default y select DA_MON_EVENTS_IMPLICIT
kernel/trace/rv/monitors/sncid/Kconfig +1 -1
@@ -2,7 +2,7 @@ # config RV_MON_SNCID depends on RV - depends on IRQSOFF_TRACER + depends on TRACE_IRQFLAGS depends on RV_MON_SCHED default y select DA_MON_EVENTS_IMPLICIT
kernel/trace/rv/monitors/snep/Kconfig +1 -1
@@ -2,7 +2,7 @@ # config RV_MON_SNEP depends on RV - depends on PREEMPT_TRACER + depends on TRACE_PREEMPT_TOGGLE depends on RV_MON_SCHED default y select DA_MON_EVENTS_IMPLICIT
kernel/trace/rv/monitors/wip/Kconfig +1 -1
@@ -2,7 +2,7 @@ # config RV_MON_WIP depends on RV - depends on PREEMPT_TRACER + depends on TRACE_PREEMPT_TOGGLE select DA_MON_EVENTS_IMPLICIT bool "wip monitor" help
Signature · 1 line
2.50.1
Gabriele Monaco
maintainerPATCH v4 8/14 verification/rvgen: Organise Kconfig entries for nested monitors
In reply to Gabriele MonacoThe current behaviour of rvgen when running with the -a option is to append the necessary lines at the end of the configuration for Kconfig, Makefile and tracepoints.
This is not always the desired behaviour in case of nested monitors:
while tracepoints are not affected by nesting and the Makefile's only requirement is that the parent monitor is built before its children, in the Kconfig it is better to have children defined right after their parent, otherwise the result has wrong indentation:
[*] foo_parent monitor
[*] foo_child1 monitor
[*] foo_child2 monitor
[*] bar_parent monitor
[*] bar_child1 monitor
[*] bar_child2 monitor
[*] foo_child3 monitor
[*] foo_child4 monitor
Adapt rvgen to look for a different marker for nested monitors in the Kconfig file and append the line right after the last sibling, instead of the last monitor.
Also add the marker when creating a new parent monitor.
--- kernel/trace/rv/Kconfig | 5 +++++ tools/verification/rvgen/rvgen/container.py | 10 ++++++++++ tools/verification/rvgen/rvgen/generator.py | 16 +++++++++++----- 3 files changed, 26 insertions(+), 5 deletions(-)
kernel/trace/rv/Kconfig +5 -0
@@ -43,6 +43,7 @@ config RV_PER_TASK_MONITORS source "kernel/trace/rv/monitors/wip/Kconfig" source "kernel/trace/rv/monitors/wwnr/Kconfig" + source "kernel/trace/rv/monitors/sched/Kconfig" source "kernel/trace/rv/monitors/tss/Kconfig" source "kernel/trace/rv/monitors/sco/Kconfig"
@@ -50,9 +51,13 @@ source "kernel/trace/rv/monitors/snroc/Kconfig" source "kernel/trace/rv/monitors/scpd/Kconfig" source "kernel/trace/rv/monitors/snep/Kconfig" source "kernel/trace/rv/monitors/sncid/Kconfig" +# Add new sched monitors here + source "kernel/trace/rv/monitors/rtapp/Kconfig" source "kernel/trace/rv/monitors/pagefault/Kconfig" source "kernel/trace/rv/monitors/sleep/Kconfig" +# Add new rtapp monitors here + # Add new monitors here config RV_REACTORS
tools/verification/rvgen/rvgen/container.py +10 -0
@@ -20,3 +20,13 @@ class Container(generator.RVGenerator): main_h = self.main_h main_h = main_h.replace("%%MODEL_NAME%%", self.name) return main_h + + def fill_kconfig_tooltip(self): + """Override to produce a marker for this container in the Kconfig""" + container_marker = self._container_marker(self.name) + "\n" + result = super().fill_kconfig_tooltip() + if self.auto_patch: + self._patch_file("Kconfig", + self._container_marker(), container_marker) + return result + return result + container_marker
tools/verification/rvgen/rvgen/generator.py +11 -5
@@ -137,7 +137,8 @@ class RVGenerator: kconfig = kconfig.replace("%%MONITOR_DEPS%%", monitor_deps) return kconfig - def __patch_file(self, file, marker, line): + def _patch_file(self, file, marker, line): + assert(self.auto_patch) file_to_patch = os.path.join(self.rv_dir, file) content = self._read_file(file_to_patch) content = content.replace(marker, line + "\n" + marker)
@@ -146,7 +147,7 @@ class RVGenerator: def fill_tracepoint_tooltip(self): monitor_class_type = self.fill_monitor_class_type() if self.auto_patch: - self.__patch_file("rv_trace.h", + self._patch_file("rv_trace.h", "// Add new monitors based on CONFIG_%s here" % monitor_class_type, "#include <monitors/%s/%s_trace.h>" % (self.name, self.name)) return " - Patching %s/rv_trace.h, double check the result" % self.rv_dir
@@ -156,10 +157,15 @@ Add this line where other tracepoints are included and %s is defined: #include <monitors/%s/%s_trace.h> """ % (self.rv_dir, monitor_class_type, self.name, self.name) + def _container_marker(self, container = None) -> str: + return "# Add new %smonitors here" % (container + " " + if container else "") + def fill_kconfig_tooltip(self): if self.auto_patch: - self.__patch_file("Kconfig", - "# Add new monitors here", + # monitors with a container should stay together in the Kconfig + self._patch_file("Kconfig", + self._container_marker(self.parent), "source \"kernel/trace/rv/monitors/%s/Kconfig\"" % (self.name)) return " - Patching %s/Kconfig, double check the result" % self.rv_dir
@@ -172,7 +178,7 @@ source \"kernel/trace/rv/monitors/%s/Kconfig\" name = self.name name_up = name.upper() if self.auto_patch: - self.__patch_file("Makefile", + self._patch_file("Makefile", "# Add new monitors here", "obj-$(CONFIG_RV_MON_%s) += monitors/%s/%s.o" % (name_up, name, name)) return " - Patching %s/Makefile, double check the result" % self.rv_dir
Signature · 1 line
2.50.1
Re: [PATCH v4 08/14] verification/rvgen: Organise Kconfig entries for nested monitors
Quoted 18 lines · Gabriele Monaco
The current behaviour of rvgen when running with the -a option is to append the necessary lines at the end of the configuration for
Kconfig,Makefileand tracepoints.
This is not always the desired behaviour in case of nested monitors:
while tracepoints are not affected by nesting and the Makefile's only requirement is that the parent monitor is built before its children, in theKconfigit is better to have children defined right after their parent, otherwise the result has wrong indentation:[*]
foo_parentmonitor
[*]foo_child1monitor
[*]foo_child2monitor
[*]bar_parentmonitor
[*]bar_child1monitor
[*]bar_child2monitor
[*]foo_child3monitor
[*]foo_child4monitor
Quoted 6 lines
Adapt rvgen to look for a different marker for nested monitors in the
Kconfigfile and append the line right after the last sibling, instead of the last monitor.
Also add the marker when creating a new parent monitor.Signed-off-by Gabriele Monaco <[email protected]>
Some nitpicks below. But regardless:
tools/verification/rvgen/rvgen/generator.py class RVGenerator: · line 141
Quoted 3 lines
tools/verification/rvgen/rvgen/generator.py +2
-1- def __patch_file(self, file, marker, line): + def _patch_file(self, file, marker, line): + assert(self.auto_patch)
Nit: follows PEP8 unless there is a reason not to: assert self.auto_patch
Quoted 13 lines
file_to_patch=os.path.join(self.rv_dir, file)content =self._read_file(file_to_patch)content =content.replace(marker, line + "\n" + marker)tools/verification/rvgen/rvgen/generator.py +1
-1@@ -146,7 +147,7 @@ class RVGenerator: def fill_tracepoint_tooltip(self): monitor_class_type = self.fill_monitor_class_type() if self.auto_patch: - self.__patch_file("rv_trace.h", + self._patch_file("rv_trace.h", "// Add new monitors based on CONFIG_%s here" % monitor_class_type, "#include <monitors/%s/%s_trace.h>" % (self.name, self.name)) return " - Patching %s/rv_trace.h, double check the result" % self.rv_dir
tools/verification/rvgen/rvgen/generator.py Add this line where other tracepoints are included and %s is defined: · line 162
Quoted 6 lines
tools/verification/rvgen/rvgen/generator.py +3
-0@@ -156,10 +157,15 @@ Add this line where other tracepoints are included and %s is defined: #include <monitors/%s/%s_trace.h> """ % (self.rv_dir, monitor_class_type, self.name, self.name) + def _container_marker(self, container = None) -> str: + return "# Add new %smonitors here" % (container + " " + if container else "")
PEP8: container=None
And this function name is misleading, it is not marker for only containers. Perhaps _kconfig_marker() is more appropriate.
Gabriele Monaco
maintainerRe: [PATCH v4 08/14] verification/rvgen: Organise Kconfig entries for nested monitors
Quoted 15 lines · Nam Cao
Quoted 4 lines · Gabriele Monaco
The current behaviour of rvgen when running with the -a option is to append the necessary lines at the end of the configuration for
Kconfig,Makefileand tracepoints.Quoted 8 lines
Adapt rvgen to look for a different marker for nested monitors in the
Kconfigfile and append the line right after the last sibling, instead
of the last monitor.
Also add the marker when creating a new parent monitor.Signed-off-by Gabriele Monaco <[email protected]>Some nitpicks below. But regardless:
Quoted 8 lines
Reviewed-by Nam Cao <[email protected]>Quoted 3 lines
- def
__patch_file(self, file, marker, line):
+ def_patch_file(self, file, marker, line):
+assert(self.auto_patch)Nit: follows PEP8 unless there is a reason not to: assert self.auto_patch
Well, all the python best practices I follow are the ones reported by pylint, I guess I need to configure it to follow those. Thanks for pointing that out!
Quoted 28 lines
Quoted 17 lines
file_to_patch=os.path.join(self.rv_dir, file)content =self._read_file(file_to_patch)content =content.replace(marker, line + "\n" + marker)(quoted diff) +0
-0@@ -146,7 +147,7 @@ class RVGenerator:def
fill_tracepoint_tooltip(self):
monitor_class_type=self.fill_monitor_class_type()if self.auto_patch:
- self.__patch_file("rv_trace.h",
+ self._patch_file("rv_trace.h",
"// Add new monitors based on CONFIG_%s here" %monitor_class_type,
"#include <monitors/%s/%s_trace.h>" % (self.name, self.name))
return " - Patching %s/rv_trace.h, double check the result" % self.rv_dir(quoted diff) +0
-0@@ -156,10 +157,15 @@ Add this line where other tracepoints areQuoted 8 lines
included and %s is defined:
#include <monitors/%s/%s_trace.h>
""" % (self.rv_dir,monitor_class_type, self.name, self.name)
+ def_container_marker(self, container = None)-> str:
+ return "# Add new %smonitors here" % (container + " " + if container else "")PEP8: container=None
Quoted 3 lines
And this function name is misleading, it is not marker for only containers.
Perhaps_kconfig_marker()is more appropriate.
Good point.
Thanks,
Gabriele
Gabriele Monaco
maintainerPATCH v4 9/14 tools/dot2c: Fix generated files going over 100 column limit
In reply to Gabriele MonacoThe dot2c.py script generates all states in a single line. This breaks the 100 column limit when the state machines are non-trivial.
Change dot2c.py to generate the states in separate lines in case the generated line is going to be too long.
Also adapt existing monitors with line length over the limit.
--- kernel/trace/rv/monitors/snep/snep.h | 14 ++++++++++++-- tools/verification/rvgen/rvgen/dot2c.py | 20 +++++++++++--------- 2 files changed, 23 insertions(+), 11 deletions(-)
kernel/trace/rv/monitors/snep/snep.h +12 -2
@@ -41,8 +41,18 @@ static const struct automaton_snep automaton_snep = {"schedule_exit" }, .function = { - { non_scheduling_context_snep, non_scheduling_context_snep, scheduling_contex_snep, INVALID_STATE }, - { INVALID_STATE, INVALID_STATE, INVALID_STATE, non_scheduling_context_snep }, + { + non_scheduling_context_snep, + non_scheduling_context_snep, + scheduling_contex_snep, +INVALID_STATE + }, + { +INVALID_STATE , +INVALID_STATE , +INVALID_STATE , + non_scheduling_context_snep + }, }, .initial_state = non_scheduling_context_snep, .final_states = { 1, 0 },
tools/verification/rvgen/rvgen/dot2c.py +11 -9
@@ -152,28 +152,30 @@ class Dot2c(Automata): max_state_name = max(self.states, key = len).__len__() return max(max_state_name, self.invalid_state_str.__len__()) - def __get_state_string_length(self): - maxlen = self.__get_max_strlen_of_states() + self.enum_suffix.__len__() - return "%" + str(maxlen) + "s" - def get_aut_init_function(self): nr_states = self.states.__len__() nr_events = self.events.__len__() buff = [] - strformat = self.__get_state_string_length() - + maxlen = self.__get_max_strlen_of_states() + len(self.enum_suffix) + tab_braces = 2 * 8 + 2 + 1 # "\t\t{ " ... "}" + comma_space = 2 # ", " count last comma here + linetoolong = tab_braces + (maxlen + comma_space) * nr_events >= self.line_length for x in range(nr_states): - line = "\t\t{ " + line = "\t\t{\n" if linetoolong else "\t\t{ " for y in range(nr_events): next_state = self.function[x][y] if next_state != self.invalid_state_str: next_state = self.function[x][y] + self.enum_suffix + if linetoolong: + line += "\t\t\t%s" % next_state + else: + line += "%*s" % (maxlen, next_state) if y != nr_events-1: - line = line + strformat % next_state + ", " + line += ",\n" if linetoolong else ", " else: - line = line + strformat % next_state + " }," + line += "\n\t\t}," if linetoolong else " }," buff.append(line) return self.__buff_to_string(buff)
Signature · 1 line
2.50.1
Re: [PATCH v4 09/14] tools/dot2c: Fix generated files going over 100 column limit
Quoted 11 lines · Gabriele Monaco
The
dot2c.pyscript generates all states in a single line. This breaks the 100 column limit when the state machines are non-trivial.Change
dot2c.pyto generate the states in separate lines in case the generated line is going to be too long.Also adapt existing monitors with line length over the limit.
Suggested-by Nam Cao <[email protected]>Signed-off-by Gabriele Monaco <[email protected]>---
tools/verification/rvgen/rvgen/dot2c.py class Dot2c(Automata): · line 162
Quoted 5 lines
tools/verification/rvgen/rvgen/dot2c.py +3
-2- strformat = self.__get_state_string_length() - + maxlen = self.__get_max_strlen_of_states() + len(self.enum_suffix) + tab_braces = 2 * 8 + 2 + 1 # "\t\t{ " ... "}" + comma_space = 2 # ", " count last comma here
PEP8 prefers two spaces before the comments.
Quoted 1 line
+ linetoolong =
tab_braces+ (maxlen +comma_space) *nr_events>= self.line_length
Shouldn't this be '>' instead of '>='? 100 columns are still within the limit.
Nam
Gabriele Monaco
maintainerRe: [PATCH v4 09/14] tools/dot2c: Fix generated files going over 100 column limit
Quoted 21 lines · Nam Cao
Quoted 12 lines · Gabriele Monaco
The
dot2c.pyscript generates all states in a single line. This breaks the
100 column limit when the state machines are non-trivial.Change
dot2c.pyto generate the states in separate lines in case the
generated line is going to be too long.Also adapt existing monitors with line length over the limit.
Suggested-by Nam Cao <[email protected]>Signed-off-by Gabriele Monaco <[email protected]>Quoted 7 lines
---
- strformat =self.__get_state_string_length()-
+ maxlen =self.__get_max_strlen_of_states()+len(self.enum_suffix)
+tab_braces= 2 * 8 + 2 + 1 # "\t\t{ " ... "}" +comma_space= 2 # ", " count last comma here
Quoted 7 lines
PEP8 prefers two spaces before the comments.
Quoted 2 lines
+ linetoolong =
tab_braces+ (maxlen +comma_space) *nr_events>= self.line_lengthShouldn't this be '>' instead of '>='? 100 columns are still within the limit.
Right, will do.
Thanks,
Gabriele
Gabriele Monaco
maintainerPATCH v4 10/14 Retry when da monitor detects race conditions
In reply to Gabriele MonacoDA monitor can be accessed from multiple cores simultaneously, this is likely, for instance when dealing with per-task monitors reacting on events that do not always occur on the CPU where the task is running. This can cause race conditions where two events change the next state and we see inconsistent values. E.g.:
[62] event_srs: 27: sleepable x sched_wakeup -> running (final)
[63] event_srs: 27: sleepable x sched_set_state_sleepable -> sleepable
[63] error_srs: 27: event sched_switch_suspend not expected in the state running
In this case the monitor fails because the event on CPU 62 wins against the one on CPU 63, although the correct state should have been sleepable, since the task get suspended.
Detect if the current state was modified by using try_cmpxchg while storing the next value. If it was, try again reading the current state. After a maximum number of failed retries, react by calling a special tracepoint, print on the console and reset the monitor.
Remove the functions da_monitor_curr_state() and da_monitor_set_state() as they only hide the underlying implementation in this case.
Monitors where this type of condition can occur must be able to account for racing events in any possible order, as we cannot know the winner.
--- include/linux/rv.h | 3 +- include/rv/da_monitor.h | 105 ++++++++++++++++++------------------- kernel/trace/rv/Kconfig | 5 ++ kernel/trace/rv/rv_trace.h | 24 +++++++++ 4 files changed, 83 insertions(+), 54 deletions(-)
include/linux/rv.h +2 -1
@@ -7,7 +7,8 @@#ifndef _LINUX_RV_H#define _LINUX_RV_H -#define MAX_DA_NAME_LEN 32 +#define MAX_DA_NAME_LEN 32 +#define MAX_DA_RETRY_RACING_EVENTS 3#ifdef CONFIG_RV #include <linux/bitops.h>
include/rv/da_monitor.h +52 -53
138 lines not on this page · raw message
kernel/trace/rv/Kconfig +5 -0
@@ -3,12 +3,17 @@ config RV_MON_EVENTS bool +config RV_MON_MAINTENANCE_EVENTS + bool + config DA_MON_EVENTS_IMPLICIT select RV_MON_EVENTS + select RV_MON_MAINTENANCE_EVENTS bool config DA_MON_EVENTS_ID select RV_MON_EVENTS + select RV_MON_MAINTENANCE_EVENTS bool config LTL_MON_EVENTS_ID
kernel/trace/rv/rv_trace.h +24 -0
@@ -176,6 +176,30 @@ DECLARE_EVENT_CLASS(error_ltl_monitor_id,#include <monitors/sleep/sleep_trace.h> // Add new monitors based on CONFIG_LTL_MON_EVENTS_ID here #endif /* CONFIG_LTL_MON_EVENTS_ID */ + +#ifdef CONFIG_RV_MON_MAINTENANCE_EVENTS +/* Tracepoint useful for monitors development, currenly only used in DA */ +TRACE_EVENT (rv_retries_error, + + TP_PROTO(char *name, int id), + +TP_ARGS (name, id), + +TP_STRUCT__entry ( + __string( name, name ) +__field ( int, id ) + ), + +TP_fast_assign ( + __assign_str(name); + __entry->id = id; + ), + + TP_printk(__stringify(MAX_DA_RETRY_RACING_EVENTS) +" retries reached, resetting monitor %s-%d" , +__get_str (name), __entry->id) +); +#endif /* CONFIG_RV_MON_MAINTENANCE_EVENTS */ #endif /* _TRACE_RV_H */ /* This part must be outside protection */
Signature · 1 line
2.50.1
Re: [PATCH v4 10/14] rv: Retry when da monitor detects race conditions
Quoted 63 lines · Gabriele Monaco
DA monitor can be accessed from multiple cores simultaneously, this is likely, for instance when dealing with per-task monitors reacting on events that do not always occur on the CPU where the task is running. This can cause race conditions where two events change the next state and we see inconsistent values. E.g.:
[62]
event_srs: 27: sleepable xsched_wakeup-> running (final)
[63]event_srs: 27: sleepable xsched_set_state_sleepable-> sleepable
[63]error_srs: 27: eventsched_switch_suspendnot expected in the state runningIn this case the monitor fails because the event on CPU 62 wins against the one on CPU 63, although the correct state should have been sleepable, since the task get suspended.
Detect if the current state was modified by using
try_cmpxchgwhile storing the next value. If it was, try again reading the current state. After a maximum number of failed retries, react by calling a special tracepoint, print on the console and reset the monitor.Remove the functions
da_monitor_curr_state()andda_monitor_set_state()as they only hide the underlying implementation in this case.Monitors where this type of condition can occur must be able to account for racing events in any possible order, as we cannot know the winner.
Cc Ingo Molnar <[email protected]>Cc Peter Zijlstra <[email protected]>Signed-off-by Gabriele Monaco <[email protected]>---
static inline bool \
da_event_##name(structda_monitor*da_mon, enum events_##name event) \
{ \include/rv/da_monitor.h +18
-12- type curr_state = da_monitor_curr_state_##name(da_mon); \ - type next_state = model_get_next_state_##name(curr_state, event); \ - \ - if (next_state != INVALID_STATE) { \ - da_monitor_set_state_##name(da_mon, next_state); \ - \ - trace_event_##name(model_get_state_name_##name(curr_state), \ - model_get_event_name_##name(event), \ - model_get_state_name_##name(next_state), \ - model_is_final_state_##name(next_state)); \ - \ - return true; \ + enum states_##name curr_state, next_state; \ + \ + curr_state = READ_ONCE(da_mon->curr_state); \ + for (int i = 0; i < MAX_DA_RETRY_RACING_EVENTS; i++) { \ + next_state = model_get_next_state_##name(curr_state, event); \ + if (next_state == INVALID_STATE) { \ + cond_react_##name(curr_state, event); \ + trace_error_##name(model_get_state_name_##name(curr_state), \ + model_get_event_name_##name(event)); \ + return false; \ + } \ + if (likely(try_cmpxchg(&da_mon->curr_state, &curr_state, next_state))) { \ + trace_event_##name(model_get_state_name_##name(curr_state), \ + model_get_event_name_##name(event), \ + model_get_state_name_##name(next_state), \ + model_is_final_state_##name(next_state)); \ + return true; \ + } \
include/rv/da_monitor.h static inline bool da_monitor_handling_event_##name(struct da_monitor *da_mon) · line 145
Quoted 10 lines
include/rv/da_monitor.h +3
-5} \ \ - cond_react_##name(curr_state, event); \ - \ - trace_error_##name(model_get_state_name_##name(curr_state), \ - model_get_event_name_##name(event)); \ - \ + trace_rv_retries_error(#name, smp_processor_id()); \ + pr_warn("rv: " __stringify(MAX_DA_RETRY_RACING_EVENTS) \ + " retries reached, resetting monitor %s", #name); \
smp_processor_id() requires preemption to be disabled.
At the moment, trace point handler is called with preemption disabled, so we are fine. But there is plan to change that:
https://lore.kernel.org/lkml/[email protected]/T/#u
Perhaps use get_cpu() and put_cpu() instead?
Nam
Gabriele Monaco
maintainerRe: [PATCH v4 10/14] rv: Retry when da monitor detects race conditions
Quoted 132 lines · Nam Cao
Quoted 127 lines · Gabriele Monaco
DA monitor can be accessed from multiple cores simultaneously, this is
likely, for instance when dealing with per-task monitors reacting on
events that do not always occur on the CPU where the task is running.
This can cause race conditions where two events change the next state
and we see inconsistent values. E.g.:[62]
event_srs: 27: sleepable xsched_wakeup-> running (final)
[63]event_srs: 27: sleepable xsched_set_state_sleepable-> sleepable
[63]error_srs: 27: eventsched_switch_suspendnot expected in the state runningIn this case the monitor fails because the event on CPU 62 wins against
the one on CPU 63, although the correct state should have been sleepable, since the task get suspended.Detect if the current state was modified by using
try_cmpxchgwhile storing the next value. If it was, try again reading the current state.
After a maximum number of failed retries, react by calling a special
tracepoint, print on the console and reset the monitor.Remove the functions
da_monitor_curr_state()andda_monitor_set_state()
as they only hide the underlying implementation in this case.Monitors where this type of condition can occur must be able to account
for racing events in any possible order, as we cannot know the winner.Cc Ingo Molnar <[email protected]>Cc Peter Zijlstra <[email protected]>Signed-off-by Gabriele Monaco <[email protected]>---
static inline
bool \
da_event_##name(structda_monitor*da_mon, enum events_##name
event) \{ \(quoted diff) +0
-1- type curr_state =da_monitor_curr_state_##name(da_mon); \
(quoted diff) +0
-1- type next_state = model_get_next_state_##name(curr_state,event); \
-\(quoted diff) +0
-1- if (next_state != INVALID_STATE){ \
(quoted diff) +0
-1- da_monitor_set_state_##name(da_mon,
next_state); \
-\ - trace_event_##name(model_get_state_name_##name(curr_state), \(quoted diff) +0
-1-model_get_event_name_##name(event), \
(quoted diff) +0
-1-model_get_state_name_##name(next_state), \
(quoted diff) +0
-1-model_is_final_state_##name(next_state)); \
-\(quoted diff) +0
-1- returntrue; \
(quoted diff) +1
-0+ enum states_##name curr_state,
next_state; \(quoted diff) +1
-0+\(quoted diff) +1
-0+ curr_state = READ_ONCE(da_mon-Quoted 1 line
curr_state); \(quoted diff) +1
-0+ for (int i = 0; i < MAX_DA_RETRY_RACING_EVENTS; i++){ \
(quoted diff) +1
-0+ next_state =model_get_next_state_##name(curr_state, event); \
(quoted diff) +1
-0+ if (next_state == INVALID_STATE){ \
(quoted diff) +1
-0+ cond_react_##name(curr_state,event); \
(quoted diff) +1
-0+ trace_error_##name(model_get_state_name_##
name(curr_state), \(quoted diff) +1
-0+model_get_event_name_##name(event)); \
(quoted diff) +1
-0+ returnfalse; \
(quoted diff) +1
-0+ }\(quoted diff) +1
-0+ if (likely(try_cmpxchg(&da_mon->curr_state,
&curr_state,next_state))) { \(quoted diff) +1
-0+ trace_event_##name(model_get_state_name_##
name(curr_state), \(quoted diff) +1
-0+model_get_event_name_##name(event), \
(quoted diff) +1
-0+model_get_state_name_##name(next_state), \
(quoted diff) +1
-0+model_is_final_state_##name(next_state)); \
(quoted diff) +1
-0+ returntrue; \
(quoted diff) +1
-0+ }\ } \ \(quoted diff) +0
-1- cond_react_##name(curr_state,event); \
-\ - trace_error_##name(model_get_state_name_##name(curr_state), \(quoted diff) +0
-1-model_get_event_name_##name(event)); \
-\(quoted diff) +1
-0+ trace_rv_retries_error(#name,
smp_processor_id()); \(quoted diff) +1
-0+ pr_warn("rv: "
__stringify(MAX_DA_RETRY_RACING_EVENTS)\(quoted diff) +1
-0+ " retries reached, resetting monitor %s",Quoted 1 line
#name); \
smp_processor_id()requires preemption to be disabled.
Quoted 6 lines
At the moment, trace point handler is called with preemption disabled, so
we are fine. But there is plan to change that:
https://lore.kernel.org/lkml/[email protected]/T/#uPerhaps use
get_cpu()andput_cpu()instead?
Mmh, then I'd need to execute them only if the tracepoint is enabled, I'm not sure it's worth the effort..
I wanted to avoid creating two different tracepoints (implicit and id), but I might have to. The CPU is rarely needed there since (for now) per-cpu monitors assume event cpu and monitor cpu are the same.
I'll have a thought about it, thanks for pointing it out!
Gabriele
Gabriele Monaco
maintainerPATCH v4 11/14 sched: Adapt sched tracepoints for RV task model
In reply to Gabriele MonacoAdd the following tracepoint:
* sched_set_need_resched(tsk, cpu, tif)
Called when a task is set the need resched [lazy] flag
Remove the unused ip parameter from sched_entry and sched_exit and alter sched_entry to have a value of preempt consistent with the one used in sched_switch.
Also adapt all monitors using sched_{entry,exit} to avoid breaking build.
These tracepoints are useful to describe the Linux task model and are adapted from the patches by Daniel Bristot de Oliveira
(https://bristot.me/linux-task-model/).
--- include/linux/sched.h | 7 ++++++- include/trace/events/sched.h | 12 ++++++++---- kernel/sched/core.c | 13 ++++++++++--- kernel/trace/rv/monitors/sco/sco.c | 4 ++-- kernel/trace/rv/monitors/scpd/scpd.c | 4 ++-- kernel/trace/rv/monitors/sncid/sncid.c | 4 ++-- kernel/trace/rv/monitors/snep/snep.c | 4 ++-- kernel/trace/rv/monitors/tss/tss.c | 4 ++-- 8 files changed, 34 insertions(+), 18 deletions(-)
include/linux/sched.h +6 -1
@@ -340,9 +340,11 @@ extern void io_schedule_finish(int token);extern long io_schedule_timeout (long timeout);extern void io_schedule (void ); -/* wrapper function to trace from this header file */ +/* wrapper functions to trace from this header file */ DECLARE_TRACEPOINT (sched_set_state_tp);extern void __trace_set_current_state (int state_value); +DECLARE_TRACEPOINT (sched_set_need_resched_tp); +extern void __trace_set_need_resched (struct task_struct *curr,int tif);/** * struct prev_cputime - snapshot of system and user cputime
@@ -2026,6 +2028,9 @@ static inline int test_tsk_thread_flag(struct task_struct *tsk, int flag)static inline void set_tsk_need_resched (struct task_struct *tsk) { +if (tracepoint_enabled (sched_set_need_resched_tp) && + !test_tsk_thread_flag (tsk,TIF_NEED_RESCHED )) +__trace_set_need_resched (tsk,TIF_NEED_RESCHED );set_tsk_thread_flag (tsk,TIF_NEED_RESCHED ); }
include/trace/events/sched.h +8 -4
@@ -882,18 +882,22 @@ DECLARE_TRACE(sched_compute_energy, TP_ARGS(p, dst_cpu, energy, max_util, busy_time));DECLARE_TRACE (sched_entry, - TP_PROTO(bool preempt, unsigned long ip), -TP_ARGS (preempt, ip)); + TP_PROTO(bool preempt), +TP_ARGS (preempt));DECLARE_TRACE (sched_exit, - TP_PROTO(bool is_switch, unsigned long ip), -TP_ARGS (is_switch, ip)); + TP_PROTO(bool is_switch), +TP_ARGS (is_switch));DECLARE_TRACE_CONDITION (sched_set_state, TP_PROTO(struct task_struct *tsk, int state),TP_ARGS (tsk, state),TP_CONDITION (!!(tsk->__state) != !!state)); +DECLARE_TRACE (sched_set_need_resched, + TP_PROTO(struct task_struct *tsk, int cpu, int tif), +TP_ARGS (tsk, cpu, tif)); +#endif /* _TRACE_SCHED_H */ /* This part must be outside protection */
kernel/sched/core.c +10 -3
@@ -1113,6 +1113,7 @@ static void __resched_curr(struct rq *rq, int tif) cpu =cpu_of (rq); +trace_sched_set_need_resched_tp (curr, cpu, tif);if (cpu ==smp_processor_id ()) {set_ti_thread_flag (cti, tif);if (tif ==TIF_NEED_RESCHED )
@@ -1128,6 +1129,11 @@ static void __resched_curr(struct rq *rq, int tif) } } +void __trace_set_need_resched (struct task_struct *curr,int tif) +{ +trace_sched_set_need_resched_tp (curr,smp_processor_id (), tif); +} +void resched_curr (struct rq *rq) {__resched_curr (rq,TIF_NEED_RESCHED );
@@ -5279,7 +5285,7 @@ asmlinkage __visible void schedule_tail(struct task_struct *prev)* switched the context for the first time. It is returning from * schedule for the first time in this path. */ -trace_sched_exit_tp (true,CALLER_ADDR0 ); +trace_sched_exit_tp (true);preempt_enable ();if (current->set_child_tid)
@@ -6822,7 +6828,8 @@ static void __sched notrace __schedule(int sched_mode)struct rq *rq;int cpu; -trace_sched_entry_tp (preempt,CALLER_ADDR0 ); +/* Trace preemptions consistently with task switches */ +trace_sched_entry_tp (sched_mode ==SM_PREEMPT ); cpu =smp_processor_id (); rq =cpu_rq (cpu);
@@ -6961,7 +6968,7 @@ static void __sched notrace __schedule(int sched_mode)__balance_callbacks (rq);raw_spin_rq_unlock_irq (rq); } -trace_sched_exit_tp (is_switch,CALLER_ADDR0 ); +trace_sched_exit_tp (is_switch); }void __noreturn do_task_dead (void )
kernel/trace/rv/monitors/sco/sco.c +2 -2
@@ -24,12 +24,12 @@ static void handle_sched_set_state(void *data, struct task_struct *tsk, int statda_handle_start_event_sco (sched_set_state_sco); } -static void handle_schedule_entry (void *data,bool preempt,unsigned long ip) +static void handle_schedule_entry (void *data,bool preempt) {da_handle_event_sco (schedule_entry_sco); } -static void handle_schedule_exit (void *data,bool is_switch,unsigned long ip) +static void handle_schedule_exit (void *data,bool is_switch) {da_handle_start_event_sco (schedule_exit_sco); }
kernel/trace/rv/monitors/scpd/scpd.c +2 -2
@@ -30,12 +30,12 @@ static void handle_preempt_enable(void *data, unsigned long ip, unsigned long pada_handle_start_event_scpd (preempt_enable_scpd); } -static void handle_schedule_entry (void *data,bool preempt,unsigned long ip) +static void handle_schedule_entry (void *data,bool preempt) {da_handle_event_scpd (schedule_entry_scpd); } -static void handle_schedule_exit (void *data,bool is_switch,unsigned long ip) +static void handle_schedule_exit (void *data,bool is_switch) {da_handle_event_scpd (schedule_exit_scpd); }
kernel/trace/rv/monitors/sncid/sncid.c +2 -2
@@ -30,12 +30,12 @@ static void handle_irq_enable(void *data, unsigned long ip, unsigned long parentda_handle_start_event_sncid (irq_enable_sncid); } -static void handle_schedule_entry (void *data,bool preempt,unsigned long ip) +static void handle_schedule_entry (void *data,bool preempt) {da_handle_start_event_sncid (schedule_entry_sncid); } -static void handle_schedule_exit (void *data,bool is_switch,unsigned long ip) +static void handle_schedule_exit (void *data,bool is_switch) {da_handle_start_event_sncid (schedule_exit_sncid); }
kernel/trace/rv/monitors/snep/snep.c +2 -2
@@ -30,12 +30,12 @@ static void handle_preempt_enable(void *data, unsigned long ip, unsigned long pada_handle_start_event_snep (preempt_enable_snep); } -static void handle_schedule_entry (void *data,bool preempt,unsigned long ip) +static void handle_schedule_entry (void *data,bool preempt) {da_handle_event_snep (schedule_entry_snep); } -static void handle_schedule_exit (void *data,bool is_switch,unsigned long ip) +static void handle_schedule_exit (void *data,bool is_switch) {da_handle_start_event_snep (schedule_exit_snep); }
kernel/trace/rv/monitors/tss/tss.c +2 -2
@@ -27,12 +27,12 @@ static void handle_sched_switch(void *data, bool preempt,da_handle_event_tss (sched_switch_tss); } -static void handle_schedule_entry (void *data,bool preempt,unsigned long ip) +static void handle_schedule_entry (void *data,bool preempt) {da_handle_event_tss (schedule_entry_tss); } -static void handle_schedule_exit (void *data,bool is_switch,unsigned long ip) +static void handle_schedule_exit (void *data,bool is_switch) {da_handle_start_event_tss (schedule_exit_tss); }
Signature · 1 line
2.50.1
Gabriele Monaco
maintainerPATCH v4 12/14 Replace tss and sncid monitors with more complete sts
In reply to Gabriele MonacoThe tss monitor currently guarantees task switches can happen only while scheduling, whereas the sncid monitor enforces scheduling occurs with interrupt disabled.
Replace the monitors with a more comprehensive specification which implies both but also ensures that:
* each scheduler call disable interrupts to switch
* each task switch happens with interrupts disabled
fixup sts remove sncid
---
Documentation/trace/rv/monitor_sched.rst | 87 +++++-----
kernel/trace/rv/Kconfig | 3 +-
kernel/trace/rv/Makefile | 3 +-
kernel/trace/rv/monitors/sncid/Kconfig | 15 --
kernel/trace/rv/monitors/sncid/sncid.c | 95 -----------
kernel/trace/rv/monitors/sncid/sncid.h | 49 ------
kernel/trace/rv/monitors/sncid/sncid_trace.h | 15 --
kernel/trace/rv/monitors/sts/Kconfig | 19 +++
kernel/trace/rv/monitors/sts/sts.c | 156 ++++++++++++++++++
kernel/trace/rv/monitors/sts/sts.h | 117 +++++++++++++
.../{tss/tss_trace.h => sts/sts_trace.h} | 8 +-
kernel/trace/rv/monitors/tss/Kconfig | 14 --
kernel/trace/rv/monitors/tss/tss.c | 90 ----------
kernel/trace/rv/monitors/tss/tss.h | 47 ------
kernel/trace/rv/rv_trace.h | 3 +-
tools/verification/models/sched/sncid.dot | 18 --
tools/verification/models/sched/sts.dot | 38 +++++
tools/verification/models/sched/tss.dot | 18 --
18 files changed, 385 insertions(+), 410 deletions(-)
delete mode 100644 kernel/trace/rv/monitors/sncid/Kconfig
delete mode 100644 kernel/trace/rv/monitors/sncid/sncid.c
delete mode 100644 kernel/trace/rv/monitors/sncid/sncid.h
delete mode 100644 kernel/trace/rv/monitors/sncid/sncid_trace.h
create mode 100644 kernel/trace/rv/monitors/sts/Kconfig
create mode 100644 kernel/trace/rv/monitors/sts/sts.c
create mode 100644 kernel/trace/rv/monitors/sts/sts.h
rename kernel/trace/rv/monitors/{tss/tss_trace.h => sts/sts_trace.h} (67%)
delete mode 100644 kernel/trace/rv/monitors/tss/Kconfig
delete mode 100644 kernel/trace/rv/monitors/tss/tss.c
delete mode 100644 kernel/trace/rv/monitors/tss/tss.h
delete mode 100644 tools/verification/models/sched/sncid.dot
create mode 100644 tools/verification/models/sched/sts.dot
delete mode 100644 tools/verification/models/sched/tss.dotDocumentation/trace/rv/monitor_sched.rst +48 -39
@@ -40,26 +40,6 @@ defined in by Daniel Bristot in [1]. Currently we included the following: -Monitor tss -~~~~~~~~~~~ - -The task switch while scheduling (tss) monitor ensures a task switch happens -only in scheduling context, that is inside a call to `__schedule`:: - - | - | - v - +-----------------+ - | thread | <+ - +-----------------+ | - | | - | schedule_entry | schedule_exit - v | - sched_switch | - +--------------- | - | sched | - +--------------> -+ - Monitor sco ~~~~~~~~~~~
@@ -144,26 +124,55 @@ does not enable preemption:: | scheduling_contex -+ -Monitor sncid -~~~~~~~~~~~~~ - -The schedule not called with interrupt disabled (sncid) monitor ensures -schedule is not called with interrupt disabled:: +Monitor sts +~~~~~~~~~~~ - | - | - v - schedule_entry +--------------+ - schedule_exit | | - +----------------- | can_sched | - | | | - +----------------> | | <+ - +--------------+ | - | | - | irq_disable | irq_enable - v | - | - cant_sched -+ +The schedule implies task switch (sts) monitor ensures a task switch happens +only in scheduling context and up to once, as well as scheduling occurs with +interrupts enabled but no task switch can happen before interrupts are +disabled. When the next task picked for execution is the same as the previously +running one, no real task switch occurs but interrupts are disabled nonetheless:: + + irq_entry | + +----+ | + v | v + +------------+ irq_enable #===================# irq_disable + | | ------------> H H irq_entry + | cant_sched | <------------ H H irq_enable + | | irq_disable H can_sched H --------------+ + +------------+ H H | + H H | + +---------------> H H <-------------+ + | #===================# + | | + schedule_exit | schedule_entry + | v + | +-------------------+ irq_enable + | | scheduling | <---------------+ + | +-------------------+ | + | | | + | | irq_disable +--------+ irq_entry + | v | | --------+ + | +-------------------+ irq_entry | in_irq | | + | | | -----------> | | <-------+ + | | disable_to_switch | +--------+ + | | | --+ + | +-------------------+ | + | | | + | | sched_switch | + | v | + | +-------------------+ | + | | switching | | irq_enable + | +-------------------+ | + | | | + | | irq_enable | + | v | + | +-------------------+ | + +-- | enable_to_exit | <-+ + +-------------------+ + ^ | irq_disable + | | irq_entry + +---------------+ irq_enable References ----------
kernel/trace/rv/Kconfig +1 -2
@@ -50,12 +50,11 @@ source "kernel/trace/rv/monitors/wip/Kconfig" source "kernel/trace/rv/monitors/wwnr/Kconfig" source "kernel/trace/rv/monitors/sched/Kconfig" -source "kernel/trace/rv/monitors/tss/Kconfig" source "kernel/trace/rv/monitors/sco/Kconfig" source "kernel/trace/rv/monitors/snroc/Kconfig" source "kernel/trace/rv/monitors/scpd/Kconfig" source "kernel/trace/rv/monitors/snep/Kconfig" -source "kernel/trace/rv/monitors/sncid/Kconfig" +source "kernel/trace/rv/monitors/sts/Kconfig" # Add new sched monitors here source "kernel/trace/rv/monitors/rtapp/Kconfig"
kernel/trace/rv/Makefile +1 -2
@@ -6,15 +6,14 @@ obj-$(CONFIG_RV) += rv.o obj-$(CONFIG_RV_MON_WIP) += monitors/wip/wip.o obj-$(CONFIG_RV_MON_WWNR) += monitors/wwnr/wwnr.o obj-$(CONFIG_RV_MON_SCHED) += monitors/sched/sched.o -obj-$(CONFIG_RV_MON_TSS) += monitors/tss/tss.o obj-$(CONFIG_RV_MON_SCO) += monitors/sco/sco.o obj-$(CONFIG_RV_MON_SNROC) += monitors/snroc/snroc.o obj-$(CONFIG_RV_MON_SCPD) += monitors/scpd/scpd.o obj-$(CONFIG_RV_MON_SNEP) += monitors/snep/snep.o -obj-$(CONFIG_RV_MON_SNCID) += monitors/sncid/sncid.o obj-$(CONFIG_RV_MON_RTAPP) += monitors/rtapp/rtapp.o obj-$(CONFIG_RV_MON_PAGEFAULT) += monitors/pagefault/pagefault.o obj-$(CONFIG_RV_MON_SLEEP) += monitors/sleep/sleep.o +obj-$(CONFIG_RV_MON_STS) += monitors/sts/sts.o # Add new monitors here obj-$(CONFIG_RV_REACTORS) += rv_reactors.o obj-$(CONFIG_RV_REACT_PRINTK) += reactor_printk.o
kernel/trace/rv/monitors/sncid/Kconfig deleted +0 -15
@@ -1,15 +0,0 @@ -# SPDX-License-Identifier: GPL-2.0-only -# -config RV_MON_SNCID - depends on RV - depends on TRACE_IRQFLAGS - depends on RV_MON_SCHED - default y - select DA_MON_EVENTS_IMPLICIT - bool "sncid monitor" - help - Monitor to ensure schedule is not called with interrupt disabled. - This monitor is part of the sched monitors collection. - - For further information, see: - Documentation/trace/rv/monitor_sched.rst
kernel/trace/rv/monitors/sncid/sncid.c deleted +0 -95
95 lines not on this page · raw message
kernel/trace/rv/monitors/sncid/sncid.h deleted +0 -49
49 lines not on this page · raw message
kernel/trace/rv/monitors/sncid/sncid_trace.h deleted +0 -15
@@ -1,15 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ - -/* -* Snippet to be included in rv_trace.h -*/ - -#ifdef CONFIG_RV_MON_SNCID -DEFINE_EVENT (event_da_monitor, event_sncid, - TP_PROTO(char *state, char *event, char *next_state, bool final_state), -TP_ARGS (state, event, next_state, final_state)); - -DEFINE_EVENT (error_da_monitor, error_sncid, -TP_PROTO (char *state, char *event), -TP_ARGS (state, event)); -#endif /* CONFIG_RV_MON_SNCID */
kernel/trace/rv/monitors/sts/Kconfig new file +19 -0
@@ -0,0 +1,19 @@ +# SPDX-License-Identifier: GPL-2.0-only +# +config RV_MON_STS + depends on RV + depends on TRACE_IRQFLAGS + depends on RV_MON_SCHED + default y + select DA_MON_EVENTS_IMPLICIT + bool "sts monitor" + help + Monitor to ensure relationships between scheduler and task switches + * the scheduler is called and returns with interrupts disabled + * each call to the scheduler has up to one switch + * switches only happen inside the scheduler + * each call to the scheduler disables interrupts to switch + This monitor is part of the sched monitors collection. + + For further information, see: + Documentation/trace/rv/monitor_sched.rst
kernel/trace/rv/monitors/sts/sts.c new file +156 -0
156 lines not on this page · raw message
kernel/trace/rv/monitors/sts/sts.h new file +117 -0
117 lines not on this page · raw message
kernel/trace/rv/monitors/sts/sts_trace.h renamed from kernel/trace/rv/monitors/tss/tss_trace.h +4 -4
@@ -4,12 +4,12 @@* Snippet to be included in rv_trace.h */ -#ifdef CONFIG_RV_MON_TSS -DEFINE_EVENT (event_da_monitor, event_tss, +#ifdef CONFIG_RV_MON_STS +DEFINE_EVENT (event_da_monitor, event_sts, TP_PROTO(char *state, char *event, char *next_state, bool final_state),TP_ARGS (state, event, next_state, final_state)); -DEFINE_EVENT (error_da_monitor, error_tss, +DEFINE_EVENT (error_da_monitor, error_sts,TP_PROTO (char *state, char *event),TP_ARGS (state, event)); -#endif /* CONFIG_RV_MON_TSS */ +#endif /* CONFIG_RV_MON_STS */
kernel/trace/rv/monitors/tss/Kconfig deleted +0 -14
@@ -1,14 +0,0 @@ -# SPDX-License-Identifier: GPL-2.0-only -# -config RV_MON_TSS - depends on RV - depends on RV_MON_SCHED - default y - select DA_MON_EVENTS_IMPLICIT - bool "tss monitor" - help - Monitor to ensure sched_switch happens only in scheduling context. - This monitor is part of the sched monitors collection. - - For further information, see: - Documentation/trace/rv/monitor_sched.rst
kernel/trace/rv/monitors/tss/tss.c deleted +0 -90
90 lines not on this page · raw message
kernel/trace/rv/monitors/tss/tss.h deleted +0 -47
47 lines not on this page · raw message
kernel/trace/rv/rv_trace.h +1 -2
@@ -58,11 +58,10 @@ DECLARE_EVENT_CLASS(error_da_monitor, );#include <monitors/wip/wip_trace.h> -#include <monitors/tss/tss_trace.h> #include <monitors/sco/sco_trace.h> #include <monitors/scpd/scpd_trace.h> #include <monitors/snep/snep_trace.h> -#include <monitors/sncid/sncid_trace.h> +#include <monitors/sts/sts_trace.h> // Add new monitors based on CONFIG_DA_MON_EVENTS_IMPLICIT here #endif /* CONFIG_DA_MON_EVENTS_IMPLICIT */
tools/verification/models/sched/sncid.dot deleted +0 -18
@@ -1,18 +0,0 @@ -digraph state_automaton { - center = true; - size = "7,11"; - {node [shape = plaintext, style=invis, label=""] "__init_can_sched"}; - {node [shape = ellipse] "can_sched"}; - {node [shape = plaintext] "can_sched"}; - {node [shape = plaintext] "cant_sched"}; - "__init_can_sched" -> "can_sched"; - "can_sched" [label = "can_sched", color = green3]; - "can_sched" -> "can_sched" [ label = "schedule_entry\nschedule_exit" ]; - "can_sched" -> "cant_sched" [ label = "irq_disable" ]; - "cant_sched" [label = "cant_sched"]; - "cant_sched" -> "can_sched" [ label = "irq_enable" ]; - { rank = min ; - "__init_can_sched"; - "can_sched"; - } -}
tools/verification/models/sched/sts.dot new file +38 -0
@@ -0,0 +1,38 @@ +digraph state_automaton { + center = true; + size = "7,11"; + {node [shape = plaintext, style=invis, label=""] "__init_can_sched"}; + {node [shape = doublecircle] "can_sched"}; + {node [shape = circle] "can_sched"}; + {node [shape = circle] "cant_sched"}; + {node [shape = circle] "disable_to_switch"}; + {node [shape = circle] "enable_to_exit"}; + {node [shape = circle] "in_irq"}; + {node [shape = circle] "scheduling"}; + {node [shape = circle] "switching"}; + "__init_can_sched" -> "can_sched"; + "can_sched" [label = "can_sched", color = green3]; + "can_sched" -> "cant_sched" [ label = "irq_disable" ]; + "can_sched" -> "scheduling" [ label = "schedule_entry" ]; + "cant_sched" [label = "cant_sched"]; + "cant_sched" -> "can_sched" [ label = "irq_enable" ]; + "cant_sched" -> "cant_sched" [ label = "irq_entry" ]; + "disable_to_switch" [label = "disable_to_switch"]; + "disable_to_switch" -> "enable_to_exit" [ label = "irq_enable" ]; + "disable_to_switch" -> "in_irq" [ label = "irq_entry" ]; + "disable_to_switch" -> "switching" [ label = "sched_switch" ]; + "enable_to_exit" [label = "enable_to_exit"]; + "enable_to_exit" -> "can_sched" [ label = "schedule_exit" ]; + "enable_to_exit" -> "enable_to_exit" [ label = "irq_disable\nirq_entry\nirq_enable" ]; + "in_irq" [label = "in_irq"]; + "in_irq" -> "in_irq" [ label = "irq_entry" ]; + "in_irq" -> "scheduling" [ label = "irq_enable" ]; + "scheduling" [label = "scheduling"]; + "scheduling" -> "disable_to_switch" [ label = "irq_disable" ]; + "switching" [label = "switching"]; + "switching" -> "enable_to_exit" [ label = "irq_enable" ]; + { rank = min ; + "__init_can_sched"; + "can_sched"; + } +}
tools/verification/models/sched/tss.dot deleted +0 -18
@@ -1,18 +0,0 @@ -digraph state_automaton { - center = true; - size = "7,11"; - {node [shape = plaintext] "sched"}; - {node [shape = plaintext, style=invis, label=""] "__init_thread"}; - {node [shape = ellipse] "thread"}; - {node [shape = plaintext] "thread"}; - "__init_thread" -> "thread"; - "sched" [label = "sched"]; - "sched" -> "sched" [ label = "sched_switch" ]; - "sched" -> "thread" [ label = "schedule_exit" ]; - "thread" [label = "thread", color = green3]; - "thread" -> "sched" [ label = "schedule_entry" ]; - { rank = min ; - "__init_thread"; - "thread"; - } -}
Signature · 1 line
2.50.1
Re: [PATCH v4 12/14] rv: Replace tss and sncid monitors with more complete sts
Quoted 9 lines · Gabriele Monaco
The tss monitor currently guarantees task switches can happen only while scheduling, whereas the sncid monitor enforces scheduling occurs with interrupt disabled.
Replace the monitors with a more comprehensive specification which implies both but also ensures that:
* each scheduler call disable interrupts to switch
* each task switch happens with interrupts disabled
Quoted 4 lines
Cc Ingo Molnar <[email protected]>Cc Peter Zijlstra <[email protected]>fixup sts remove sncid
Is this here by accident?
I cannot comment on the model. The CONFIG_X86_LOCAL_APIC case looks complex, but I cannot comment on that either.
But things look fine from RV perspective, so:
Gabriele Monaco
maintainerRe: [PATCH v4 12/14] rv: Replace tss and sncid monitors with more complete sts
Quoted 12 lines · Nam Cao
Quoted 11 lines · Gabriele Monaco
The tss monitor currently guarantees task switches can happen only while
scheduling, whereas the sncid monitor enforces scheduling occurs with
interrupt disabled.Replace the monitors with a more comprehensive specification which implies both but also ensures that:
* each scheduler call disable interrupts to switch
* each task switch happens with interrupts disabled
Quoted 7 lines
Quoted 4 lines
Cc Ingo Molnar <[email protected]>Cc Peter Zijlstra <[email protected]>fixup sts remove sncid
Is this here by accident?
Damn, again.. thanks for spotting.
Quoted 2 lines
I cannot comment on the model. The
CONFIG_X86_LOCAL_APICcase looks complex, but I cannot comment on that either.
Do you mean the amount of tracepoints or the state in the monitor?
As far as I'm aware some special IRQs on x86 use those tracepoints, and I needed to use all of them not to miss real interrupts, which I need to understand if interrupts where disabled programmatically or by a hardware IRQ.
Quoted 3 lines
But things look fine from RV perspective, so:
Acked-by Nam Cao <[email protected]>
Thanks!
Gabriele
Gabriele Monaco
maintainerPATCH v4 13/14 Add nrp and sssw per-task monitors
In reply to Gabriele MonacoAdd 2 per-task monitors as part of the sched model:
* nrp: need-resched preempts
Monitor to ensure preemption requires need resched.
* sssw: set state sleep and wakeup
Monitor to ensure sched_set_state to sleepable leads to sleeping and sleeping tasks require wakeup.
--- Documentation/trace/rv/monitor_sched.rst | 167 +++++++++++++++++++++ kernel/trace/rv/Kconfig | 2 + kernel/trace/rv/Makefile | 2 + kernel/trace/rv/monitors/nrp/Kconfig | 16 ++ kernel/trace/rv/monitors/nrp/nrp.c | 138 +++++++++++++++++ kernel/trace/rv/monitors/nrp/nrp.h | 75 +++++++++ kernel/trace/rv/monitors/nrp/nrp_trace.h | 15 ++ kernel/trace/rv/monitors/sched/Kconfig | 1 + kernel/trace/rv/monitors/sssw/Kconfig | 15 ++ kernel/trace/rv/monitors/sssw/sssw.c | 116 ++++++++++++++ kernel/trace/rv/monitors/sssw/sssw.h | 105 +++++++++++++ kernel/trace/rv/monitors/sssw/sssw_trace.h | 15 ++ kernel/trace/rv/rv_trace.h | 2 + tools/verification/models/sched/nrp.dot | 29 ++++ tools/verification/models/sched/sssw.dot | 30 ++++ 15 files changed, 728 insertions(+) create mode 100644 kernel/trace/rv/monitors/nrp/Kconfig create mode 100644 kernel/trace/rv/monitors/nrp/nrp.c create mode 100644 kernel/trace/rv/monitors/nrp/nrp.h create mode 100644 kernel/trace/rv/monitors/nrp/nrp_trace.h create mode 100644 kernel/trace/rv/monitors/sssw/Kconfig create mode 100644 kernel/trace/rv/monitors/sssw/sssw.c create mode 100644 kernel/trace/rv/monitors/sssw/sssw.h create mode 100644 kernel/trace/rv/monitors/sssw/sssw_trace.h create mode 100644 tools/verification/models/sched/nrp.dot create mode 100644 tools/verification/models/sched/sssw.dot
Documentation/trace/rv/monitor_sched.rst +167 -0
@@ -174,6 +174,173 @@ running one, no real task switch occurs but interrupts are disabled nonetheless: | | irq_entry +---------------+ irq_enable +Monitor nrp +----------- + +The need resched preempts (nrp) monitor ensures preemption requires +``need_resched``. Only kernel preemption is considered, since preemption +while returning to userspace, for this monitor, is indistinguishable from +``sched_switch_yield`` (described in the sssw monitor). +A kernel preemption is whenever ``__schedule`` is called with the preemption +flag set to true (e.g. from preempt_enable or exiting from interrupts). This +type of preemption occurs after the need for ``rescheduling`` has been set. +This is not valid for the *lazy* variant of the flag, which causes only +userspace preemption. +A ``schedule_entry_preempt`` may involve a task switch or not, in the latter +case, a task goes through the scheduler from a preemption context but it is +picked as the next task to run. Since the scheduler runs, this clears the need +to reschedule. The ``any_thread_running`` state does not imply the monitored +task is not running as this monitor does not track the outcome of scheduling. + +In theory, a preemption can only occur after the ``need_resched`` flag is set. In +practice, however, it is possible to see a preemption where the flag is not +set. This can happen in one specific condition:: + + need_resched + preempt_schedule() + preempt_schedule_irq() + __schedule() + !need_resched + __schedule() + +In the situation above, standard preemption starts (e.g. from preempt_enable +when the flag is set), an interrupt occurs before scheduling and, on its exit +path, it schedules, which clears the ``need_resched`` flag. +When the preempted task runs again, the standard preemption started earlier +resumes, although the flag is no longer set. The monitor considers this a +``nested_preemption``, this allows another preemption without re-setting the +flag. This condition relaxes the monitor constraints and may catch false +negatives (i.e. no real ``nested_preemptions``) but makes the monitor more +robust and able to validate other scenarios. +For simplicity, the monitor starts in ``preempt_irq``, although no interrupt +occurred, as the situation above is hard to pinpoint:: + + schedule_entry + irq_entry #===========================================# + +-------------------------- H H + | H H + +-------------------------> H any_thread_running H + H H + +-------------------------> H H + | #===========================================# + | schedule_entry | ^ + | schedule_entry_preempt | sched_need_resched | schedule_entry + | | schedule_entry_preempt + | v | + | +----------------------+ | + | +--- | | | + | sched_need_resched | | rescheduling | -+ + | +--> | | + | +----------------------+ + | | irq_entry + | v + | +----------------------+ + | | | ---+ + | ---> | | | sched_need_resched + | | preempt_irq | | irq_entry + | | | <--+ + | | | <--+ + | +----------------------+ | + | | schedule_entry | sched_need_resched + | | schedule_entry_preempt | + | v | + | +-----------------------+ | + +-------------------------- | nested_preempt | --+ + +-----------------------+ + ^ irq_entry | + +-------------------+ + +Due to how the ``need_resched`` flag on the preemption count works on arm64, +this monitor is unstable on that architecture, as it often records preemption +when the flag is not set, even in presence of the workaround above. +For the time being, the monitor is disabled by default on arm64. + +Monitor sssw +------------ + +The set state sleep and wakeup (sssw) monitor ensures ``set_state`` to +sleepable leads to sleeping and sleeping tasks require wakeup. It includes the +following types of switch: + +* ``switch_suspend``: + a task puts itself to sleep, this can happen only after explicitly setting + the task to ``sleepable``. After a task is suspended, it needs to be woken up + (``waking`` state) before being switched in again. + Setting the task's state to ``sleepable`` can be reverted before switching if it + is woken up or set to ``runnable``. +* ``switch_blocking``: + a special case of a ``switch_suspend`` where the task is waiting on a + sleeping RT lock (``PREEMPT_RT`` only), it is common to see wakeup and set + state events racing with each other and this leads the model to perceive this + type of switch when the task is not set to sleepable. This is a limitation of + the model in SMP system and workarounds may slow down the system. +* ``switch_preempt``: + a task switch as a result of kernel preemption (``schedule_entry_preempt`` in + the nrp model). +* ``switch_yield``: + a task explicitly calls the scheduler or is preempted while returning to + userspace. It can happen after a ``yield`` system call, from the idle task or + if the ``need_resched`` flag is set. By definition, a task cannot yield while + ``sleepable`` as that would be a suspension. A special case of a yield occurs + when a task in ``TASK_INTERRUPTIBLE`` calls the scheduler while a signal is + pending. The task doesn't go through the usual blocking/waking and is set + back to runnable, the resulting switch (if there) looks like a yield to the + ``signal_wakeup`` state and is followed by the signal delivery. From this + state, the monitor expects a signal even if it sees a wakeup event, although + not necessary, to rule out false negatives. + +This monitor doesn't include a running state, ``sleepable`` and ``runnable`` +are only referring to the task's desired state, which could be scheduled out +(e.g. due to preemption). However, it does include the event +``sched_switch_in`` to represent when a task is allowed to become running. This +can be triggered also by preemption, but cannot occur after the task got to +``sleeping`` before a ``wakeup`` occurs:: + + +--------------------------------------------------------------------------+ + | | + | | + | switch_suspend | | + | switch_blocking | | + v v | + +----------+ #==========================# set_state_runnable | + | | H H wakeup | + | | H H switch_in | + | | H H switch_yield | + | sleeping | H H switch_preempt | + | | H H signal_deliver | + | | switch_ H H ------+ | + | | _blocking H runnable H | | + | | <----------- H H <-----+ | + +----------+ H H | + | wakeup H H | + +---------------------> H H | + H H | + +---------> H H | + | #==========================# | + | | ^ | + | | | set_state_runnable | + | | | wakeup | + | set_state_sleepable | +------------------------+ + | v | | + | +--------------------------+ set_state_sleepable + | | | switch_in + | | | switch_preempt + signal_deliver | sleepable | signal_deliver + | | | ------+ + | | | | + | | | <-----+ + | +--------------------------+ + | | ^ + | switch_yield | set_state_sleepable + | v | + | +---------------+ | + +---------- | signal_wakeup | -+ + +---------------+ + ^ | switch_in + | | switch_preempt + | | switch_yield + +-----------+ wakeup + References ----------
kernel/trace/rv/Kconfig +2 -0
@@ -55,6 +55,8 @@ source "kernel/trace/rv/monitors/snroc/Kconfig" source "kernel/trace/rv/monitors/scpd/Kconfig" source "kernel/trace/rv/monitors/snep/Kconfig" source "kernel/trace/rv/monitors/sts/Kconfig" +source "kernel/trace/rv/monitors/nrp/Kconfig" +source "kernel/trace/rv/monitors/sssw/Kconfig" # Add new sched monitors here source "kernel/trace/rv/monitors/rtapp/Kconfig"
kernel/trace/rv/Makefile +2 -0
@@ -14,6 +14,8 @@ obj-$(CONFIG_RV_MON_RTAPP) += monitors/rtapp/rtapp.o obj-$(CONFIG_RV_MON_PAGEFAULT) += monitors/pagefault/pagefault.o obj-$(CONFIG_RV_MON_SLEEP) += monitors/sleep/sleep.o obj-$(CONFIG_RV_MON_STS) += monitors/sts/sts.o +obj-$(CONFIG_RV_MON_NRP) += monitors/nrp/nrp.o +obj-$(CONFIG_RV_MON_SSSW) += monitors/sssw/sssw.o # Add new monitors here obj-$(CONFIG_RV_REACTORS) += rv_reactors.o obj-$(CONFIG_RV_REACT_PRINTK) += reactor_printk.o
kernel/trace/rv/monitors/nrp/Kconfig new file +16 -0
@@ -0,0 +1,16 @@ +# SPDX-License-Identifier: GPL-2.0-only +# +config RV_MON_NRP + depends on RV + depends on RV_MON_SCHED + default y if !ARM64 + select DA_MON_EVENTS_ID + bool "nrp monitor" + help + Monitor to ensure preemption requires need resched. + This monitor is part of the sched monitors collection. + + This monitor is unstable on arm64, say N unless you are testing it. + + For further information, see: + Documentation/trace/rv/monitor_sched.rst
kernel/trace/rv/monitors/nrp/nrp.c new file +138 -0
138 lines not on this page · raw message
kernel/trace/rv/monitors/nrp/nrp.h new file +75 -0
75 lines not on this page · raw message
kernel/trace/rv/monitors/nrp/nrp_trace.h new file +15 -0
@@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +/* +* Snippet to be included in rv_trace.h +*/ + +#ifdef CONFIG_RV_MON_NRP +DEFINE_EVENT (event_da_monitor_id, event_nrp, + TP_PROTO(int id, char *state, char *event, char *next_state, bool final_state), +TP_ARGS (id, state, event, next_state, final_state)); + +DEFINE_EVENT (error_da_monitor_id, error_nrp, + TP_PROTO(int id, char *state, char *event), +TP_ARGS (id, state, event)); +#endif /* CONFIG_RV_MON_NRP */
kernel/trace/rv/monitors/sched/Kconfig +1 -0
@@ -2,6 +2,7 @@ # config RV_MON_SCHED depends on RV + depends on RV_PER_TASK_MONITORS >= 3 bool "sched monitor" help Collection of monitors to check the scheduler behaves according to specifications.
kernel/trace/rv/monitors/sssw/Kconfig new file +15 -0
@@ -0,0 +1,15 @@ +# SPDX-License-Identifier: GPL-2.0-only +# +config RV_MON_SSSW + depends on RV + depends on RV_MON_SCHED + default y + select DA_MON_EVENTS_ID + bool "sssw monitor" + help + Monitor to ensure sched_set_state to sleepable leads to sleeping and + sleeping tasks require wakeup. + This monitor is part of the sched monitors collection. + + For further information, see: + Documentation/trace/rv/monitor_sched.rst
kernel/trace/rv/monitors/sssw/sssw.c new file +116 -0
116 lines not on this page · raw message
kernel/trace/rv/monitors/sssw/sssw.h new file +105 -0
105 lines not on this page · raw message
kernel/trace/rv/monitors/sssw/sssw_trace.h new file +15 -0
@@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +/* +* Snippet to be included in rv_trace.h +*/ + +#ifdef CONFIG_RV_MON_SSSW +DEFINE_EVENT (event_da_monitor_id, event_sssw, + TP_PROTO(int id, char *state, char *event, char *next_state, bool final_state), +TP_ARGS (id, state, event, next_state, final_state)); + +DEFINE_EVENT (error_da_monitor_id, error_sssw, + TP_PROTO(int id, char *state, char *event), +TP_ARGS (id, state, event)); +#endif /* CONFIG_RV_MON_SSSW */
kernel/trace/rv/rv_trace.h +2 -0
@@ -123,6 +123,8 @@ DECLARE_EVENT_CLASS(error_da_monitor_id,#include <monitors/wwnr/wwnr_trace.h> #include <monitors/snroc/snroc_trace.h> +#include <monitors/nrp/nrp_trace.h> +#include <monitors/sssw/sssw_trace.h> // Add new monitors based on CONFIG_DA_MON_EVENTS_ID here #endif /* CONFIG_DA_MON_EVENTS_ID */
tools/verification/models/sched/nrp.dot new file +29 -0
@@ -0,0 +1,29 @@ +digraph state_automaton { + center = true; + size = "7,11"; + {node [shape = doublecircle] "any_thread_running"}; + {node [shape = circle] "any_thread_running"}; + {node [shape = circle] "nested_preempt"}; + {node [shape = plaintext, style=invis, label=""] "__init_preempt_irq"}; + {node [shape = circle] "preempt_irq"}; + {node [shape = circle] "rescheduling"}; + "__init_preempt_irq" -> "preempt_irq"; + "any_thread_running" [label = "any_thread_running", color = green3]; + "any_thread_running" -> "any_thread_running" [ label = "schedule_entry\nirq_entry" ]; + "any_thread_running" -> "rescheduling" [ label = "sched_need_resched" ]; + "nested_preempt" [label = "nested_preempt"]; + "nested_preempt" -> "any_thread_running" [ label = "schedule_entry_preempt\nschedule_entry" ]; + "nested_preempt" -> "nested_preempt" [ label = "irq_entry" ]; + "nested_preempt" -> "preempt_irq" [ label = "sched_need_resched" ]; + "preempt_irq" [label = "preempt_irq"]; + "preempt_irq" -> "nested_preempt" [ label = "schedule_entry_preempt\nschedule_entry" ]; + "preempt_irq" -> "preempt_irq" [ label = "irq_entry\nsched_need_resched" ]; + "rescheduling" [label = "rescheduling"]; + "rescheduling" -> "any_thread_running" [ label = "schedule_entry_preempt\nschedule_entry" ]; + "rescheduling" -> "preempt_irq" [ label = "irq_entry" ]; + "rescheduling" -> "rescheduling" [ label = "sched_need_resched" ]; + { rank = min ; + "__init_preempt_irq"; + "preempt_irq"; + } +}
tools/verification/models/sched/sssw.dot new file +30 -0
@@ -0,0 +1,30 @@ +digraph state_automaton { + center = true; + size = "7,11"; + {node [shape = plaintext, style=invis, label=""] "__init_runnable"}; + {node [shape = doublecircle] "runnable"}; + {node [shape = circle] "runnable"}; + {node [shape = circle] "signal_wakeup"}; + {node [shape = circle] "sleepable"}; + {node [shape = circle] "sleeping"}; + "__init_runnable" -> "runnable"; + "runnable" [label = "runnable", color = green3]; + "runnable" -> "runnable" [ label = "sched_set_state_runnable\nsched_wakeup\nsched_switch_in\nsched_switch_yield\nsched_switch_preempt\nsignal_deliver" ]; + "runnable" -> "sleepable" [ label = "sched_set_state_sleepable" ]; + "runnable" -> "sleeping" [ label = "sched_switch_blocking" ]; + "signal_wakeup" [label = "signal_wakeup"]; + "signal_wakeup" -> "runnable" [ label = "signal_deliver" ]; + "signal_wakeup" -> "signal_wakeup" [ label = "sched_switch_in\nsched_switch_preempt\nsched_switch_yield\nsched_wakeup" ]; + "signal_wakeup" -> "sleepable" [ label = "sched_set_state_sleepable" ]; + "sleepable" [label = "sleepable"]; + "sleepable" -> "runnable" [ label = "sched_set_state_runnable\nsched_wakeup" ]; + "sleepable" -> "signal_wakeup" [ label = "sched_switch_yield" ]; + "sleepable" -> "sleepable" [ label = "sched_set_state_sleepable\nsched_switch_in\nsched_switch_preempt\nsignal_deliver" ]; + "sleepable" -> "sleeping" [ label = "sched_switch_suspend\nsched_switch_blocking" ]; + "sleeping" [label = "sleeping"]; + "sleeping" -> "runnable" [ label = "sched_wakeup" ]; + { rank = min ; + "__init_runnable"; + "runnable"; + } +}
Signature · 1 line
2.50.1
Gabriele Monaco
maintainerPATCH v4 14/14 Add opid per-cpu monitor
In reply to Gabriele MonacoAdd a per-cpu monitor as part of the sched model:
* opid: operations with preemption and irq disabled Monitor to ensure wakeup and need_resched occur with irq and preemption disabled or in irq handlers.
--- Documentation/trace/rv/monitor_sched.rst | 55 +++++++ kernel/trace/rv/Kconfig | 1 + kernel/trace/rv/Makefile | 1 + kernel/trace/rv/monitors/opid/Kconfig | 19 +++ kernel/trace/rv/monitors/opid/opid.c | 169 +++++++++++++++++++++ kernel/trace/rv/monitors/opid/opid.h | 104 +++++++++++++ kernel/trace/rv/monitors/opid/opid_trace.h | 15 ++ kernel/trace/rv/rv_trace.h | 1 + tools/verification/models/sched/opid.dot | 35 +++++ 9 files changed, 400 insertions(+) create mode 100644 kernel/trace/rv/monitors/opid/Kconfig create mode 100644 kernel/trace/rv/monitors/opid/opid.c create mode 100644 kernel/trace/rv/monitors/opid/opid.h create mode 100644 kernel/trace/rv/monitors/opid/opid_trace.h create mode 100644 tools/verification/models/sched/opid.dot
Documentation/trace/rv/monitor_sched.rst +55 -0
@@ -341,6 +341,61 @@ can be triggered also by preemption, but cannot occur after the task got to | | switch_yield +-----------+ wakeup +Monitor opid +------------ + +The operations with preemption and irq disabled (opid) monitor ensures +operations like ``wakeup`` and ``need_resched`` occur with interrupts and +preemption disabled or during interrupt context, in such case preemption may +not be disabled explicitly. +``need_resched`` can be set by some RCU internals functions, in which case it +doesn't match a task wakeup and might occur with only interrupts disabled:: + + | sched_need_resched + | sched_waking + | irq_entry + | +--------------------+ + v v | + +------------------------------------------------------+ + +----------- | disabled | <+ + | +------------------------------------------------------+ | + | | ^ | + | | preempt_disable sched_need_resched | + | preempt_enable | +--------------------+ | + | v | v | | + | +------------------------------------------------------+ | + | | irq_disabled | | + | +------------------------------------------------------+ | + | | | ^ | + | irq_entry irq_entry | | | + | sched_need_resched v | irq_disable | + | sched_waking +--------------+ | | | + | +----- | | irq_enable | | + | | | in_irq | | | | + | +----> | | | | | + | +--------------+ | | irq_disable + | | | | | + | irq_enable | irq_enable | | | + | v v | | + | #======================================================# | + | H enabled H | + | #======================================================# | + | | ^ ^ preempt_enable | | + | preempt_disable preempt_enable +--------------------+ | + | v | | + | +------------------+ | | + +----------> | preempt_disabled | -+ | + +------------------+ | + | | + +-------------------------------------------------------+ + +This monitor is designed to work on ``PREEMPT_RT`` kernels, the special case of +events occurring in interrupt context is a shortcut to identify valid scenarios +where the preemption tracepoints might not be visible, during interrupts +preemption is always disabled. On non- ``PREEMPT_RT`` kernels, the interrupts +might invoke a softirq to set ``need_resched`` and wake up a task. This is +another special case that is currently not supported by the monitor. + References ----------
kernel/trace/rv/Kconfig +1 -0
@@ -57,6 +57,7 @@ source "kernel/trace/rv/monitors/snep/Kconfig" source "kernel/trace/rv/monitors/sts/Kconfig" source "kernel/trace/rv/monitors/nrp/Kconfig" source "kernel/trace/rv/monitors/sssw/Kconfig" +source "kernel/trace/rv/monitors/opid/Kconfig" # Add new sched monitors here source "kernel/trace/rv/monitors/rtapp/Kconfig"
kernel/trace/rv/Makefile +1 -0
@@ -16,6 +16,7 @@ obj-$(CONFIG_RV_MON_SLEEP) += monitors/sleep/sleep.o obj-$(CONFIG_RV_MON_STS) += monitors/sts/sts.o obj-$(CONFIG_RV_MON_NRP) += monitors/nrp/nrp.o obj-$(CONFIG_RV_MON_SSSW) += monitors/sssw/sssw.o +obj-$(CONFIG_RV_MON_OPID) += monitors/opid/opid.o # Add new monitors here obj-$(CONFIG_RV_REACTORS) += rv_reactors.o obj-$(CONFIG_RV_REACT_PRINTK) += reactor_printk.o
kernel/trace/rv/monitors/opid/Kconfig new file +19 -0
@@ -0,0 +1,19 @@ +# SPDX-License-Identifier: GPL-2.0-only +# +config RV_MON_OPID + depends on RV + depends on TRACE_IRQFLAGS + depends on TRACE_PREEMPT_TOGGLE + depends on RV_MON_SCHED + default y if PREEMPT_RT + select DA_MON_EVENTS_IMPLICIT + bool "opid monitor" + help + Monitor to ensure operations like wakeup and need resched occur with + interrupts and preemption disabled or during IRQs, where preemption + may not be disabled explicitly. + + This monitor is unstable on !PREEMPT_RT, say N unless you are testing it. + + For further information, see: + Documentation/trace/rv/monitor_sched.rst
kernel/trace/rv/monitors/opid/opid.c new file +169 -0
169 lines not on this page · raw message
kernel/trace/rv/monitors/opid/opid.h new file +104 -0
104 lines not on this page · raw message
kernel/trace/rv/monitors/opid/opid_trace.h new file +15 -0
@@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +/* +* Snippet to be included in rv_trace.h +*/ + +#ifdef CONFIG_RV_MON_OPID +DEFINE_EVENT (event_da_monitor, event_opid, + TP_PROTO(char *state, char *event, char *next_state, bool final_state), +TP_ARGS (state, event, next_state, final_state)); + +DEFINE_EVENT (error_da_monitor, error_opid, +TP_PROTO (char *state, char *event), +TP_ARGS (state, event)); +#endif /* CONFIG_RV_MON_OPID */
kernel/trace/rv/rv_trace.h +1 -0
@@ -62,6 +62,7 @@ DECLARE_EVENT_CLASS(error_da_monitor,#include <monitors/scpd/scpd_trace.h> #include <monitors/snep/snep_trace.h> #include <monitors/sts/sts_trace.h> +#include <monitors/opid/opid_trace.h> // Add new monitors based on CONFIG_DA_MON_EVENTS_IMPLICIT here #endif /* CONFIG_DA_MON_EVENTS_IMPLICIT */
tools/verification/models/sched/opid.dot new file +35 -0
@@ -0,0 +1,35 @@ +digraph state_automaton { + center = true; + size = "7,11"; + {node [shape = plaintext, style=invis, label=""] "__init_disabled"}; + {node [shape = circle] "disabled"}; + {node [shape = doublecircle] "enabled"}; + {node [shape = circle] "enabled"}; + {node [shape = circle] "in_irq"}; + {node [shape = circle] "irq_disabled"}; + {node [shape = circle] "preempt_disabled"}; + "__init_disabled" -> "disabled"; + "disabled" [label = "disabled"]; + "disabled" -> "disabled" [ label = "sched_need_resched\nsched_waking\nirq_entry" ]; + "disabled" -> "irq_disabled" [ label = "preempt_enable" ]; + "disabled" -> "preempt_disabled" [ label = "irq_enable" ]; + "enabled" [label = "enabled", color = green3]; + "enabled" -> "enabled" [ label = "preempt_enable" ]; + "enabled" -> "irq_disabled" [ label = "irq_disable" ]; + "enabled" -> "preempt_disabled" [ label = "preempt_disable" ]; + "in_irq" [label = "in_irq"]; + "in_irq" -> "enabled" [ label = "irq_enable" ]; + "in_irq" -> "in_irq" [ label = "sched_need_resched\nsched_waking\nirq_entry" ]; + "irq_disabled" [label = "irq_disabled"]; + "irq_disabled" -> "disabled" [ label = "preempt_disable" ]; + "irq_disabled" -> "enabled" [ label = "irq_enable" ]; + "irq_disabled" -> "in_irq" [ label = "irq_entry" ]; + "irq_disabled" -> "irq_disabled" [ label = "sched_need_resched" ]; + "preempt_disabled" [label = "preempt_disabled"]; + "preempt_disabled" -> "disabled" [ label = "irq_disable" ]; + "preempt_disabled" -> "enabled" [ label = "preempt_enable" ]; + { rank = min ; + "__init_disabled"; + "disabled"; + } +}
Signature · 1 line
2.50.1
Steven Rostedt
maintainerQuoted 33 lines · On Mon, 21 Jul 2025 10:23:10 +0200 Gabriele Monaco <[email protected]> wrote:
Documentation/trace/rv/monitor_sched.rst | 307 +++++++++++++++---
include/linux/rv.h | 3 +-
include/linux/sched.h | 7 +-
include/rv/da_monitor.h | 129 +++++---
include/trace/events/sched.h | 12 +-
kernel/sched/core.c | 13 +-
kernel/trace/rv/Kconfig | 16 +-
kernel/trace/rv/Makefile | 6 +-
kernel/trace/rv/monitors/{tss => nrp}/Kconfig | 12 +-
kernel/trace/rv/monitors/nrp/nrp.c | 138 ++++++++
kernel/trace/rv/monitors/nrp/nrp.h | 75 +++++
kernel/trace/rv/monitors/nrp/nrp_trace.h | 15 +
kernel/trace/rv/monitors/opid/Kconfig | 19 ++
kernel/trace/rv/monitors/opid/opid.c | 169 ++++++++++
kernel/trace/rv/monitors/opid/opid.h | 104 ++++++
.../sncid_trace.h => opid/opid_trace.h} | 8 +-
kernel/trace/rv/monitors/sched/Kconfig | 1 +
kernel/trace/rv/monitors/sched/sched.c | 3 +-
kernel/trace/rv/monitors/sco/sco.c | 7 +-
kernel/trace/rv/monitors/scpd/Kconfig | 2 +-
kernel/trace/rv/monitors/scpd/scpd.c | 7 +-
kernel/trace/rv/monitors/sncid/sncid.c | 96 ------
kernel/trace/rv/monitors/sncid/sncid.h | 49 ---
kernel/trace/rv/monitors/snep/Kconfig | 2 +-
kernel/trace/rv/monitors/snep/snep.c | 7 +-
kernel/trace/rv/monitors/snep/snep.h | 14 +-
kernel/trace/rv/monitors/snroc/snroc.c | 3 +-
.../trace/rv/monitors/{sncid => sssw}/Kconfig | 10 +-
kernel/trace/rv/monitors/sssw/sssw.c | 116 +++++++
kernel/trace/rv/monitors/sssw/sssw.h | 105 ++++++
kernel/trace/rv/monitors/sssw/sssw_trace.h | 15 +
kernel/trace/rv/monitors/sts/Kconfig | 19 ++
kernel/trace/rv/monitors/sts/sts.c | 156 +++++++++Quoted 8 lines
kernel/trace/rv/monitors/sts/sts.h | 117 +++++++
.../{tss/tss_trace.h => sts/sts_trace.h} | 8 +-
kernel/trace/rv/monitors/tss/tss.c | 91 ------
kernel/trace/rv/monitors/tss/tss.h | 47 ---
kernel/trace/rv/monitors/wip/Kconfig | 2 +-
kernel/trace/rv/monitors/wip/wip.c | 3 +-
kernel/trace/rv/monitors/wwnr/wwnr.c | 3 +-
kernel/trace/rv/rv_trace.h | 114 ++++---Can you break this up into two patch series? One that modifies the kernel and one that modifies the tools directory. Linus prefers changes to tools come in separately to changes in the kernel. So do I as I test them differently.
Thanks,
-- Steve
Quoted 11 lines
tools/verification/models/sched/nrp.dot | 29 ++ tools/verification/models/sched/opid.dot | 35 ++ tools/verification/models/sched/sncid.dot | 18 - tools/verification/models/sched/sssw.dot | 30 ++ tools/verification/models/sched/sts.dot | 38 +++ tools/verification/models/sched/tss.dot | 18 - tools/verification/rv/src/in_kernel.c | 4 +- tools/verification/rv/src/rv.c | 1 + tools/verification/rvgen/rvgen/container.py | 10 + tools/verification/rvgen/rvgen/dot2c.py | 20 +- tools/verification/rvgen/rvgen/generator.py | 16 +-
Gabriele Monaco
maintainerQuoted 5 lines · Steven Rostedt
Can you break this up into two patch series? One that modifies the kernel and one that modifies the tools directory. Linus prefers changes to tools come in separately to changes in the kernel. So do I as I test them differently.
Mmh, I see. The problem with splitting those patches that strictly is that patches changing the generating tools also include the adaptation of kernel files, I could create something like:
verification/rvgen: Organise Kconfig entries for nested monitors
Do the tools/ stuff...
The kernel changes are required to test this!
rv: Organise Kconfig entries for nested monitors
As introduced in commit XYZ, adapt the Kconfig...
And send them in separate series, but it doesn't look too clean to me as the tool change requires the kernel change or, in general (see the other patch about line length), the two things belong with each other.
Likewise, patches about monitors touch the dot models in tools/ but those definitely belong in the same patch, otherwise we lose context.
What about keeping the patches as they are right now and send them separately like this:
kernel series:
rv: Add opid per-cpu monitor
tools/verification/models/sched/opid.dot | 35 ++++++
rv: Add nrp and sssw per-task monitors
tools/verification/models/sched/nrp.dot | 29 +++++
tools/verification/models/sched/sssw.dot | 30 ++++++
rv: Replace tss and sncid monitors with more complete sts
tools/verification/models/sched/sncid.dot | 18 ---
tools/verification/models/sched/sts.dot | 38 +++++
tools/verification/models/sched/tss.dot | 18 ---
sched: Adapt sched tracepoints for RV task model
rv: Retry when da monitor detects race conditions
rv: Adjust monitor dependencies
rv: Use strings in da monitors tracepoints
rv: Remove trailing whitespace from tracepoint string
rv: Add da_handle_start_run_event_ to per-task monitors
tools series:
tools/dot2c: Fix generated files going over 100 column limit
kernel/trace/rv/monitors/snep/snep.h | 14 ++++++++++++--
verification/rvgen: Organise Kconfig entries for nested monitors
kernel/trace/rv/Kconfig | 5 +++++
rv: Return init error when registering monitors
tools/verification/rvgen/rvgen/templates/container/main.c | 3 +--
tools/verification/rvgen/rvgen/templates/dot2k/main.c | 3 +--
kernel/trace/rv/monitors/sched/sched.c | 3 +--
kernel/trace/rv/monitors/sco/sco.c | 3 +--
...
kernel/trace/rv/monitors/wwnr/wwnr.c | 3 +--
tools/rv: Stop gracefully also on SIGTERM
tools/rv: Do not skip idle in trace
The rationale is that tools files changed in the kernel patches are not really tool stuff (dot models). And kernel stuff changed in the tools are something that the tools generate, and to test them a build should suffice (kernel robot would do that). Having them together eases testing the tool, I believe.
Note: I missed the tools templates from "rv: Return init error when registering monitors" (now in the tools series with added files), I believe that belongs more to tools but I could also move it or split them in two if you prefer.
Does it make sense to you?
Thanks,
Gabriele
Steven Rostedt
maintainerQuoted 5 lines · On Wed, 23 Jul 2025 11:55:50 +0200 Gabriele Monaco <[email protected]> wrote:
The rationale is that tools files changed in the kernel patches are not really tool stuff (dot models). And kernel stuff changed in the tools are something that the tools generate, and to test them a build should suffice (kernel robot would do that). Having them together eases testing the tool, I believe.
Yes, I agree with the above.
If you make kernel changes that a new dot file is going to use, then sure, keep that in the kernel side.
I'm basically saying that any tools enhancements beyond adding new models and such should be in their own series.
Thanks,
-- Steve