Fixes on Deterministic and Hybrid Automata
Subsystems (2)
Gabriele Monaco
maintainerv2 cover Fixes on Deterministic and Hybrid Automata
Fix issues that were reported by bots or visible only after integration:
* Make sure timers are always terminated and waited for when disabling the monitor or when the target terminates
* Run per-cpu monitors with migration disabled since preemption is now enabled from tracepoints
* Fix a wrong __user specifier in a helper function
* Other cleanup and concurrency issues
Differences since V1 [1]:
* Fix memory consistency with timer callbacks racing with resets
* Add per-obj deallocation hook in rvgen generated code
* Do not rely on clean monitor when initialising HA
* Add tracepoint synchronisation before returning per-task slots
* Fix suffix strip in dot2k
* Generate stub deallocation hooks instead of failing build when per-obj miss those
[1] - https://lore.kernel.org/lkml/[email protected]
Gabriele Monaco (11):
rv: Fix __user specifier usage in extract_params()
rv: Fix read_lock scope in per-task DA cleanup
rv: Reset per-task DA monitors before releasing the slot
rv: Prevent task migration while handling per-CPU events
rv: Prevent in-flight per-task handlers from using invalid slots
rv: Ensure all pending probes terminate on per-obj monitor destroy
rv: Ensure synchronous cleanup for HA monitors
rv: Do not rely on clean monitor when initialising HA
rv: Add automatic cleanup handlers for per-task HA monitors verification/rvgen: Generate cleanup hook for per-obj monitor verification/rvgen: Fix suffix strip in dot2k
Wen Yang (1):
rv: Fix monitor start ordering and memory ordering for monitoring flag
include/rv/da_monitor.h | 67 +++++++++---- include/rv/ha_monitor.h | 95 ++++++++++++++++++- include/rv/ltl_monitor.h | 1 + kernel/trace/rv/monitors/deadline/deadline.h | 3 +- kernel/trace/rv/monitors/nomiss/nomiss.c | 4 +- kernel/trace/rv/monitors/opid/opid.c | 4 +- kernel/trace/rv/monitors/stall/stall.c | 4 +- tools/verification/rvgen/rvgen/dot2k.py | 19 +++- .../rvgen/rvgen/templates/dot2k/main.c | 4 +- 9 files changed, 171 insertions(+), 30 deletions(-)
base-commit: 8bc67e4db64aa72732c474b44ea8622062c903f0
Signature · 1 line
2.54.0
Gabriele Monaco
maintainerPATCH v2 1/12 Fix __user specifier usage in extract_params()
The attributes variables extracted from syscalls in the helper are both defined with the __user specifier although only the actual pointer to user data should be marked.
Remove the __user specifier from attr.
--- kernel/trace/rv/monitors/deadline/deadline.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
kernel/trace/rv/monitors/deadline/deadline.h +2 -1
@@ -95,7 +95,8 @@ static inline u8 get_server_type(struct task_struct *tsk)static inline int extract_params (struct pt_regs *regs,long id,pid_t *pid_out) { size_t size = offsetofend(struct sched_attr, sched_flags); -struct sched_attr __user *uattr, attr; +struct sched_attr __user *uattr; +struct sched_attr attr;int new_policy =-1 , ret;unsigned long args[6 ];
Signature · 1 line
2.54.0
Re: [PATCH v2 01/12] rv: Fix __user specifier usage in extract_params()
Quoted 11 lines · Gabriele Monaco
The attributes variables extracted from syscalls in the helper are both defined with the
__userspecifier although only the actual pointer to user data should be marked.Remove the
__userspecifier from attr.Reported-by kernel test robot <[email protected]>Fixes b133207deb72 ("rv: Add nomiss deadline monitor")Reviewed-by Wen Yang <[email protected]>Signed-off-by Gabriele Monaco <[email protected]>
Gabriele Monaco
maintainerPATCH v2 2/12 Fix read_lock scope in per-task DA cleanup
In reply to Gabriele MonacoThe da_monitor_reset_all() function for per-task monitors takes tasklist_lock while iterating over tasks, then keeps it also while iterating over idle tasks (one per CPU). The latter is not necessary since the lock needs to guard only for_each_process_thread().
Use a scoped_guard for more compact syntax and adjust the scope only where the lock is necessary.
--- include/rv/da_monitor.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
include/rv/da_monitor.h +4 -4
@@ -272,12 +272,12 @@ static void da_monitor_reset_all(void)struct task_struct *g, *p;int cpu; -read_lock (&tasklist_lock); -for_each_process_thread (g, p) -da_monitor_reset (da_get_monitor (p)); + scoped_guard(read_lock, &tasklist_lock) { +for_each_process_thread (g, p) +da_monitor_reset (da_get_monitor (p)); + }for_each_present_cpu (cpu)da_monitor_reset (da_get_monitor (idle_task (cpu))); -read_unlock (&tasklist_lock); }/*
Signature · 1 line
2.54.0
Re: [PATCH v2 02/12] rv: Fix read_lock scope in per-task DA cleanup
Quoted 10 lines · Gabriele Monaco
The
da_monitor_reset_all()function for per-task monitors takestasklist_lockwhile iterating over tasks, then keeps it also while iterating over idle tasks (one per CPU). The latter is not necessary since the lock needs to guard onlyfor_each_process_thread().Use a
scoped_guardfor more compact syntax and adjust the scope only where the lock is necessary.Fixes 30984ccf31b7f ("rv: Refactor da_monitor to minimise macros")Fixes 8259cb14a7068 ("rv: Reset per-task monitors also for idle tasks")
Fixes: tag "indicates that the patch fixes a bug in a previous commit". There is no bug here, so I don't think Fixes tags are applicable.
Quoted 2 lines
Reviewed-by Wen Yang <[email protected]>Signed-off-by Gabriele Monaco <[email protected]>
Gabriele Monaco
maintainerRe: [PATCH v2 02/12] rv: Fix read_lock scope in per-task DA cleanup
Quoted 11 lines · Nam Cao
Quoted 10 lines · Gabriele Monaco
The
da_monitor_reset_all()function for per-task monitors takestasklist_lockwhile iterating over tasks, then keeps it also while iterating over idle tasks (one per CPU). The latter is not necessary
since the lock needs to guard onlyfor_each_process_thread().Use a
scoped_guardfor more compact syntax and adjust the scope only
where the lock is necessary.
Quoted 7 lines
Quoted 3 lines
Fixes 30984ccf31b7f ("rv: Refactor da_monitor to minimise macros")Fixes 8259cb14a7068 ("rv: Reset per-task monitors also for idletasks")
Fixes: tag "indicates that the patch fixes a bug in a previous commit". There is no bug here, so I don't think Fixes tags are applicable.
Yeah good point, that isn't a real bug.. We're just holding a lock for a bit too long but there's no harm in that. Will remove the tags.
Thanks,
Gabriele
Quoted 5 lines
Quoted 3 lines
Reviewed-by Wen Yang <[email protected]>Signed-off-by Gabriele Monaco <[email protected]>Reviewed-by Nam Cao <[email protected]>
Gabriele Monaco
maintainerPATCH v2 3/12 Reset per-task DA monitors before releasing the slot
In reply to Gabriele MonacoPer-task monitors use task_mon_slot to determine which slot in the array to use for the monitor. During destruction, this slot is returned but this is done before resetting the monitor. As a result, the monitor's reset is in fact resetting a slot that is outside of the array (RV_PER_TASK_MONITOR_INIT).
Release the slot only after the reset to avoid out-of-bound memory access.
--- include/rv/da_monitor.h | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
include/rv/da_monitor.h +3 -2
@@ -309,10 +309,11 @@ static inline void da_monitor_destroy(void)WARN_ONCE (1 ,"Disabling a disabled monitor: " __stringify(MONITOR_NAME )); return; } -rv_put_task_monitor_slot (task_mon_slot); - task_mon_slot =RV_PER_TASK_MONITOR_INIT ;da_monitor_reset_all (); + +rv_put_task_monitor_slot (task_mon_slot); + task_mon_slot =RV_PER_TASK_MONITOR_INIT ; }#elif RV_MON_TYPE == RV_MON_PER_OBJ
Signature · 1 line
2.54.0
Re: [PATCH v2 03/12] rv: Reset per-task DA monitors before releasing the slot
Quoted 5 lines · Gabriele Monaco
Per-task monitors use
task_mon_slotto determine which slot in the array to use for the monitor. During destruction, this slot is returned but this is done before resetting the monitor. As a result, the monitor's reset is in fact resetting a slot that is outside of the array (RV_PER_TASK_MONITOR_INIT).
Oh crap.
Quoted 2 lines
Release the slot only after the reset to avoid out-of-bound memory access.
I think KASAN can catch this type of issue.
Quoted 4 lines
Fixes f5587d1b6ec93 ("rv: Add Hybrid Automata monitor type")Suggested-by Wen Yang <[email protected]>Reviewed-by Wen Yang <[email protected]>Signed-off-by Gabriele Monaco <[email protected]>
Should we have
Cc: [email protected]
?
Gabriele Monaco
maintainerPATCH v2 4/12 Prevent task migration while handling per-CPU events
In reply to Gabriele MonacoTracepoint handlers are now fully preemptible. When a per-CPU monitor handles an event, it retrieves the monitor state using a per-CPU pointer. If the event itself doesn't disable preemption, the task can migrate to a different CPU and we risk updating the wrong monitor.
Mitigate this by explicitly disabling task migration before acquiring the monitor pointer. This cannot guarantee the monitor runs on the correct CPU but reduces the race condition window and prevents warnings.
--- include/rv/da_monitor.h | 11 +++++++++++ 1 file changed, 11 insertions(+)
include/rv/da_monitor.h +11 -0
@@ -181,6 +181,10 @@ static inline void da_monitor_destroy(void) da_monitor_reset_all(); } +#ifndef da_implicit_guard +#define da_implicit_guard () +#endif +#elif RV_MON_TYPE == RV_MON_PER_CPU/* * Functions to define, init and get a per-cpu monitor.
@@ -230,6 +234,10 @@ static inline void da_monitor_destroy(void) da_monitor_reset_all(); } +#ifndef da_implicit_guard +#define da_implicit_guard () guard(migrate)() +#endif +#elif RV_MON_TYPE == RV_MON_PER_TASK/* * Functions to define, init and get a per-task monitor.
@@ -677,6 +685,7 @@ static inline bool __da_handle_start_run_event(struct da_monitor *da_mon,*/ static inline void da_handle_event (enum events event) { +da_implicit_guard ();__da_handle_event (da_get_monitor (), event,0 ); }
@@ -692,6 +701,7 @@ static inline void da_handle_event(enum events event)*/ static inline bool da_handle_start_event (enum events event) { +da_implicit_guard ();return __da_handle_start_event (da_get_monitor (), event,0 ); }
@@ -703,6 +713,7 @@ static inline bool da_handle_start_event(enum events event)*/ static inline bool da_handle_start_run_event (enum events event) { +da_implicit_guard ();return __da_handle_start_run_event (da_get_monitor (), event,0 ); }
Signature · 1 line
2.54.0
Re: [PATCH v2 04/12] rv: Prevent task migration while handling per-CPU events
Quoted 1 line · Gabriele Monaco
Tracepoint handlers are now fully preemptible.
Would be nice to mention the commit that did this:
a46023d5616e ("tracing: Guard __DECLARE_TRACE() use of __DO_TRACE_CALL() with SRCU-fast")
Quoted 11 lines
When a per-CPU monitor
handles an event, it retrieves the monitor state using a per-CPU pointer. If the event itself doesn't disable preemption, the task can migrate to a different CPU and we risk updating the wrong monitor.Mitigate this by explicitly disabling task migration before acquiring the monitor pointer. This cannot guarantee the monitor runs on the correct CPU but reduces the race condition window and prevents warnings.
Reviewed-by Wen Yang <[email protected]>Signed-off-by Gabriele Monaco <[email protected]>
Gabriele Monaco
maintainerPATCH v2 5/12 Prevent in-flight per-task handlers from using invalid slots
In reply to Gabriele MonacoPer-task monitors use a slot in the task_struct->rv[] array and store that locally (e.g. task_mon_slot), this slot is returned during the destruction process but currently hanlers can be running while that slot is returning and this race may lead to accessing an invalid slot.
Synchronise with all in-flight tracepoint handlers using tracepoint_synchronize_unregister() before returning the slot.
--- include/rv/da_monitor.h | 4 ++++ include/rv/ltl_monitor.h | 1 + 2 files changed, 5 insertions(+)
include/rv/da_monitor.h +4 -0
@@ -310,6 +310,9 @@ static int da_monitor_init(void)/* * da_monitor_destroy - return the allocated slot +* +* Wait for all in-flight handlers before returning the slot to avoid +* out-of-bound accesses. */ static inline void da_monitor_destroy (void ) {
@@ -320,6 +323,7 @@ static inline void da_monitor_destroy(void)da_monitor_reset_all (); +tracepoint_synchronize_unregister ();rv_put_task_monitor_slot (task_mon_slot); task_mon_slot =RV_PER_TASK_MONITOR_INIT ; }
include/rv/ltl_monitor.h +1 -0
@@ -77,6 +77,7 @@ static void ltl_monitor_destroy(void) {rv_detach_trace_probe (name, task_newtask, handle_task_newtask); +tracepoint_synchronize_unregister ();rv_put_task_monitor_slot (ltl_monitor_slot); ltl_monitor_slot =RV_PER_TASK_MONITOR_INIT ; }
Signature · 1 line
2.54.0
Re: [PATCH v2 05/12] rv: Prevent in-flight per-task handlers from using invalid slots
Quoted 5 lines · Gabriele Monaco
Per-task monitors use a slot in the task_struct->rv[] array and store that locally (e.g.
task_mon_slot), this slot is returned during the destruction process but currently hanlers can be running while that slot is returning and this race may lead to accessing an invalid slot.
Quoted 7 lines
Synchronise with all in-flight tracepoint handlers using
tracepoint_synchronize_unregister()before returning the slot.Fixes f5587d1b6ec9 ("rv: Add Hybrid Automata monitor type")Fixes a9769a5b9878 ("rv: Add support for LTL monitors")Suggested-by Wen Yang <[email protected]>Signed-off-by Gabriele Monaco <[email protected]>
Gabriele Monaco
maintainerPATCH v2 6/12 Ensure all pending probes terminate on per-obj monitor destroy
In reply to Gabriele MonacoThe monitor disable/destroy sequence detaches all probes and resets the monitor's data, however it doesn't wait for pending probes. This is an issue with per-object monitors, which free the monitor storage.
Call tracepoint_synchronize_unregister() to make sure to wait for all pending probes before destroying the monitor storage.
--- include/rv/da_monitor.h | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
include/rv/da_monitor.h +3 -2
@@ -519,9 +519,10 @@ static inline void da_monitor_destroy(void)struct hlist_node *tmp;int bkt; +tracepoint_synchronize_unregister ();/* -* This function is called after all probes are disabled, we need only -* worry about concurrency against old events. +* This function is called after all probes are disabled and no longer +* pending, we can safely assume no concurrent user. */ synchronize_rcu ();hash_for_each_safe (da_monitor_ht, bkt, tmp, mon_storage, node) {
Signature · 1 line
2.54.0
Re: [PATCH v2 06/12] rv: Ensure all pending probes terminate on per-obj monitor destroy
Quoted 10 lines · Gabriele Monaco
The monitor disable/destroy sequence detaches all probes and resets the monitor's data, however it doesn't wait for pending probes. This is an issue with per-object monitors, which free the monitor storage.
Call
tracepoint_synchronize_unregister()to make sure to wait for all pending probes before destroying the monitor storage.Fixes 4a24127bd6cb ("rv: Add support for per-object monitors in DA/HA")Reviewed-by Wen Yang <[email protected]>Signed-off-by Gabriele Monaco <[email protected]>
Gabriele Monaco
maintainerPATCH v2 7/12 Fix monitor start ordering and memory ordering for monitoring flag
In reply to Gabriele MonacoFrom: Wen Yang <[email protected]>
da_monitor_start() set monitoring=1 before calling da_monitor_init_hook(), may racing with the sched_switch handler:
da_monitor_start() sched_switch handler
------------------------- ---------------------------------
da_mon->monitoring = 1;
if (da_monitoring(da_mon)) /* true */
ha_start_timer_ns(...);
/* hrtimer->base == NULL, crash */
da_monitor_init_hook(da_mon);
/* hrtimer_setup() sets base */Fix the ordering and pair with release/acquire semantics:
da_monitor_init_hook(da_mon); smp_store_release(&da_mon->monitoring, 1); /* da_monitor_start() */ return smp_load_acquire(&da_mon->monitoring); /* da_monitoring() */
On ARM64 a plain STR + LDR does not form a release-acquire pair, so the load can observe monitoring=1 while hrtimer->base is still NULL. The plain accesses are also data races under KCSAN.
Use WRITE_ONCE for the monitoring=0 store in da_monitor_reset() to cover the reset path.
--- include/rv/da_monitor.h | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
include/rv/da_monitor.h +5 -3
@@ -82,7 +82,7 @@ static void react(enum states curr_state, enum events event)static inline void da_monitor_reset (struct da_monitor *da_mon) {da_monitor_reset_hook (da_mon); - da_mon->monitoring =0 ; +WRITE_ONCE (da_mon->monitoring,0 ); da_mon->curr_state =model_get_initial_state (); }
@@ -95,8 +95,9 @@ static inline void da_monitor_reset(struct da_monitor *da_mon)static inline void da_monitor_start (struct da_monitor *da_mon) { da_mon->curr_state =model_get_initial_state (); - da_mon->monitoring =1 ;da_monitor_init_hook (da_mon); +/* Pairs with smp_load_acquire in da_monitoring(). */ +smp_store_release (&da_mon->monitoring,1 ); }/*
@@ -104,7 +105,8 @@ static inline void da_monitor_start(struct da_monitor *da_mon)*/ static inline bool da_monitoring (struct da_monitor *da_mon) { -return da_mon->monitoring; +/* Pairs with smp_store_release in da_monitor_start(). */ +return smp_load_acquire (&da_mon->monitoring); }/*
Signature · 1 line
2.54.0
Re: [PATCH v2 07/12] rv: Fix monitor start ordering and memory ordering for monitoring flag
Quoted 24 lines · Gabriele Monaco
From: Wen Yang <[email protected]>
da_monitor_start()set monitoring=1 before callingda_monitor_init_hook(), may racing with thesched_switchhandler:da_monitor_start() sched_switch handler ------------------------- --------------------------------- da_mon->monitoring = 1; if (da_monitoring(da_mon)) /* true */ ha_start_timer_ns(...); /* hrtimer->base == NULL, crash */ da_monitor_init_hook(da_mon); /* hrtimer_setup() sets base */Fix the ordering and pair with release/acquire semantics:
da_monitor_init_hook(da_mon); smp_store_release(&da_mon->monitoring, 1); /* da_monitor_start() */ return smp_load_acquire(&da_mon->monitoring); /* da_monitoring() */On ARM64 a plain STR + LDR does not form a release-acquire pair, so the load can observe monitoring=1 while
hrtimer->baseis still NULL. The plain accesses are also data races under KCSAN.
Quoted 7 lines
Use
WRITE_ONCEfor the monitoring=0 store inda_monitor_reset()to cover the reset path.Fixes 792575348ff7 ("rv/include: Add deterministic automata monitor definition via C macros")Signed-off-by Wen Yang <[email protected]>Reviewed-by Gabriele Monaco <[email protected]>Signed-off-by Gabriele Monaco <[email protected]>
Looks correct to me.
Wen, I am curious, how did you find this issue?
Nam
Re: [PATCH v2 07/12] rv: Fix monitor start ordering and memory ordering for monitoring flag
Quoted 33 lines · Nam Cao
Quoted 24 lines · Gabriele Monaco
From: Wen Yang <[email protected]>
da_monitor_start()set monitoring=1 before callingda_monitor_init_hook(), may racing with thesched_switchhandler:da_monitor_start() sched_switch handler ------------------------- --------------------------------- da_mon->monitoring = 1; if (da_monitoring(da_mon)) /* true */ ha_start_timer_ns(...); /* hrtimer->base == NULL, crash */ da_monitor_init_hook(da_mon); /* hrtimer_setup() sets base */Fix the ordering and pair with release/acquire semantics:
da_monitor_init_hook(da_mon); smp_store_release(&da_mon->monitoring, 1); /* da_monitor_start() */ return smp_load_acquire(&da_mon->monitoring); /* da_monitoring() */On ARM64 a plain STR + LDR does not form a release-acquire pair, so the load can observe monitoring=1 while
hrtimer->baseis still NULL. The plain accesses are also data races under KCSAN.Quoted 7 lines
Use
WRITE_ONCEfor the monitoring=0 store inda_monitor_reset()to cover the reset path.Fixes 792575348ff7 ("rv/include: Add deterministic automata monitor definition via C macros")Signed-off-by Wen Yang <[email protected]>Reviewed-by Gabriele Monaco <[email protected]>Signed-off-by Gabriele Monaco <[email protected]>
Quoted 5 lines
Looks correct to me.
Reviewed-by Nam Cao <[email protected]>Wen, I am curious, how did you find this issue?
It was caught during development of the tlob monitor by a KUnit test
(since moved to selftests) that monitored a kthread running concurrently
on another CPU. On a multi-core box the window between setting
monitoring=1 and da_monitor_init_hook() was wide enough to trigger the
crash, and KCSAN flagged the plain monitoring accesses as data
races as well.
--
Best wishes,
Wen
Gabriele Monaco
maintainerPATCH v2 8/12 Ensure synchronous cleanup for HA monitors
In reply to Gabriele MonacoHA monitors may start timers, all cleanup functions currently stop the timers asynchronously to avoid sleeping in the wrong context. Nothing makes sure running callbacks terminate on cleanup.
Run the entire HA timer callback in an RCU read-side critical section, this way we can simply synchronize_rcu() with any pending timer and are sure any cleanup using kfree_rcu() runs after callbacks terminated. Additionally make sure any unlikely callback running late won't run any code if the monitor is marked as disabled.
Use memory barriers to serialise with racing resets.
--- include/rv/da_monitor.h | 26 +++++++++++++++++++++----- include/rv/ha_monitor.h | 19 +++++++++++++++++-- 2 files changed, 38 insertions(+), 7 deletions(-)
include/rv/da_monitor.h +21 -5
@@ -57,6 +57,15 @@ static struct rv_monitor rv_this;#define da_monitor_reset_hook (da_mon)#endif +/* +* Hook to allow the implementation of hybrid automata: define it with a +* function that waits for the termination of all monitors background +* activities (e.g. all timers). This hook can sleep. +*/ +#ifndef da_monitor_sync_hook +#define da_monitor_sync_hook () +#endif +/* * Type for the target id, default to int but can be overridden. * A long type can work as hash table key (PER_OBJ) but will be downgraded to
@@ -83,7 +92,8 @@ static inline void da_monitor_reset(struct da_monitor *da_mon) {da_monitor_reset_hook (da_mon);WRITE_ONCE (da_mon->monitoring,0 ); - da_mon->curr_state =model_get_initial_state (); +/* Pair with load in __ha_monitor_timer_callback */ +smp_store_release (&da_mon->curr_state,model_get_initial_state ()); }/*
@@ -181,6 +191,7 @@ static inline int da_monitor_init(void)static inline void da_monitor_destroy (void ) {da_monitor_reset_all (); +da_monitor_sync_hook (); }#ifndef da_implicit_guard
@@ -234,6 +245,7 @@ static inline int da_monitor_init(void)static inline void da_monitor_destroy (void ) {da_monitor_reset_all (); +da_monitor_sync_hook (); }#ifndef da_implicit_guard
@@ -324,6 +336,7 @@ static inline void da_monitor_destroy(void) }da_monitor_reset_all (); +da_monitor_sync_hook ();tracepoint_synchronize_unregister ();rv_put_task_monitor_slot (task_mon_slot);
@@ -503,10 +516,9 @@ static void da_monitor_reset_all(void)struct da_monitor_storage *mon_storage;int bkt; -rcu_read_lock (); +guard (rcu)();hash_for_each_rcu (da_monitor_ht, bkt, mon_storage, node)da_monitor_reset (&mon_storage->rv.da_mon); -rcu_read_unlock (); }static inline int da_monitor_init (void )
@@ -522,13 +534,17 @@ static inline void da_monitor_destroy(void)int bkt;tracepoint_synchronize_unregister (); +scoped_guard (rcu) { +hash_for_each_rcu (da_monitor_ht, bkt, mon_storage, node) { +da_monitor_reset_hook (&mon_storage->rv.da_mon); + } + } +da_monitor_sync_hook ();/* * This function is called after all probes are disabled and no longer * pending, we can safely assume no concurrent user. */ -synchronize_rcu ();hash_for_each_safe (da_monitor_ht, bkt, tmp, mon_storage, node) { -da_monitor_reset_hook (&mon_storage->rv.da_mon);hash_del_rcu (&mon_storage->node);kfree (mon_storage); }
include/rv/ha_monitor.h +17 -2
@@ -36,6 +36,7 @@ static bool ha_monitor_handle_constraint(struct da_monitor *da_mon,#define da_monitor_event_hook ha_monitor_handle_constraint#define da_monitor_init_hook ha_monitor_init_env#define da_monitor_reset_hook ha_monitor_reset_env +#define da_monitor_sync_hook () synchronize_rcu()#include <rv/da_monitor.h> #include <linux/seq_buf.h>
@@ -237,12 +238,26 @@ static bool ha_monitor_handle_constraint(struct da_monitor *da_mon,return false; } +/* +* __ha_monitor_timer_callback - generic callback representation +* +* This callback runs in an RCU read-side critical section to allow the +* destruction sequence to easily synchronize_rcu() with all pending timer +* after asynchronously disabling them. +*/ static inline void __ha_monitor_timer_callback (struct ha_monitor *ha_mon) { -enum states curr_state =READ_ONCE (ha_mon->da_mon.curr_state);DECLARE_SEQ_BUF (env_string,ENV_BUFFER_SIZE ); -u64 time_ns =ha_get_ns (); +enum states curr_state; +u64 time_ns; + +guard (rcu)(); +/* Ensure consistent curr_state if we race with da_monitor_reset */ + curr_state =smp_load_acquire (&ha_mon->da_mon.curr_state); +if (unlikely (!da_monitor_handling_event (&ha_mon->da_mon))) +return ; + time_ns =ha_get_ns ();ha_get_env_string (&env_string, ha_mon, time_ns);ha_react (curr_state,EVENT_NONE , env_string.buffer); ha_trace_error_env(ha_mon, model_get_state_name(curr_state),
Signature · 1 line
2.54.0
Gabriele Monaco
maintainerPATCH v2 9/12 Do not rely on clean monitor when initialising HA
In reply to Gabriele MonacoHybrid Automata monitors hook into the DA implementation when doing da_monitor_reset(). This function is called both on initialisation and teardown, HA monitors try to cancel a timer only when it's initialised relying on the da_mon->monitoring flag. This flag could however be corrupted during initialisation. This happens for instance on per-task monitors that share the same storage with different type of monitors like LTL or in case of races during a previous teardown.
Stop relying on the monitoring flag during initialisation, assume that can have any value, so skip timer cancellation in any case when a local flag is set. New monitors (e.g. new tasks) are always zero-initialised so they are safe.
--- include/rv/ha_monitor.h | 31 ++++++++++++++++++- kernel/trace/rv/monitors/nomiss/nomiss.c | 4 +-- kernel/trace/rv/monitors/opid/opid.c | 4 +-- kernel/trace/rv/monitors/stall/stall.c | 4 +-- .../rvgen/rvgen/templates/dot2k/main.c | 4 +-- 5 files changed, 38 insertions(+), 9 deletions(-)
include/rv/ha_monitor.h +30 -1
@@ -116,6 +116,35 @@ static enum hrtimer_restart ha_monitor_timer_callback(struct hrtimer *hrtimer);#define ha_get_ns () 0#endif /* HA_CLK_NS */ +static bool ha_mon_initializing; + +static int ha_monitor_init (void ) +{ +int ret; + + ha_mon_initializing = true; + ret =da_monitor_init (); + ha_mon_initializing = false; +return ret; +} + +static void ha_monitor_destroy (void ) +{ +da_monitor_destroy (); +} + +/* +* ha_monitor_uninitialized - are fields like the timer initialized? +* +* On a clean monitor, we can assume an active monitor (monitoring) is +* initialized, however the monitoring field cannot be trusted during +* initialization. +*/ +static inline bool ha_monitor_uninitialized (struct da_monitor *da_mon) +{ +return ha_mon_initializing || !da_monitoring (da_mon); +} +/* Should be supplied by the monitor */ static u64 ha_get_env (struct ha_monitor *ha_mon,enum envs env,u64 time_ns); static bool ha_verify_constraint(struct ha_monitor *ha_mon,
@@ -160,7 +189,7 @@ static inline void ha_monitor_reset_env(struct da_monitor *da_mon)struct ha_monitor *ha_mon =to_ha_monitor (da_mon);/* Initialisation resets the monitor before initialising the timer */ -if (likely (da_monitoring (da_mon))) +if (likely (!ha_monitor_uninitialized (da_mon)))ha_cancel_timer (ha_mon); }
kernel/trace/rv/monitors/nomiss/nomiss.c +2 -2
@@ -227,7 +227,7 @@ static int enable_nomiss(void) {int retval; - retval =da_monitor_init (); + retval =ha_monitor_init ();if (retval)return retval;
@@ -263,7 +263,7 @@ static void disable_nomiss(void)rv_detach_trace_probe ("nomiss" , sched_switch, handle_sched_switch);rv_detach_trace_probe ("nomiss" , sched_wakeup, handle_sched_wakeup); -da_monitor_destroy (); +ha_monitor_destroy (); }static struct rv_monitor rv_this = {
kernel/trace/rv/monitors/opid/opid.c +2 -2
@@ -73,7 +73,7 @@ static int enable_opid(void) {int retval; - retval =da_monitor_init (); + retval =ha_monitor_init ();if (retval)return retval;
@@ -90,7 +90,7 @@ static void disable_opid(void)rv_detach_trace_probe ("opid" , sched_set_need_resched_tp, handle_sched_need_resched);rv_detach_trace_probe ("opid" , sched_waking, handle_sched_waking); -da_monitor_destroy (); +ha_monitor_destroy (); }/*
kernel/trace/rv/monitors/stall/stall.c +2 -2
@@ -103,7 +103,7 @@ static int enable_stall(void) {int retval; - retval =da_monitor_init (); + retval =ha_monitor_init ();if (retval)return retval;
@@ -120,7 +120,7 @@ static void disable_stall(void)rv_detach_trace_probe ("stall" , sched_switch, handle_sched_switch);rv_detach_trace_probe ("stall" , sched_wakeup, handle_sched_wakeup); -da_monitor_destroy (); +ha_monitor_destroy (); }static struct rv_monitor rv_this = {
tools/verification/rvgen/rvgen/templates/dot2k/main.c +2 -2
@@ -35,7 +35,7 @@ static int enable_%%MODEL_NAME%%(void) {int retval; - retval =da_monitor_init (); + retval = %%MONITOR_CLASS%%_monitor_init();if (retval)return retval;
@@ -50,7 +50,7 @@ static void disable_%%MODEL_NAME%%(void) %%TRACEPOINT_DETACH%% -da_monitor_destroy (); + %%MONITOR_CLASS%%_monitor_destroy(); }/*
Signature · 1 line
2.54.0
Gabriele Monaco
maintainerPATCH v2 10/12 Add automatic cleanup handlers for per-task HA monitors
In reply to Gabriele MonacoHybrid automata monitors may start timers, depending on the model, these may remain active on an exiting task and cause false positives or even access freed memory.
Add an enable/disable hook in the HA code, currently only populated by the per-task handler for registration and deregistration. This hooks to the sched_process_exit event and ensures the timer is stopped for every exiting task. The handler is enabled automatically but may be disabled, for instance if the monitor uses the event for another purpose (but should still manually ensure timers are stopped).
--- include/rv/ha_monitor.h | 45 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+)
include/rv/ha_monitor.h +45 -0
@@ -28,6 +28,7 @@ static inline void ha_monitor_init_env(struct da_monitor *da_mon);static inline void ha_monitor_reset_env (struct da_monitor *da_mon);static inline void ha_setup_timer (struct ha_monitor *ha_mon);static inline bool ha_cancel_timer (struct ha_monitor *ha_mon); +static inline void ha_cancel_timer_sync (struct ha_monitor *ha_mon); static bool ha_monitor_handle_constraint(struct da_monitor *da_mon, enum states curr_state, enum events event,
@@ -38,6 +39,26 @@ static bool ha_monitor_handle_constraint(struct da_monitor *da_mon,#define da_monitor_reset_hook ha_monitor_reset_env#define da_monitor_sync_hook () synchronize_rcu() +#if !defined(HA_SKIP_AUTO_CLEANUP ) &&RV_MON_TYPE ==RV_MON_PER_TASK +/* +* Automatic cleanup handlers for per-task HA monitors, only skip if you know +* what you are doing (e.g. you want to implement cleanup manually in another +* handler doing more things). +*/ +static void ha_handle_sched_process_exit (void *data,struct task_struct *p, +bool group_dead); + +#define ha_monitor_enable_hook () \ + rv_attach_trace_probe(__stringify(MONITOR_NAME), sched_process_exit, \ + ha_handle_sched_process_exit) +#define ha_monitor_disable_hook () \ + rv_detach_trace_probe(__stringify(MONITOR_NAME), sched_process_exit, \ + ha_handle_sched_process_exit) +#else +#define ha_monitor_enable_hook () ((void)0) +#define ha_monitor_disable_hook () ((void)0) +#endif +#include <rv/da_monitor.h> #include <linux/seq_buf.h>
@@ -124,12 +145,15 @@ static int ha_monitor_init(void) ha_mon_initializing = true; ret =da_monitor_init (); +if (ret ==0 ) +ha_monitor_enable_hook (); ha_mon_initializing = false;return ret; }static void ha_monitor_destroy (void ) { +ha_monitor_disable_hook ();da_monitor_destroy (); }
@@ -230,6 +254,18 @@ static inline void ha_trace_error_env(struct ha_monitor *ha_mon, {CONCATENATE (trace_error_env_,MONITOR_NAME )(id, curr_state, event, env); } + +#if !defined(HA_SKIP_AUTO_CLEANUP ) &&RV_MON_TYPE ==RV_MON_PER_TASK +static void ha_handle_sched_process_exit (void *data,struct task_struct *p, +bool group_dead) +{ +struct da_monitor *da_mon =da_get_monitor (p); + +if (likely (!ha_monitor_uninitialized (da_mon))) +ha_cancel_timer_sync (to_ha_monitor (da_mon)); +} +#endif +#endif /* RV_MON_TYPE */ /*
@@ -456,6 +492,10 @@ static inline bool ha_cancel_timer(struct ha_monitor *ha_mon) {return timer_delete (&ha_mon->timer); } +static inline void ha_cancel_timer_sync (struct ha_monitor *ha_mon) +{ +timer_delete_sync (&ha_mon->timer); +}#elif HA_TIMER_TYPE ==HA_TIMER_HRTIMER /* * Helper functions to handle the monitor timer.
@@ -507,6 +547,10 @@ static inline bool ha_cancel_timer(struct ha_monitor *ha_mon) {return hrtimer_try_to_cancel (&ha_mon->hrtimer) ==1 ; } +static inline void ha_cancel_timer_sync (struct ha_monitor *ha_mon) +{ +hrtimer_cancel (&ha_mon->hrtimer); +}#else /* HA_TIMER_NONE */ /* * Start function is intentionally not defined, monitors using timers must
@@ -517,6 +561,7 @@ static inline bool ha_cancel_timer(struct ha_monitor *ha_mon) {return false; } +static inline void ha_cancel_timer_sync (struct ha_monitor *ha_mon) { }#endif #endif
Signature · 1 line
2.54.0
Gabriele Monaco
maintainerPATCH v2 11/12 verification/rvgen: Generate cleanup hook for per-obj monitor
In reply to Gabriele MonacoPer-object monitors can allocate memory dynamically and such memory is required for the lifetime of the object, then it should be freed with the appropriate call.
Force the generation scripts to add a cleanup function the user will need to wire to the appropriate event (e.g. sched_process_exit for tasks). This can be safely removed if the object will never cease to exist before disabling the monitor (e.g. if following only static variables).
--- tools/verification/rvgen/rvgen/dot2k.py | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
tools/verification/rvgen/rvgen/dot2k.py +15 -0
@@ -17,6 +17,9 @@ from .automata import _EventConstraintKey, _StateConstraintKey, AutomataError class dot2k(Monitor, Dot2c): template_dir = "dot2k" + # only needed for the per-obj cleanup hook + cleanup_marker = "obj_cleanup" + def __init__(self, file_path, MonitorType, extra_params={}): self.monitor_type = MonitorType Monitor.__init__(self, extra_params)
@@ -56,18 +59,30 @@ class dot2k(Monitor, Dot2c): buff.append(f"\tda_{handle}({event}{self.enum_suffix});") buff.append("}") buff.append("") + if self.monitor_type == "per_obj": + buff.append("/* XXX: obj is being destroyed, remove if not required (e.g. obj is static) */") + buff.append(f"static void handle_{self.cleanup_marker}(void *data, /* XXX: fill header */)") + buff.append("{") + buff.append("\tint id = /* XXX: how do I get the id? */;") + buff.append("\tda_destroy_storage(id);") + buff.append("}") + buff.append("") return '\n'.join(buff) def fill_tracepoint_attach_probe(self) -> str: buff = [] for event in self.events: buff.append(f"\trv_attach_trace_probe(\"{self.name}\", /* XXX: tracepoint */, handle_{event});") + if self.monitor_type == "per_obj": + buff.append(f"\trv_attach_trace_probe(\"{self.name}\", /* XXX: cleanup tracepoint */, handle_{self.cleanup_marker});") return '\n'.join(buff) def fill_tracepoint_detach_helper(self) -> str: buff = [] for event in self.events: buff.append(f"\trv_detach_trace_probe(\"{self.name}\", /* XXX: tracepoint */, handle_{event});") + if self.monitor_type == "per_obj": + buff.append(f"\trv_detach_trace_probe(\"{self.name}\", /* XXX: cleanup tracepoint */, handle_{self.cleanup_marker});") return '\n'.join(buff) def fill_model_h_header(self) -> list[str]:
Signature · 1 line
2.54.0
Gabriele Monaco
maintainerPATCH v2 12/12 verification/rvgen: Fix suffix strip in dot2k
In reply to Gabriele Monaco__start_to_invariant_check() and __get_constraint_env() parse the environment variable's name from sources that have it padded with the monitor name. This is removed using rstrip(), which is not meant to strip a substring but rather a set of characters.
Use removesuffix() to actually get rid of the trailing _<monitor name>.
--- tools/verification/rvgen/rvgen/dot2k.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
tools/verification/rvgen/rvgen/dot2k.py +2 -2
@@ -230,14 +230,14 @@ class ha2k(dot2k): def __get_constraint_env(self, constr: str) -> str: """Extract the second argument from an ha_ function""" env = constr.split("(")[1].split()[1].rstrip(")").rstrip(",") - assert env.rstrip(f"_{self.name}") in self.envs + assert env.removesuffix(f"_{self.name}") in self.envs return env def __start_to_invariant_check(self, constr: str) -> str: # by default assume the timer has ns expiration env = self.__get_constraint_env(constr) clock_type = "ns" - if self.env_types.get(env.rstrip(f"_{self.name}")) == "j": + if self.env_types.get(env.removesuffix(f"_{self.name}")) == "j": clock_type = "jiffy" return f"return ha_check_invariant_{clock_type}(ha_mon, {env}, time_ns)"
Signature · 1 line
2.54.0