Skip to contentSkip to conversation

Fixes on Deterministic and Hybrid Automata

PATCH Superseded Gabriele Monaco 3 participants 22 messages May 27 – May 31

Subsystems (2)

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Gabriele Monaco

author maintainer
#0 lore raw

v2 cover Fixes on Deterministic and Hybrid Automata

Fix issues that were reported by bots or visible only after integration:
* Make sure timers are always terminated and waited for when disabling the monitor or when the target terminates
* Run per-cpu monitors with migration disabled since preemption is now enabled from tracepoints
* Fix a wrong __user specifier in a helper function
* Other cleanup and concurrency issues

Differences since V1 [1]:
* Fix memory consistency with timer callbacks racing with resets
* Add per-obj deallocation hook in rvgen generated code
* Do not rely on clean monitor when initialising HA
* Add tracepoint synchronisation before returning per-task slots
* Fix suffix strip in dot2k
* Generate stub deallocation hooks instead of failing build when per-obj miss those

[1] - https://lore.kernel.org/lkml/[email protected]

Gabriele Monaco (11):
rv: Fix __user specifier usage in extract_params()
rv: Fix read_lock scope in per-task DA cleanup
rv: Reset per-task DA monitors before releasing the slot
rv: Prevent task migration while handling per-CPU events
rv: Prevent in-flight per-task handlers from using invalid slots
rv: Ensure all pending probes terminate on per-obj monitor destroy
rv: Ensure synchronous cleanup for HA monitors
rv: Do not rely on clean monitor when initialising HA
rv: Add automatic cleanup handlers for per-task HA monitors verification/rvgen: Generate cleanup hook for per-obj monitor verification/rvgen: Fix suffix strip in dot2k

Wen Yang (1):
rv: Fix monitor start ordering and memory ordering for monitoring flag

 include/rv/da_monitor.h                       | 67 +++++++++----
 include/rv/ha_monitor.h                       | 95 ++++++++++++++++++-
 include/rv/ltl_monitor.h                      |  1 +
 kernel/trace/rv/monitors/deadline/deadline.h  |  3 +-
 kernel/trace/rv/monitors/nomiss/nomiss.c      |  4 +-
 kernel/trace/rv/monitors/opid/opid.c          |  4 +-
 kernel/trace/rv/monitors/stall/stall.c        |  4 +-
 tools/verification/rvgen/rvgen/dot2k.py       | 19 +++-
 .../rvgen/rvgen/templates/dot2k/main.c        |  4 +-
 9 files changed, 171 insertions(+), 30 deletions(-)

base-commit: 8bc67e4db64aa72732c474b44ea8622062c903f0

Signature · 1 line
2.54.0
author maintainer
#1 lore raw

PATCH v2 1/12 Fix __user specifier usage in extract_params()

The attributes variables extracted from syscalls in the helper are both defined with the __user specifier although only the actual pointer to user data should be marked.

Remove the __user specifier from attr.

---
 kernel/trace/rv/monitors/deadline/deadline.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
kernel/trace/rv/monitors/deadline/deadline.h +2 -1
index 0bbfd2543329..78fca873d61e 100644
@@ -95,7 +95,8 @@ static inline u8 get_server_type(struct task_struct *tsk)
 static inline int extract_params(struct pt_regs *regs, long id, pid_t *pid_out)
 {
 	size_t size = offsetofend(struct sched_attr, sched_flags);
-	struct sched_attr __user *uattr, attr;
+	struct sched_attr __user *uattr;
+	struct sched_attr attr;
 	int new_policy = -1, ret;
 	unsigned long args[6];
 
Signature · 1 line
2.54.0
Reviewed-by on v2 1/12
#2 lore raw

Re: [PATCH v2 01/12] rv: Fix __user specifier usage in extract_params()

Quoted 11 lines · Gabriele Monaco

The attributes variables extracted from syscalls in the helper are both defined with the __user specifier although only the actual pointer to user data should be marked.

Remove the __user specifier from attr.

author maintainer
#3 lore raw

PATCH v2 2/12 Fix read_lock scope in per-task DA cleanup

In reply to Gabriele Monaco

The da_monitor_reset_all() function for per-task monitors takes tasklist_lock while iterating over tasks, then keeps it also while iterating over idle tasks (one per CPU). The latter is not necessary since the lock needs to guard only for_each_process_thread().

Use a scoped_guard for more compact syntax and adjust the scope only where the lock is necessary.

Fixes 30984ccf31b7f ("rv: Refactor da_monitor to minimise macros")
Fixes 8259cb14a7068 ("rv: Reset per-task monitors also for idle tasks")
---
 include/rv/da_monitor.h | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
include/rv/da_monitor.h +4 -4
index 39765ff6f098..250888812125 100644
@@ -272,12 +272,12 @@ static void da_monitor_reset_all(void)
 	struct task_struct *g, *p;
 	int cpu;
 
-	read_lock(&tasklist_lock);
-	for_each_process_thread(g, p)
-		da_monitor_reset(da_get_monitor(p));
+	scoped_guard(read_lock, &tasklist_lock) {
+		for_each_process_thread(g, p)
+			da_monitor_reset(da_get_monitor(p));
+	}
 	for_each_present_cpu(cpu)
 		da_monitor_reset(da_get_monitor(idle_task(cpu)));
-	read_unlock(&tasklist_lock);
 }
 
 /*
Signature · 1 line
2.54.0
Reviewed-by on v2 2/12
#4 lore raw

Re: [PATCH v2 02/12] rv: Fix read_lock scope in per-task DA cleanup

Quoted 10 lines · Gabriele Monaco

The da_monitor_reset_all() function for per-task monitors takes tasklist_lock while iterating over tasks, then keeps it also while iterating over idle tasks (one per CPU). The latter is not necessary since the lock needs to guard only for_each_process_thread().

Use a scoped_guard for more compact syntax and adjust the scope only where the lock is necessary.

Fixes 30984ccf31b7f ("rv: Refactor da_monitor to minimise macros")
Fixes 8259cb14a7068 ("rv: Reset per-task monitors also for idle tasks")

Fixes: tag "indicates that the patch fixes a bug in a previous commit". There is no bug here, so I don't think Fixes tags are applicable.

Quoted 2 lines
author maintainer
#5 lore raw

Re: [PATCH v2 02/12] rv: Fix read_lock scope in per-task DA cleanup

Quoted 11 lines · Nam Cao
Quoted 10 lines · Gabriele Monaco

The da_monitor_reset_all() function for per-task monitors takes tasklist_lock while iterating over tasks, then keeps it also while iterating over idle tasks (one per CPU). The latter is not necessary
since the lock needs to guard only for_each_process_thread().

Use a scoped_guard for more compact syntax and adjust the scope only
where the lock is necessary.

Quoted 7 lines
Quoted 3 lines
Fixes 30984ccf31b7f ("rv: Refactor da_monitor to minimise macros")
Fixes 8259cb14a7068 ("rv: Reset per-task monitors also for idle

tasks")

Fixes: tag "indicates that the patch fixes a bug in a previous commit". There is no bug here, so I don't think Fixes tags are applicable.

Yeah good point, that isn't a real bug.. We're just holding a lock for a bit too long but there's no harm in that. Will remove the tags.

Thanks,
Gabriele

Quoted 5 lines
Quoted 3 lines
author maintainer
#6 lore raw

PATCH v2 3/12 Reset per-task DA monitors before releasing the slot

In reply to Gabriele Monaco

Per-task monitors use task_mon_slot to determine which slot in the array to use for the monitor. During destruction, this slot is returned but this is done before resetting the monitor. As a result, the monitor's reset is in fact resetting a slot that is outside of the array (RV_PER_TASK_MONITOR_INIT).

Release the slot only after the reset to avoid out-of-bound memory access.

Fixes f5587d1b6ec93 ("rv: Add Hybrid Automata monitor type")
---
 include/rv/da_monitor.h | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
include/rv/da_monitor.h +3 -2
index 250888812125..0b7028df08fb 100644
@@ -309,10 +309,11 @@ static inline void da_monitor_destroy(void)
 		WARN_ONCE(1, "Disabling a disabled monitor: " __stringify(MONITOR_NAME));
 		return;
 	}
-	rv_put_task_monitor_slot(task_mon_slot);
-	task_mon_slot = RV_PER_TASK_MONITOR_INIT;
 
 	da_monitor_reset_all();
+
+	rv_put_task_monitor_slot(task_mon_slot);
+	task_mon_slot = RV_PER_TASK_MONITOR_INIT;
 }
 
 #elif RV_MON_TYPE == RV_MON_PER_OBJ
Signature · 1 line
2.54.0
Reviewed-by on v2 3/12
#7 lore raw

Re: [PATCH v2 03/12] rv: Reset per-task DA monitors before releasing the slot

Quoted 5 lines · Gabriele Monaco

Per-task monitors use task_mon_slot to determine which slot in the array to use for the monitor. During destruction, this slot is returned but this is done before resetting the monitor. As a result, the monitor's reset is in fact resetting a slot that is outside of the array (RV_PER_TASK_MONITOR_INIT).

Oh crap.

Quoted 2 lines

Release the slot only after the reset to avoid out-of-bound memory access.

I think KASAN can catch this type of issue.

Quoted 4 lines
Fixes f5587d1b6ec93 ("rv: Add Hybrid Automata monitor type")

Should we have
Cc: [email protected]
?

author maintainer
#8 lore raw

PATCH v2 4/12 Prevent task migration while handling per-CPU events

In reply to Gabriele Monaco

Tracepoint handlers are now fully preemptible. When a per-CPU monitor handles an event, it retrieves the monitor state using a per-CPU pointer. If the event itself doesn't disable preemption, the task can migrate to a different CPU and we risk updating the wrong monitor.

Mitigate this by explicitly disabling task migration before acquiring the monitor pointer. This cannot guarantee the monitor runs on the correct CPU but reduces the race condition window and prevents warnings.

---
 include/rv/da_monitor.h | 11 +++++++++++
 1 file changed, 11 insertions(+)
include/rv/da_monitor.h +11 -0
index 0b7028df08fb..a9fd284195ee 100644
@@ -181,6 +181,10 @@ static inline void da_monitor_destroy(void)
 	da_monitor_reset_all();
 }
 
+#ifndef da_implicit_guard
+#define da_implicit_guard()
+#endif
+
 #elif RV_MON_TYPE == RV_MON_PER_CPU
 /*
  * Functions to define, init and get a per-cpu monitor.
@@ -230,6 +234,10 @@ static inline void da_monitor_destroy(void)
 	da_monitor_reset_all();
 }
 
+#ifndef da_implicit_guard
+#define da_implicit_guard() guard(migrate)()
+#endif
+
 #elif RV_MON_TYPE == RV_MON_PER_TASK
 /*
  * Functions to define, init and get a per-task monitor.
@@ -677,6 +685,7 @@ static inline bool __da_handle_start_run_event(struct da_monitor *da_mon,
  */
 static inline void da_handle_event(enum events event)
 {
+	da_implicit_guard();
 	__da_handle_event(da_get_monitor(), event, 0);
 }
 
@@ -692,6 +701,7 @@ static inline void da_handle_event(enum events event)
  */
 static inline bool da_handle_start_event(enum events event)
 {
+	da_implicit_guard();
 	return __da_handle_start_event(da_get_monitor(), event, 0);
 }
 
@@ -703,6 +713,7 @@ static inline bool da_handle_start_event(enum events event)
  */
 static inline bool da_handle_start_run_event(enum events event)
 {
+	da_implicit_guard();
 	return __da_handle_start_run_event(da_get_monitor(), event, 0);
 }
 
Signature · 1 line
2.54.0
Reviewed-by on v2 4/12
#9 lore raw

Re: [PATCH v2 04/12] rv: Prevent task migration while handling per-CPU events

Quoted 1 line · Gabriele Monaco

Tracepoint handlers are now fully preemptible.

Would be nice to mention the commit that did this:

a46023d5616e ("tracing: Guard __DECLARE_TRACE() use of __DO_TRACE_CALL() with SRCU-fast")

Quoted 11 lines

When a per-CPU monitor
handles an event, it retrieves the monitor state using a per-CPU pointer. If the event itself doesn't disable preemption, the task can migrate to a different CPU and we risk updating the wrong monitor.

Mitigate this by explicitly disabling task migration before acquiring the monitor pointer. This cannot guarantee the monitor runs on the correct CPU but reduces the race condition window and prevents warnings.

author maintainer
#10 lore raw

PATCH v2 5/12 Prevent in-flight per-task handlers from using invalid slots

In reply to Gabriele Monaco

Per-task monitors use a slot in the task_struct->rv[] array and store that locally (e.g. task_mon_slot), this slot is returned during the destruction process but currently hanlers can be running while that slot is returning and this race may lead to accessing an invalid slot.

Synchronise with all in-flight tracepoint handlers using tracepoint_synchronize_unregister() before returning the slot.

Fixes f5587d1b6ec9 ("rv: Add Hybrid Automata monitor type")
Fixes a9769a5b9878 ("rv: Add support for LTL monitors")
---
 include/rv/da_monitor.h  | 4 ++++
 include/rv/ltl_monitor.h | 1 +
 2 files changed, 5 insertions(+)
include/rv/da_monitor.h +4 -0
index a9fd284195ee..446a4d53d99c 100644
@@ -310,6 +310,9 @@ static int da_monitor_init(void)
 
 /*
  * da_monitor_destroy - return the allocated slot
+ *
+ * Wait for all in-flight handlers before returning the slot to avoid
+ * out-of-bound accesses.
  */
 static inline void da_monitor_destroy(void)
 {
@@ -320,6 +323,7 @@ static inline void da_monitor_destroy(void)
 
 	da_monitor_reset_all();
 
+	tracepoint_synchronize_unregister();
 	rv_put_task_monitor_slot(task_mon_slot);
 	task_mon_slot = RV_PER_TASK_MONITOR_INIT;
 }
include/rv/ltl_monitor.h +1 -0
index eff60cd61106..38e792401f76 100644
@@ -77,6 +77,7 @@ static void ltl_monitor_destroy(void)
 {
 	rv_detach_trace_probe(name, task_newtask, handle_task_newtask);
 
+	tracepoint_synchronize_unregister();
 	rv_put_task_monitor_slot(ltl_monitor_slot);
 	ltl_monitor_slot = RV_PER_TASK_MONITOR_INIT;
 }
Signature · 1 line
2.54.0
Reviewed-by on v2 5/12
#11 lore raw

Re: [PATCH v2 05/12] rv: Prevent in-flight per-task handlers from using invalid slots

Quoted 5 lines · Gabriele Monaco

Per-task monitors use a slot in the task_struct->rv[] array and store that locally (e.g. task_mon_slot), this slot is returned during the destruction process but currently hanlers can be running while that slot is returning and this race may lead to accessing an invalid slot.

Quoted 7 lines

Synchronise with all in-flight tracepoint handlers using tracepoint_synchronize_unregister() before returning the slot.

Fixes f5587d1b6ec9 ("rv: Add Hybrid Automata monitor type")
Fixes a9769a5b9878 ("rv: Add support for LTL monitors")
author maintainer
#12 lore raw

PATCH v2 6/12 Ensure all pending probes terminate on per-obj monitor destroy

In reply to Gabriele Monaco

The monitor disable/destroy sequence detaches all probes and resets the monitor's data, however it doesn't wait for pending probes. This is an issue with per-object monitors, which free the monitor storage.

Call tracepoint_synchronize_unregister() to make sure to wait for all pending probes before destroying the monitor storage.

Fixes 4a24127bd6cb ("rv: Add support for per-object monitors in DA/HA")
---
 include/rv/da_monitor.h | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
include/rv/da_monitor.h +3 -2
index 446a4d53d99c..5f790e1694b4 100644
@@ -519,9 +519,10 @@ static inline void da_monitor_destroy(void)
 	struct hlist_node *tmp;
 	int bkt;
 
+	tracepoint_synchronize_unregister();
 	/*
-	 * This function is called after all probes are disabled, we need only
-	 * worry about concurrency against old events.
+	 * This function is called after all probes are disabled and no longer
+	 * pending, we can safely assume no concurrent user.
 	 */
 	synchronize_rcu();
 	hash_for_each_safe(da_monitor_ht, bkt, tmp, mon_storage, node) {
Signature · 1 line
2.54.0
Reviewed-by on v2 6/12
#13 lore raw

Re: [PATCH v2 06/12] rv: Ensure all pending probes terminate on per-obj monitor destroy

Quoted 10 lines · Gabriele Monaco

The monitor disable/destroy sequence detaches all probes and resets the monitor's data, however it doesn't wait for pending probes. This is an issue with per-object monitors, which free the monitor storage.

Call tracepoint_synchronize_unregister() to make sure to wait for all pending probes before destroying the monitor storage.

Fixes 4a24127bd6cb ("rv: Add support for per-object monitors in DA/HA")
author maintainer
#14 lore raw

PATCH v2 7/12 Fix monitor start ordering and memory ordering for monitoring flag

In reply to Gabriele Monaco

From: Wen Yang <[email protected]>

da_monitor_start() set monitoring=1 before calling da_monitor_init_hook(), may racing with the sched_switch handler:

  da_monitor_start()               sched_switch handler
  -------------------------        ---------------------------------
  da_mon->monitoring = 1;
                                   if (da_monitoring(da_mon))  /* true  */
                                       ha_start_timer_ns(...);
                                       /* hrtimer->base == NULL, crash */
  da_monitor_init_hook(da_mon);
  /* hrtimer_setup() sets base */

Fix the ordering and pair with release/acquire semantics:

  da_monitor_init_hook(da_mon);
  smp_store_release(&da_mon->monitoring, 1);    /* da_monitor_start()  */
  return smp_load_acquire(&da_mon->monitoring); /* da_monitoring()     */

On ARM64 a plain STR + LDR does not form a release-acquire pair, so the load can observe monitoring=1 while hrtimer->base is still NULL. The plain accesses are also data races under KCSAN.

Use WRITE_ONCE for the monitoring=0 store in da_monitor_reset() to cover the reset path.

Fixes 792575348ff7 ("rv/include: Add deterministic automata monitor definition via C macros")
---
 include/rv/da_monitor.h | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
include/rv/da_monitor.h +5 -3
index 5f790e1694b4..c1e347f9deaf 100644
@@ -82,7 +82,7 @@ static void react(enum states curr_state, enum events event)
 static inline void da_monitor_reset(struct da_monitor *da_mon)
 {
 	da_monitor_reset_hook(da_mon);
-	da_mon->monitoring = 0;
+	WRITE_ONCE(da_mon->monitoring, 0);
 	da_mon->curr_state = model_get_initial_state();
 }
 
@@ -95,8 +95,9 @@ static inline void da_monitor_reset(struct da_monitor *da_mon)
 static inline void da_monitor_start(struct da_monitor *da_mon)
 {
 	da_mon->curr_state = model_get_initial_state();
-	da_mon->monitoring = 1;
 	da_monitor_init_hook(da_mon);
+	/* Pairs with smp_load_acquire in da_monitoring(). */
+	smp_store_release(&da_mon->monitoring, 1);
 }
 
 /*
@@ -104,7 +105,8 @@ static inline void da_monitor_start(struct da_monitor *da_mon)
  */
 static inline bool da_monitoring(struct da_monitor *da_mon)
 {
-	return da_mon->monitoring;
+	/* Pairs with smp_store_release in da_monitor_start(). */
+	return smp_load_acquire(&da_mon->monitoring);
 }
 
 /*
Signature · 1 line
2.54.0
Reviewed-by on v2 7/12
#15 lore raw

Re: [PATCH v2 07/12] rv: Fix monitor start ordering and memory ordering for monitoring flag

Quoted 24 lines · Gabriele Monaco

From: Wen Yang <[email protected]>

da_monitor_start() set monitoring=1 before calling da_monitor_init_hook(), may racing with the sched_switch handler:

  da_monitor_start()               sched_switch handler
  -------------------------        ---------------------------------
  da_mon->monitoring = 1;
                                   if (da_monitoring(da_mon))  /* true  */
                                       ha_start_timer_ns(...);
                                       /* hrtimer->base == NULL, crash */
  da_monitor_init_hook(da_mon);
  /* hrtimer_setup() sets base */

Fix the ordering and pair with release/acquire semantics:

  da_monitor_init_hook(da_mon);
  smp_store_release(&da_mon->monitoring, 1);    /* da_monitor_start()  */
  return smp_load_acquire(&da_mon->monitoring); /* da_monitoring()     */

On ARM64 a plain STR + LDR does not form a release-acquire pair, so the load can observe monitoring=1 while hrtimer->base is still NULL. The plain accesses are also data races under KCSAN.

Quoted 7 lines

Use WRITE_ONCE for the monitoring=0 store in da_monitor_reset() to cover the reset path.

Fixes 792575348ff7 ("rv/include: Add deterministic automata monitor definition via C macros")

Looks correct to me.

Wen, I am curious, how did you find this issue?

Nam

#16 lore raw

Re: [PATCH v2 07/12] rv: Fix monitor start ordering and memory ordering for monitoring flag

Quoted 33 lines · Nam Cao
Quoted 24 lines · Gabriele Monaco

From: Wen Yang <[email protected]>

da_monitor_start() set monitoring=1 before calling da_monitor_init_hook(), may racing with the sched_switch handler:

   da_monitor_start()               sched_switch handler
   -------------------------        ---------------------------------
   da_mon->monitoring = 1;
                                    if (da_monitoring(da_mon))  /* true  */
                                        ha_start_timer_ns(...);
                                        /* hrtimer->base == NULL, crash */
   da_monitor_init_hook(da_mon);
   /* hrtimer_setup() sets base */

Fix the ordering and pair with release/acquire semantics:

   da_monitor_init_hook(da_mon);
   smp_store_release(&da_mon->monitoring, 1);    /* da_monitor_start()  */
   return smp_load_acquire(&da_mon->monitoring); /* da_monitoring()     */

On ARM64 a plain STR + LDR does not form a release-acquire pair, so the load can observe monitoring=1 while hrtimer->base is still NULL. The plain accesses are also data races under KCSAN.

Quoted 7 lines

Use WRITE_ONCE for the monitoring=0 store in da_monitor_reset() to cover the reset path.

Fixes 792575348ff7 ("rv/include: Add deterministic automata monitor definition via C macros")
Quoted 5 lines

Looks correct to me.

Wen, I am curious, how did you find this issue?

It was caught during development of the tlob monitor by a KUnit test

(since moved to selftests) that monitored a kthread running concurrently

on another CPU. On a multi-core box the window between setting

monitoring=1 and da_monitor_init_hook() was wide enough to trigger the

crash, and KCSAN flagged the plain monitoring accesses as data
races as well.

--
Best wishes,
Wen

author maintainer
#17 lore raw

PATCH v2 8/12 Ensure synchronous cleanup for HA monitors

In reply to Gabriele Monaco

HA monitors may start timers, all cleanup functions currently stop the timers asynchronously to avoid sleeping in the wrong context. Nothing makes sure running callbacks terminate on cleanup.

Run the entire HA timer callback in an RCU read-side critical section, this way we can simply synchronize_rcu() with any pending timer and are sure any cleanup using kfree_rcu() runs after callbacks terminated. Additionally make sure any unlikely callback running late won't run any code if the monitor is marked as disabled.
Use memory barriers to serialise with racing resets.

Fixes f5587d1b6ec9 ("rv: Add Hybrid Automata monitor type")
Fixes 4a24127bd6cb ("rv: Add support for per-object monitors in DA/HA")
---
 include/rv/da_monitor.h | 26 +++++++++++++++++++++-----
 include/rv/ha_monitor.h | 19 +++++++++++++++++--
 2 files changed, 38 insertions(+), 7 deletions(-)
include/rv/da_monitor.h +21 -5
index c1e347f9deaf..268f22933663 100644
@@ -57,6 +57,15 @@ static struct rv_monitor rv_this;
 #define da_monitor_reset_hook(da_mon)
 #endif
 
+/*
+ * Hook to allow the implementation of hybrid automata: define it with a
+ * function that waits for the termination of all monitors background
+ * activities (e.g. all timers). This hook can sleep.
+ */
+#ifndef da_monitor_sync_hook
+#define da_monitor_sync_hook()
+#endif
+
 /*
  * Type for the target id, default to int but can be overridden.
  * A long type can work as hash table key (PER_OBJ) but will be downgraded to
@@ -83,7 +92,8 @@ static inline void da_monitor_reset(struct da_monitor *da_mon)
 {
 	da_monitor_reset_hook(da_mon);
 	WRITE_ONCE(da_mon->monitoring, 0);
-	da_mon->curr_state = model_get_initial_state();
+	/* Pair with load in __ha_monitor_timer_callback */
+	smp_store_release(&da_mon->curr_state, model_get_initial_state());
 }
 
 /*
@@ -181,6 +191,7 @@ static inline int da_monitor_init(void)
 static inline void da_monitor_destroy(void)
 {
 	da_monitor_reset_all();
+	da_monitor_sync_hook();
 }
 
 #ifndef da_implicit_guard
@@ -234,6 +245,7 @@ static inline int da_monitor_init(void)
 static inline void da_monitor_destroy(void)
 {
 	da_monitor_reset_all();
+	da_monitor_sync_hook();
 }
 
 #ifndef da_implicit_guard
@@ -324,6 +336,7 @@ static inline void da_monitor_destroy(void)
 	}
 
 	da_monitor_reset_all();
+	da_monitor_sync_hook();
 
 	tracepoint_synchronize_unregister();
 	rv_put_task_monitor_slot(task_mon_slot);
@@ -503,10 +516,9 @@ static void da_monitor_reset_all(void)
 	struct da_monitor_storage *mon_storage;
 	int bkt;
 
-	rcu_read_lock();
+	guard(rcu)();
 	hash_for_each_rcu(da_monitor_ht, bkt, mon_storage, node)
 		da_monitor_reset(&mon_storage->rv.da_mon);
-	rcu_read_unlock();
 }
 
 static inline int da_monitor_init(void)
@@ -522,13 +534,17 @@ static inline void da_monitor_destroy(void)
 	int bkt;
 
 	tracepoint_synchronize_unregister();
+	scoped_guard(rcu) {
+		hash_for_each_rcu(da_monitor_ht, bkt, mon_storage, node) {
+			da_monitor_reset_hook(&mon_storage->rv.da_mon);
+		}
+	}
+	da_monitor_sync_hook();
 	/*
 	 * This function is called after all probes are disabled and no longer
 	 * pending, we can safely assume no concurrent user.
 	 */
-	synchronize_rcu();
 	hash_for_each_safe(da_monitor_ht, bkt, tmp, mon_storage, node) {
-		da_monitor_reset_hook(&mon_storage->rv.da_mon);
 		hash_del_rcu(&mon_storage->node);
 		kfree(mon_storage);
 	}
include/rv/ha_monitor.h +17 -2
index d59507e8cb30..661631bc933a 100644
@@ -36,6 +36,7 @@ static bool ha_monitor_handle_constraint(struct da_monitor *da_mon,
 #define da_monitor_event_hook ha_monitor_handle_constraint
 #define da_monitor_init_hook ha_monitor_init_env
 #define da_monitor_reset_hook ha_monitor_reset_env
+#define da_monitor_sync_hook() synchronize_rcu()
 
 #include <rv/da_monitor.h>
 #include <linux/seq_buf.h>
@@ -237,12 +238,26 @@ static bool ha_monitor_handle_constraint(struct da_monitor *da_mon,
 	return false;
 }
 
+/*
+ * __ha_monitor_timer_callback - generic callback representation
+ *
+ * This callback runs in an RCU read-side critical section to allow the
+ * destruction sequence to easily synchronize_rcu() with all pending timer
+ * after asynchronously disabling them.
+ */
 static inline void __ha_monitor_timer_callback(struct ha_monitor *ha_mon)
 {
-	enum states curr_state = READ_ONCE(ha_mon->da_mon.curr_state);
 	DECLARE_SEQ_BUF(env_string, ENV_BUFFER_SIZE);
-	u64 time_ns = ha_get_ns();
+	enum states curr_state;
+	u64 time_ns;
+
+	guard(rcu)();
+	/* Ensure consistent curr_state if we race with da_monitor_reset */
+	curr_state = smp_load_acquire(&ha_mon->da_mon.curr_state);
+	if (unlikely(!da_monitor_handling_event(&ha_mon->da_mon)))
+		return;
 
+	time_ns = ha_get_ns();
 	ha_get_env_string(&env_string, ha_mon, time_ns);
 	ha_react(curr_state, EVENT_NONE, env_string.buffer);
 	ha_trace_error_env(ha_mon, model_get_state_name(curr_state),
Signature · 1 line
2.54.0
author maintainer
#18 lore raw

PATCH v2 9/12 Do not rely on clean monitor when initialising HA

In reply to Gabriele Monaco

Hybrid Automata monitors hook into the DA implementation when doing da_monitor_reset(). This function is called both on initialisation and teardown, HA monitors try to cancel a timer only when it's initialised relying on the da_mon->monitoring flag. This flag could however be corrupted during initialisation. This happens for instance on per-task monitors that share the same storage with different type of monitors like LTL or in case of races during a previous teardown.

Stop relying on the monitoring flag during initialisation, assume that can have any value, so skip timer cancellation in any case when a local flag is set. New monitors (e.g. new tasks) are always zero-initialised so they are safe.

---
 include/rv/ha_monitor.h                       | 31 ++++++++++++++++++-
 kernel/trace/rv/monitors/nomiss/nomiss.c      |  4 +--
 kernel/trace/rv/monitors/opid/opid.c          |  4 +--
 kernel/trace/rv/monitors/stall/stall.c        |  4 +--
 .../rvgen/rvgen/templates/dot2k/main.c        |  4 +--
 5 files changed, 38 insertions(+), 9 deletions(-)
include/rv/ha_monitor.h +30 -1
index 661631bc933a..2d7187c16f55 100644
@@ -116,6 +116,35 @@ static enum hrtimer_restart ha_monitor_timer_callback(struct hrtimer *hrtimer);
 #define ha_get_ns() 0
 #endif /* HA_CLK_NS */
 
+static bool ha_mon_initializing;
+
+static int ha_monitor_init(void)
+{
+	int ret;
+
+	ha_mon_initializing = true;
+	ret = da_monitor_init();
+	ha_mon_initializing = false;
+	return ret;
+}
+
+static void ha_monitor_destroy(void)
+{
+	da_monitor_destroy();
+}
+
+/*
+ * ha_monitor_uninitialized - are fields like the timer initialized?
+ *
+ * On a clean monitor, we can assume an active monitor (monitoring) is
+ * initialized, however the monitoring field cannot be trusted during
+ * initialization.
+ */
+static inline bool ha_monitor_uninitialized(struct da_monitor *da_mon)
+{
+	return ha_mon_initializing || !da_monitoring(da_mon);
+}
+
 /* Should be supplied by the monitor */
 static u64 ha_get_env(struct ha_monitor *ha_mon, enum envs env, u64 time_ns);
 static bool ha_verify_constraint(struct ha_monitor *ha_mon,
@@ -160,7 +189,7 @@ static inline void ha_monitor_reset_env(struct da_monitor *da_mon)
 	struct ha_monitor *ha_mon = to_ha_monitor(da_mon);
 
 	/* Initialisation resets the monitor before initialising the timer */
-	if (likely(da_monitoring(da_mon)))
+	if (likely(!ha_monitor_uninitialized(da_mon)))
 		ha_cancel_timer(ha_mon);
 }
 
kernel/trace/rv/monitors/nomiss/nomiss.c +2 -2
index 31f90f3638d8..8ead8783c29f 100644
@@ -227,7 +227,7 @@ static int enable_nomiss(void)
 {
 	int retval;
 
-	retval = da_monitor_init();
+	retval = ha_monitor_init();
 	if (retval)
 		return retval;
 
@@ -263,7 +263,7 @@ static void disable_nomiss(void)
 	rv_detach_trace_probe("nomiss", sched_switch, handle_sched_switch);
 	rv_detach_trace_probe("nomiss", sched_wakeup, handle_sched_wakeup);
 
-	da_monitor_destroy();
+	ha_monitor_destroy();
 }
 
 static struct rv_monitor rv_this = {
kernel/trace/rv/monitors/opid/opid.c +2 -2
index 4594c7c46601..2922318c6112 100644
@@ -73,7 +73,7 @@ static int enable_opid(void)
 {
 	int retval;
 
-	retval = da_monitor_init();
+	retval = ha_monitor_init();
 	if (retval)
 		return retval;
 
@@ -90,7 +90,7 @@ static void disable_opid(void)
 	rv_detach_trace_probe("opid", sched_set_need_resched_tp, handle_sched_need_resched);
 	rv_detach_trace_probe("opid", sched_waking, handle_sched_waking);
 
-	da_monitor_destroy();
+	ha_monitor_destroy();
 }
 
 /*
kernel/trace/rv/monitors/stall/stall.c +2 -2
index 9ccfda6b0e73..3c38fb1a0159 100644
@@ -103,7 +103,7 @@ static int enable_stall(void)
 {
 	int retval;
 
-	retval = da_monitor_init();
+	retval = ha_monitor_init();
 	if (retval)
 		return retval;
 
@@ -120,7 +120,7 @@ static void disable_stall(void)
 	rv_detach_trace_probe("stall", sched_switch, handle_sched_switch);
 	rv_detach_trace_probe("stall", sched_wakeup, handle_sched_wakeup);
 
-	da_monitor_destroy();
+	ha_monitor_destroy();
 }
 
 static struct rv_monitor rv_this = {
tools/verification/rvgen/rvgen/templates/dot2k/main.c +2 -2
index bf0999f6657a..889446760e3c 100644
@@ -35,7 +35,7 @@ static int enable_%%MODEL_NAME%%(void)
 {
 	int retval;
 
-	retval = da_monitor_init();
+	retval = %%MONITOR_CLASS%%_monitor_init();
 	if (retval)
 		return retval;
 
@@ -50,7 +50,7 @@ static void disable_%%MODEL_NAME%%(void)
 
 %%TRACEPOINT_DETACH%%
 
-	da_monitor_destroy();
+	%%MONITOR_CLASS%%_monitor_destroy();
 }
 
 /*
Signature · 1 line
2.54.0
author maintainer
#19 lore raw

PATCH v2 10/12 Add automatic cleanup handlers for per-task HA monitors

In reply to Gabriele Monaco

Hybrid automata monitors may start timers, depending on the model, these may remain active on an exiting task and cause false positives or even access freed memory.

Add an enable/disable hook in the HA code, currently only populated by the per-task handler for registration and deregistration. This hooks to the sched_process_exit event and ensures the timer is stopped for every exiting task. The handler is enabled automatically but may be disabled, for instance if the monitor uses the event for another purpose (but should still manually ensure timers are stopped).

Fixes f5587d1b6ec9 ("rv: Add Hybrid Automata monitor type")
---
 include/rv/ha_monitor.h | 45 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 45 insertions(+)
include/rv/ha_monitor.h +45 -0
index 2d7187c16f55..df67d5e9d132 100644
@@ -28,6 +28,7 @@ static inline void ha_monitor_init_env(struct da_monitor *da_mon);
 static inline void ha_monitor_reset_env(struct da_monitor *da_mon);
 static inline void ha_setup_timer(struct ha_monitor *ha_mon);
 static inline bool ha_cancel_timer(struct ha_monitor *ha_mon);
+static inline void ha_cancel_timer_sync(struct ha_monitor *ha_mon);
 static bool ha_monitor_handle_constraint(struct da_monitor *da_mon,
 					 enum states curr_state,
 					 enum events event,
@@ -38,6 +39,26 @@ static bool ha_monitor_handle_constraint(struct da_monitor *da_mon,
 #define da_monitor_reset_hook ha_monitor_reset_env
 #define da_monitor_sync_hook() synchronize_rcu()
 
+#if !defined(HA_SKIP_AUTO_CLEANUP) && RV_MON_TYPE == RV_MON_PER_TASK
+/*
+ * Automatic cleanup handlers for per-task HA monitors, only skip if you know
+ * what you are doing (e.g. you want to implement cleanup manually in another
+ * handler doing more things).
+ */
+static void ha_handle_sched_process_exit(void *data, struct task_struct *p,
+					 bool group_dead);
+
+#define ha_monitor_enable_hook()                                             \
+	rv_attach_trace_probe(__stringify(MONITOR_NAME), sched_process_exit, \
+			      ha_handle_sched_process_exit)
+#define ha_monitor_disable_hook()                                            \
+	rv_detach_trace_probe(__stringify(MONITOR_NAME), sched_process_exit, \
+			      ha_handle_sched_process_exit)
+#else
+#define ha_monitor_enable_hook() ((void)0)
+#define ha_monitor_disable_hook() ((void)0)
+#endif
+
 #include <rv/da_monitor.h>
 #include <linux/seq_buf.h>
 
@@ -124,12 +145,15 @@ static int ha_monitor_init(void)
 
 	ha_mon_initializing = true;
 	ret = da_monitor_init();
+	if (ret == 0)
+		ha_monitor_enable_hook();
 	ha_mon_initializing = false;
 	return ret;
 }
 
 static void ha_monitor_destroy(void)
 {
+	ha_monitor_disable_hook();
 	da_monitor_destroy();
 }
 
@@ -230,6 +254,18 @@ static inline void ha_trace_error_env(struct ha_monitor *ha_mon,
 {
 	CONCATENATE(trace_error_env_, MONITOR_NAME)(id, curr_state, event, env);
 }
+
+#if !defined(HA_SKIP_AUTO_CLEANUP) && RV_MON_TYPE == RV_MON_PER_TASK
+static void ha_handle_sched_process_exit(void *data, struct task_struct *p,
+					 bool group_dead)
+{
+	struct da_monitor *da_mon = da_get_monitor(p);
+
+	if (likely(!ha_monitor_uninitialized(da_mon)))
+		ha_cancel_timer_sync(to_ha_monitor(da_mon));
+}
+#endif
+
 #endif /* RV_MON_TYPE */
 
 /*
@@ -456,6 +492,10 @@ static inline bool ha_cancel_timer(struct ha_monitor *ha_mon)
 {
 	return timer_delete(&ha_mon->timer);
 }
+static inline void ha_cancel_timer_sync(struct ha_monitor *ha_mon)
+{
+	timer_delete_sync(&ha_mon->timer);
+}
 #elif HA_TIMER_TYPE == HA_TIMER_HRTIMER
 /*
  * Helper functions to handle the monitor timer.
@@ -507,6 +547,10 @@ static inline bool ha_cancel_timer(struct ha_monitor *ha_mon)
 {
 	return hrtimer_try_to_cancel(&ha_mon->hrtimer) == 1;
 }
+static inline void ha_cancel_timer_sync(struct ha_monitor *ha_mon)
+{
+	hrtimer_cancel(&ha_mon->hrtimer);
+}
 #else /* HA_TIMER_NONE */
 /*
  * Start function is intentionally not defined, monitors using timers must
@@ -517,6 +561,7 @@ static inline bool ha_cancel_timer(struct ha_monitor *ha_mon)
 {
 	return false;
 }
+static inline void ha_cancel_timer_sync(struct ha_monitor *ha_mon) { }
 #endif
 
 #endif
Signature · 1 line
2.54.0
author maintainer
#20 lore raw

PATCH v2 11/12 verification/rvgen: Generate cleanup hook for per-obj monitor

In reply to Gabriele Monaco

Per-object monitors can allocate memory dynamically and such memory is required for the lifetime of the object, then it should be freed with the appropriate call.

Force the generation scripts to add a cleanup function the user will need to wire to the appropriate event (e.g. sched_process_exit for tasks). This can be safely removed if the object will never cease to exist before disabling the monitor (e.g. if following only static variables).

---
 tools/verification/rvgen/rvgen/dot2k.py | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
tools/verification/rvgen/rvgen/dot2k.py +15 -0
index e6f476b903b0..a17294283a1f 100644
@@ -17,6 +17,9 @@ from .automata import _EventConstraintKey, _StateConstraintKey, AutomataError
 class dot2k(Monitor, Dot2c):
     template_dir = "dot2k"
 
+    # only needed for the per-obj cleanup hook
+    cleanup_marker = "obj_cleanup"
+
     def __init__(self, file_path, MonitorType, extra_params={}):
         self.monitor_type = MonitorType
         Monitor.__init__(self, extra_params)
@@ -56,18 +59,30 @@ class dot2k(Monitor, Dot2c):
                 buff.append(f"\tda_{handle}({event}{self.enum_suffix});")
             buff.append("}")
             buff.append("")
+        if self.monitor_type == "per_obj":
+            buff.append("/* XXX: obj is being destroyed, remove if not required (e.g. obj is static) */")
+            buff.append(f"static void handle_{self.cleanup_marker}(void *data, /* XXX: fill header */)")
+            buff.append("{")
+            buff.append("\tint id = /* XXX: how do I get the id? */;")
+            buff.append("\tda_destroy_storage(id);")
+            buff.append("}")
+            buff.append("")
         return '\n'.join(buff)
 
     def fill_tracepoint_attach_probe(self) -> str:
         buff = []
         for event in self.events:
             buff.append(f"\trv_attach_trace_probe(\"{self.name}\", /* XXX: tracepoint */, handle_{event});")
+        if self.monitor_type == "per_obj":
+            buff.append(f"\trv_attach_trace_probe(\"{self.name}\", /* XXX: cleanup tracepoint */, handle_{self.cleanup_marker});")
         return '\n'.join(buff)
 
     def fill_tracepoint_detach_helper(self) -> str:
         buff = []
         for event in self.events:
             buff.append(f"\trv_detach_trace_probe(\"{self.name}\", /* XXX: tracepoint */, handle_{event});")
+        if self.monitor_type == "per_obj":
+            buff.append(f"\trv_detach_trace_probe(\"{self.name}\", /* XXX: cleanup tracepoint */, handle_{self.cleanup_marker});")
         return '\n'.join(buff)
 
     def fill_model_h_header(self) -> list[str]:
Signature · 1 line
2.54.0
author maintainer
#21 lore raw

PATCH v2 12/12 verification/rvgen: Fix suffix strip in dot2k

In reply to Gabriele Monaco

__start_to_invariant_check() and __get_constraint_env() parse the environment variable's name from sources that have it padded with the monitor name. This is removed using rstrip(), which is not meant to strip a substring but rather a set of characters.

Use removesuffix() to actually get rid of the trailing _<monitor name>.

Fixes a82adadb16894 ("verification/rvgen: Add support for Hybrid Automata")
---
 tools/verification/rvgen/rvgen/dot2k.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tools/verification/rvgen/rvgen/dot2k.py +2 -2
index a17294283a1f..3060aa4b945d 100644
@@ -230,14 +230,14 @@ class ha2k(dot2k):
     def __get_constraint_env(self, constr: str) -> str:
         """Extract the second argument from an ha_ function"""
         env = constr.split("(")[1].split()[1].rstrip(")").rstrip(",")
-        assert env.rstrip(f"_{self.name}") in self.envs
+        assert env.removesuffix(f"_{self.name}") in self.envs
         return env
 
     def __start_to_invariant_check(self, constr: str) -> str:
         # by default assume the timer has ns expiration
         env = self.__get_constraint_env(constr)
         clock_type = "ns"
-        if self.env_types.get(env.rstrip(f"_{self.name}")) == "j":
+        if self.env_types.get(env.removesuffix(f"_{self.name}")) == "j":
             clock_type = "jiffy"
 
         return f"return ha_check_invariant_{clock_type}(ha_mon, {env}, time_ns)"
Signature · 1 line
2.54.0