Skip to content

Add task latency over budget RV monitor

[email protected] linux-trace-kernel lkml 8 versions – New

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v8 Latest14 patches 0 repliesNew
  2. v714 patches 11 repliesSuperseded
  3. v69 patches 10 repliesSuperseded
Earlier versions · 5
VersionPostedPatchesRepliesStatus
v59 patches0 repliesSuperseded
v48 patches13 repliesSuperseded
v39 patches15 repliesSuperseded
v2 RFC10 patches20 repliesSuperseded
v1 RFC4 patches6 repliesSuperseded

Changes in v8 from the cover letter · 24 lines

=============
This version is a comprehensive rework addressing Gabriele Monaco's review on v7, with substantial simplifications to both framework support and the tlob monitor itself:

1. HA timer teardown and underflow fixes (Patches 1-5):
- Deleted ha_timer_ready() completely. Following Gabriele's suggestion, ha_monitor_free_env() now checks da_monitoring() to disarm the hrtimer on the storage free path (Patch 4).
- Isolated the HA underflow and timer free path KUnit tests into a dedicated compilation unit (kernel/trace/rv/rv_ha_kunit.c) guarded by CONFIG_RV_HA_MONITOR, avoiding polluting rv_monitors_test.c (Patches 3, 5).

2. Framework per-object storage improvements (Patches 6-8):
- Added DA_OBJ_EMBEDDED_TYPE support based on Gabriele's PoC. Per-object DA storage can now directly embed monitor-specific target data inside struct da_monitor_storage, completely eliminating secondary dynamic memory allocations and double pointer dereferences (Patch 8).
- Added smp_store_release/smp_load_acquire memory barriers between da_storage_set_target() and da_storage_is_empty(), and ensured da_get_target() returns NULL if the storage slot is not yet filled.

3. Generic uprobe helper infrastructure (Patch 9):
- Extracted reusable rv_uprobe registration and synchronization helpers into include/rv/rv_uprobe.h and kernel/trace/rv/rv_uprobe.c.

4. Major tlob simplification (Patch 11):
- Reduced tlob.c from ~1065 lines to ~660 lines (-38% code size).
- Completely removed the private mempool (tlob_record_pool) and custom allocator; tlob now relies entirely on the framework's pre-allocated storage via DA_MON_POOL_SIZE.
- Removed the parallel state machine (OPEN, CLOSED, RETIRED) and tlob_binding_lock. Window lifecycle is now 100% managed by the automaton states and framework storage lifecycle.
- Replaced complex quiesce/reap/RCU callouts with a clean RCU walk (tlob_release_binding()) executed after rv_uprobe_unregister_sync().
- Optimized timer re-arming on the scheduler hot path.
- Refined error semantics: nested START is detected as an illegal event (running x start) and reported via error_tlob without altering the running window; pool exhaustion cleanly returns -ENOSPC.

5. Test coverage and selftests expansion (Patches 12-14):
- Added 3 new selftests covering critical edge cases:
* uprobe_task_exit.tc: verifies task exit/SIGKILL inside an active window disarms the timer with no orphaned/late violations.
* uprobe_nested_start.tc: verifies nested START reporting and window integrity.
* uprobe_pool_exhausted.tc: verifies the 64-task cap and graceful rejection of the 65th task.
- Extended tlob_target with "nest" mode.
- All 13 selftests and all KUnit test suites pass cleanly.

Patches in v8 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/14Force environment reset action on HA monitor startinclude/rv/ha_monitor.h kernel/trace/rv/monitors/opid/opid.h tools/verification/rvgen/rvgen/dot2c.py+10 -10 replies
2/14Fix ha_monitor timer and invariant underflowinclude/rv/ha_monitor.h+24 -50 replies
3/14Add a KUnit test for the HA timer underflowkernel/trace/rv/Makefile kernel/trace/rv/rv_ha_kunit.c+153 -00 replies
4/14Disarm HA timers on the storage free pathinclude/rv/da_monitor.h include/rv/ha_monitor.h+34 -10 replies
5/14Add a KUnit test for the storage free pathkernel/trace/rv/rv_ha_kunit.c+50 -10 replies
6/14Make da_monitor_reset_hook, sync_hook and EVENT_NONE_LBL overridableinclude/rv/ha_monitor.h+8 -00 replies
7/14Introduce DA_MON_ALLOCATION_STRATEGYinclude/rv/da_monitor.h kernel/trace/rv/monitors/nomiss/nomiss.c+112 -110 replies
8/14Support embedded target types in per-object DA storageinclude/rv/da_monitor.h+76 -60 replies
9/14Add generic uprobe infrastructure for RV monitorsinclude/rv/rv_uprobe.h kernel/trace/rv/Kconfig kernel/trace/rv/Makefile +1 more+96 -00 replies
10/14Add tlob model DOT filetools/verification/models/tlob.dot+27 -00 replies
11/14Add tlob hybrid automaton monitorDocumentation/trace/rv/index.rst Documentation/trace/rv/monitor_tlob.rst kernel/trace/rv/Kconfig +6 more+1073 -00 replies
12/14Add KUnit tests for the tlob monitorkernel/trace/rv/monitors/tlob/tlob.c kernel/trace/rv/monitors/tlob/tlob_kunit.c kernel/trace/rv/monitors/tlob/tlob_kunit.h +1 more+182 -00 replies
13/14selftests/verification: Add tlob selfteststools/testing/selftests/verification/.gitignore tools/testing/selftests/verification/Makefile tools/testing/selftests/verification/config +16 more+971 -00 replies
14/14selftests/ftrace: Walk up parent dirs to find test.d/functionstools/testing/selftests/ftrace/ftracetest+16 -30 replies

Reviews trailers per patch

4 of 14 patches have a Reviewed-by or Acked-by.

PatchWen YangGabriele Monaco
1/14T-b
2/14R-b
3/14
4/14
5/14
6/14R-b
7/14
8/14
9/14
10/14
11/14
12/14R-b
13/14R-b
14/14

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v8 thread

Cover letter [PATCH v8 00/14] rv: Add task latency over budget RV monitor

From: Wen Yang <[email protected]>

This series introduces tlob (task latency over budget), a per-task hybrid automaton RV monitor that measures elapsed wall-clock time across a user-delimited code section and emits an error when the elapsed time exceeds a configurable budget.

The monitor tracks four states (running, waiting, sleeping, stopped) driven by sched_switch and sched_wakeup tracepoints. A single clock invariant, clk_elapsed < BUDGET_NS(), is enforced by a per-task hrtimer (HRTIMER_MODE_REL_HARD). On expiry error_env_tlob is emitted; which phase consumed the budget (running, waiting or sleeping) is recovered from event_tlob in user space (see monitor_tlob.rst).

Tasks are registered for monitoring by writing to the tracefs monitor file:
echo "p:myregion /path/to/binary:OFFSET_START OFFSET_STOP threshold=NS" \
>> /sys/kernel/tracing/rv/monitors/tlob/monitor

Two uprobes delimit the measured section without modifying the target binary. Multiple uprobe pairs can be active simultaneously. A task may repeat the START/STOP sequence any number of times: the entry parks in the automaton's "stopped" state and restarts in place, reusing the same pre-allocated slot.

Changes in v8
=============
This version is a comprehensive rework addressing Gabriele Monaco's review on v7, with substantial simplifications to both framework support and the tlob monitor itself:

1. HA timer teardown and underflow fixes (Patches 1-5):
- Deleted ha_timer_ready() completely. Following Gabriele's suggestion, ha_monitor_free_env() now checks da_monitoring() to disarm the hrtimer on the storage free path (Patch 4).
- Isolated the HA underflow and timer free path KUnit tests into a dedicated compilation unit (kernel/trace/rv/rv_ha_kunit.c) guarded by CONFIG_RV_HA_MONITOR, avoiding polluting rv_monitors_test.c (Patches 3, 5).

2. Framework per-object storage improvements (Patches 6-8):
- Added DA_OBJ_EMBEDDED_TYPE support based on Gabriele's PoC. Per-object DA storage can now directly embed monitor-specific target data inside struct da_monitor_storage, completely eliminating secondary dynamic memory allocations and double pointer dereferences (Patch 8).
- Added smp_store_release/smp_load_acquire memory barriers between da_storage_set_target() and da_storage_is_empty(), and ensured da_get_target() returns NULL if the storage slot is not yet filled.

3. Generic uprobe helper infrastructure (Patch 9):
- Extracted reusable rv_uprobe registration and synchronization helpers into include/rv/rv_uprobe.h and kernel/trace/rv/rv_uprobe.c.

4. Major tlob simplification (Patch 11):
- Reduced tlob.c from ~1065 lines to ~660 lines (-38% code size).
- Completely removed the private mempool (tlob_record_pool) and custom allocator; tlob now relies entirely on the framework's pre-allocated storage via DA_MON_POOL_SIZE.
- Removed the parallel state machine (OPEN, CLOSED, RETIRED) and tlob_binding_lock. Window lifecycle is now 100% managed by the automaton states and framework storage lifecycle.
- Replaced complex quiesce/reap/RCU callouts with a clean RCU walk (tlob_release_binding()) executed after rv_uprobe_unregister_sync().
- Optimized timer re-arming on the scheduler hot path.
- Refined error semantics: nested START is detected as an illegal event (running x start) and reported via error_tlob without altering the running window; pool exhaustion cleanly returns -ENOSPC.

5. Test coverage and selftests expansion (Patches 12-14):
- Added 3 new selftests covering critical edge cases:
* uprobe_task_exit.tc: verifies task exit/SIGKILL inside an active window disarms the timer with no orphaned/late violations.
* uprobe_nested_start.tc: verifies nested START reporting and window integrity.
* uprobe_pool_exhausted.tc: verifies the 64-task cap and graceful rejection of the 65th task.
- Extended tlob_target with "nest" mode.
- All 13 selftests and all KUnit test suites pass cleanly.

Series structure
================

Part 1: Independent fixes and enhancements to the RV/HA framework
Patch 1: rv: Force environment reset action on HA monitor start Patch 2: rv: Fix ha_monitor timer and invariant underflow Patch 3: rv: Add a KUnit test for the HA timer underflow Patch 4: rv: Disarm HA timers on the storage free path Patch 5: rv: Add a KUnit test for the storage free path Patch 6: rv: Make da_monitor_reset_hook, sync_hook and EVENT_NONE_LBL
overridable
Patch 7: rv: Introduce DA_MON_ALLOCATION_STRATEGY
Patch 8: rv: Support embedded target types in per-object DA storage Patch 9: rv: Add generic uprobe infrastructure for RV monitors

Part 2: The tlob monitor and models
Patch 10: rv: Add tlob model DOT file
Patch 11: rv: Add tlob hybrid automaton monitor

Part 3: Testing and selftests
Patch 12: rv: Add KUnit tests for the tlob monitor
Patch 13: selftests/verification: Add tlob selftests
Patch 14: selftests/ftrace: Walk up parent dirs to find test.d/functions

v7: https://lore.kernel.org/all/[email protected]/
v6: https://lore.kernel.org/all/[email protected]/
v4: https://lore.kernel.org/all/[email protected]/
v3: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v1: https://lore.kernel.org/all/[email protected]/

Gabriele Monaco (1):
rv: Force environment reset action on HA monitor start

Wen Yang (13):
rv: Fix ha_monitor timer and invariant underflow
rv: Add a KUnit test for the HA timer underflow
rv: Disarm HA timers on the storage free path
rv: Add a KUnit test for the storage free path
rv: Make da_monitor_reset_hook, sync_hook and EVENT_NONE_LBL
overridable
rv: Introduce DA_MON_ALLOCATION_STRATEGY
rv: Support embedded target types in per-object DA storage
rv: Add generic uprobe infrastructure for RV monitors
rv: Add tlob model DOT file
rv: Add tlob hybrid automaton monitor
rv: Add KUnit tests for the tlob monitor selftests/verification: Add tlob selftests selftests/ftrace: Walk up parent dirs to find test.d/functions

 Documentation/trace/rv/index.rst              |   1 +
 Documentation/trace/rv/monitor_tlob.rst       | 162 ++++
 include/rv/da_monitor.h                       | 226 +++++-
 include/rv/ha_monitor.h                       |  53 +-
 include/rv/rv_uprobe.h                        |  35 +
 kernel/trace/rv/Kconfig                       |   5 +
 kernel/trace/rv/Makefile                      |   3 +
 kernel/trace/rv/monitors/nomiss/nomiss.c      |   2 +-
 kernel/trace/rv/monitors/opid/opid.h          |   1 +
 kernel/trace/rv/monitors/tlob/Kconfig         |  12 +
 kernel/trace/rv/monitors/tlob/tlob.c          | 743 ++++++++++++++++++
 kernel/trace/rv/monitors/tlob/tlob.h          | 143 ++++
 kernel/trace/rv/monitors/tlob/tlob_kunit.c    | 151 ++++
 kernel/trace/rv/monitors/tlob/tlob_kunit.h    |  18 +
 kernel/trace/rv/monitors/tlob/tlob_trace.h    |  20 +
 kernel/trace/rv/rv_ha_kunit.c                 | 201 +++++
 kernel/trace/rv/rv_monitors_test.c            |   2 +
 kernel/trace/rv/rv_trace.h                    |   1 +
 kernel/trace/rv/rv_uprobe.c                   |  56 ++
 tools/testing/selftests/ftrace/ftracetest     |  19 +-
 .../testing/selftests/verification/.gitignore |   1 +
 tools/testing/selftests/verification/Makefile |   4 +
 tools/testing/selftests/verification/config   |   3 +
 .../selftests/verification/test.d/functions   |   9 +
 .../test.d/tlob/run_tlob_tests.sh             |  32 +
 .../verification/test.d/tlob/uprobe_bind.tc   |  47 ++
 .../test.d/tlob/uprobe_disable_live.tc        |  79 ++
 .../verification/test.d/tlob/uprobe_multi.tc  |  58 ++
 .../test.d/tlob/uprobe_multi_instance.tc      |  65 ++
 .../test.d/tlob/uprobe_nested_start.tc        |  62 ++
 .../test.d/tlob/uprobe_no_event.tc            |  17 +
 .../test.d/tlob/uprobe_overlap_register.tc    |  66 ++
 .../test.d/tlob/uprobe_pool_exhausted.tc      |  75 ++
 .../test.d/tlob/uprobe_restart.tc             |  66 ++
 .../test.d/tlob/uprobe_restart_violation.tc   |  48 ++
 .../test.d/tlob/uprobe_task_exit.tc           |  65 ++
 .../test.d/tlob/uprobe_unbind_live.tc         |  65 ++
 .../test.d/tlob/uprobe_violation.tc           |  53 ++
 .../selftests/verification/tlob_target.c      | 156 ++++
 tools/verification/models/tlob.dot            |  27 +
 tools/verification/rvgen/rvgen/dot2c.py       |   2 +
 41 files changed, 2829 insertions(+), 25 deletions(-)
 create mode 100644 Documentation/trace/rv/monitor_tlob.rst
 create mode 100644 include/rv/rv_uprobe.h
 create mode 100644 kernel/trace/rv/monitors/tlob/Kconfig
 create mode 100644 kernel/trace/rv/monitors/tlob/tlob.c
 create mode 100644 kernel/trace/rv/monitors/tlob/tlob.h
 create mode 100644 kernel/trace/rv/monitors/tlob/tlob_kunit.c
 create mode 100644 kernel/trace/rv/monitors/tlob/tlob_kunit.h
 create mode 100644 kernel/trace/rv/monitors/tlob/tlob_trace.h
 create mode 100644 kernel/trace/rv/rv_ha_kunit.c
 create mode 100644 kernel/trace/rv/rv_uprobe.c
 create mode 100755 tools/testing/selftests/verification/test.d/tlob/run_tlob_tests.sh
 create mode 100644 tools/testing/selftests/verification/test.d/tlob/uprobe_bind.tc
 create mode 100644 tools/testing/selftests/verification/test.d/tlob/uprobe_disable_live.tc
 create mode 100644 tools/testing/selftests/verification/test.d/tlob/uprobe_multi.tc
 create mode 100644 tools/testing/selftests/verification/test.d/tlob/uprobe_multi_instance.tc
 create mode 100644 tools/testing/selftests/verification/test.d/tlob/uprobe_nested_start.tc
 create mode 100644 tools/testing/selftests/verification/test.d/tlob/uprobe_no_event.tc
 create mode 100644 tools/testing/selftests/verification/test.d/tlob/uprobe_overlap_register.tc
 create mode 100644 tools/testing/selftests/verification/test.d/tlob/uprobe_pool_exhausted.tc
 create mode 100644 tools/testing/selftests/verification/test.d/tlob/uprobe_restart.tc
 create mode 100644 tools/testing/selftests/verification/test.d/tlob/uprobe_restart_violation.tc
 create mode 100644 tools/testing/selftests/verification/test.d/tlob/uprobe_task_exit.tc
 create mode 100644 tools/testing/selftests/verification/test.d/tlob/uprobe_unbind_live.tc
 create mode 100644 tools/testing/selftests/verification/test.d/tlob/uprobe_violation.tc
 create mode 100644 tools/testing/selftests/verification/tlob_target.c
 create mode 100644 tools/verification/models/tlob.dot
Signature · 1 line
2.25.1