Skip to content

Add monitors to validate task switch

Gabriele Monaco linux-doc linux-trace-kernel lkml 5 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v5 Latest9 patches 14 repliesIn review
  2. v414 patches 11 repliesSuperseded
  3. v317 patches 39 repliesSuperseded
Earlier versions · 2
VersionPostedPatchesRepliesStatus
v2 RFC12 patches25 repliesSuperseded
v1 RFC9 patches10 repliesSuperseded

Changes in v5 from the cover letter

* Drop already applied patches for the tools/ directory
* Avoid using smp_processor_id() from tracepoint context

Patches in v5 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/9Add da_handle_start_run_event_ to per-task monitorsinclude/rv/da_monitor.h+24 -00 replies
2/9Remove trailing whitespace from tracepoint stringkernel/trace/rv/rv_trace.h+4 -40 replies
3/9Use strings in da monitors tracepointskernel/trace/rv/rv_trace.h+38 -380 replies
4/9Adjust monitor dependencieskernel/trace/rv/monitors/scpd/Kconfig kernel/trace/rv/monitors/sncid/Kconfig kernel/trace/rv/monitors/snep/Kconfig +1 more+4 -40 replies
5/9Retry when da monitor detects race conditionsinclude/linux/rv.h include/rv/da_monitor.h kernel/trace/rv/Kconfig +1 more+85 -541 reply
6/9sched: Adapt sched tracepoints for RV task modelinclude/linux/sched.h include/trace/events/sched.h kernel/sched/core.c +5 more+34 -181 reply
7/9Replace tss and sncid monitors with more complete stsDocumentation/trace/rv/monitor_sched.rst kernel/trace/rv/Kconfig kernel/trace/rv/Makefile +15 more+385 -41010 replies
8/9Add nrp and sssw per-task monitorsDocumentation/trace/rv/monitor_sched.rst kernel/trace/rv/Kconfig kernel/trace/rv/Makefile +12 more+728 -01 reply
9/9Add opid per-cpu monitorDocumentation/trace/rv/monitor_sched.rst kernel/trace/rv/Kconfig kernel/trace/rv/Makefile +6 more+399 -01 reply

Reviews trailers per patch

7 of 9 patches have a Reviewed-by or Acked-by.

PatchNam Cao
1/9R-b
2/9R-b
3/9R-b
4/9A-b
5/9R-b
6/9
7/9
8/9A-b T-b
9/9A-b T-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v5 thread

Cover letter [PATCH v5 0/9] rv: Add monitors to validate task switch

This series adds three monitors to the sched collection, extends and replaces previously existing monitors:

{tss,snroc} => sts:
Not only prove that switches occur in scheduling context and scheduling needs interrupt disabled but also that each call to the scheduler disables interrupts to (optionally) switch.

nrp (NEW):
* preemption requires need resched which is cleared by any switch (includes a non optimal workaround for /nested/ preemptions)

sssw (NEW):
* suspension requires setting the task to sleepable and, after the switch occurs, the task requires a wakeup to come back to runnable

opid (NEW):
* waking and need-resched operations occur with interrupts and preemption disabled or in IRQ without explicitly disabling preemption

Also include some minor cleanup patches (1-4) tracepoints (6) and preparatory fixes (5) covering some corner cases:

The series is currently based on the tracing rv/for-next tree.

Patch 1 adds da_handle_start_run_event_ also to per-task monitors

Patch 2 removes a trailing whitespace from the rv tracepoint string

Patch 3 fixes an out-of-bound memory access in DA tracepoints

Patch 4 adjusts monitors to have minimised Kconfig dependencies

Patch 5 detects race conditions when rv monitors run concurrently and retries applying the events

Patch 6 adds the need_resched and removes unused arguments from schedule entry/exit tracepoints

Patch 7 adds the sts monitor to replace tss and sncid

Patch 8 adds the nrp and sssw monitors

Patch 9 adds the opid monitor

NOTES

The nrp and sssw monitors include workarounds for racy conditions:

* A sleeping task requires to set the state to sleepable, but in case of a task sleeping on an rtlock, the set sleepable and wakeup events race and we don't always see the right order:

5d..2. 107.488369: event: 639: sleepable x set_sleepable -> sleepable 4d..5. 107.488369: event: 639: sleepable x wakeup -> running (final) 5d..3. 107.488385: error: 639: switch_suspend not expected in the state running

wakeup() set_state()
state=RUNNING
trace_set_state()
trace_wakeup()
state=SLEEPING

I added a special event (switch_block) but there may be a better way. Taking a pi_lock in rtlock_slowlock_locked when setting the state to TASK_RTLOCK_WAIT avoids this race, although this is likely not something we want to do.

* I consider preemption any scheduling with preempt==true and assume this can happen only if need resched is set.
In practice, however, we may see a preemption where the flag is not set. This can happen in one specific condition:

need_resched

                  preempt_schedule()
                                        preempt_schedule_irq()
                                            __schedule()
  !need_resched
                      __schedule()

We start a standard preemption (e.g. from preempt_enable when the flag is set), an interrupts occurs before we schedule and, on its exit path, it schedules, which clears the need_resched flag.
When the preempted task runs again, we continue the standard preemption started earlier, although the flag is no longer set.

I added a workaround to allow the model not to fail in this condition, by allowing a preemption without need_resched if an interrupt is received. This might catch false negatives too.

Changes since V4:
* Drop already applied patches for the tools/ directory
* Avoid using smp_processor_id() from tracepoint context

Changes since V3 [1]:
* Fix condition for lines shorter than 100 columns in dot2c.
* Fix Kconfig tooltip for container monitors in rvgen.
* Improve condition not to skip pid-0 in userspace tool.
* Rearrange monitors to reduce needed tracepoints and arguments.
* Separately track errors when DAs run out of retries due to races.
* Fix issue in opid with multiple handlers in the same interrupt.
* Cleanup patches by removing, squashing and reordering.

Changes since RFC2:
* Arrange commits to prevent failed build while bisecting.
* Avoid dot2k generated files to reach the column limit. (Nam Cao)
* Rearrange and simplify da_monitor retry on racing events.
* Improve nrp monitor to handle /nested/ preemption on IRQ.
* Added minor patches (6-10).
* Cleanup and rearrange order.
Changes since RFC [2]:
* Remove wakeup tracepoint in try_to_block_task and use a different flavour of sched_set_state
* Split the large srs monitor in two separate monitors for preemption and sleep. These no longer have a concept of running task, they just enforce the requirements for the different types of sched out.
* Restore the snroc monitor to describe the relationship between generic sched out and sched in.
* Add opid monitor.
* Fix some build errors and cleanup.

[1] - https://lore.kernel.org/lkml/[email protected]
[2] - https://lore.kernel.org/lkml/[email protected]

To: Ingo Molnar <[email protected]>
To: Peter Zijlstra <[email protected]>
To: Steven Rostedt <[email protected]>
To: Nam Cao <[email protected]>
Cc: Tomas Glozar <[email protected]>
Cc: Juri Lelli <[email protected]>
Cc: Clark Williams <[email protected]>
Cc: John Kacur <[email protected]>

Gabriele Monaco (9):
rv: Add da_handle_start_run_event_ to per-task monitors
rv: Remove trailing whitespace from tracepoint string
rv: Use strings in da monitors tracepoints
rv: Adjust monitor dependencies
rv: Retry when da monitor detects race conditions
sched: Adapt sched tracepoints for RV task model
rv: Replace tss and sncid monitors with more complete sts
rv: Add nrp and sssw per-task monitors
rv: Add opid per-cpu monitor

 Documentation/trace/rv/monitor_sched.rst      | 307 +++++++++++++++---
 include/linux/rv.h                            |   3 +-
 include/linux/sched.h                         |   7 +-
 include/rv/da_monitor.h                       | 131 +++++---
 include/trace/events/sched.h                  |  12 +-
 kernel/sched/core.c                           |  13 +-
 kernel/trace/rv/Kconfig                       |  11 +-
 kernel/trace/rv/Makefile                      |   6 +-
 kernel/trace/rv/monitors/{tss => nrp}/Kconfig |  12 +-
 kernel/trace/rv/monitors/nrp/nrp.c            | 138 ++++++++
 kernel/trace/rv/monitors/nrp/nrp.h            |  75 +++++
 kernel/trace/rv/monitors/nrp/nrp_trace.h      |  15 +
 kernel/trace/rv/monitors/opid/Kconfig         |  19 ++
 kernel/trace/rv/monitors/opid/opid.c          | 168 ++++++++++
 kernel/trace/rv/monitors/opid/opid.h          | 104 ++++++
 .../sncid_trace.h => opid/opid_trace.h}       |   8 +-
 kernel/trace/rv/monitors/sched/Kconfig        |   1 +
 kernel/trace/rv/monitors/sco/sco.c            |   4 +-
 kernel/trace/rv/monitors/scpd/Kconfig         |   2 +-
 kernel/trace/rv/monitors/scpd/scpd.c          |   4 +-
 kernel/trace/rv/monitors/sncid/sncid.c        |  95 ------
 kernel/trace/rv/monitors/sncid/sncid.h        |  49 ---
 kernel/trace/rv/monitors/snep/Kconfig         |   2 +-
 kernel/trace/rv/monitors/snep/snep.c          |   4 +-
 .../trace/rv/monitors/{sncid => sssw}/Kconfig |  10 +-
 kernel/trace/rv/monitors/sssw/sssw.c          | 116 +++++++
 kernel/trace/rv/monitors/sssw/sssw.h          | 105 ++++++
 kernel/trace/rv/monitors/sssw/sssw_trace.h    |  15 +
 kernel/trace/rv/monitors/sts/Kconfig          |  19 ++
 kernel/trace/rv/monitors/sts/sts.c            | 156 +++++++++
 kernel/trace/rv/monitors/sts/sts.h            | 117 +++++++
 .../{tss/tss_trace.h => sts/sts_trace.h}      |   8 +-
 kernel/trace/rv/monitors/tss/tss.c            |  90 -----
 kernel/trace/rv/monitors/tss/tss.h            |  47 ---
 kernel/trace/rv/monitors/wip/Kconfig          |   2 +-
 kernel/trace/rv/rv_trace.h                    | 114 ++++---
 tools/verification/models/sched/nrp.dot       |  29 ++
 tools/verification/models/sched/opid.dot      |  35 ++
 tools/verification/models/sched/sncid.dot     |  18 -
 tools/verification/models/sched/sssw.dot      |  30 ++
 tools/verification/models/sched/sts.dot       |  38 +++
 tools/verification/models/sched/tss.dot       |  18 -
 42 files changed, 1665 insertions(+), 492 deletions(-)
 rename kernel/trace/rv/monitors/{tss => nrp}/Kconfig (51%)
 create mode 100644 kernel/trace/rv/monitors/nrp/nrp.c
 create mode 100644 kernel/trace/rv/monitors/nrp/nrp.h
 create mode 100644 kernel/trace/rv/monitors/nrp/nrp_trace.h
 create mode 100644 kernel/trace/rv/monitors/opid/Kconfig
 create mode 100644 kernel/trace/rv/monitors/opid/opid.c
 create mode 100644 kernel/trace/rv/monitors/opid/opid.h
 rename kernel/trace/rv/monitors/{sncid/sncid_trace.h => opid/opid_trace.h} (66%)
 delete mode 100644 kernel/trace/rv/monitors/sncid/sncid.c
 delete mode 100644 kernel/trace/rv/monitors/sncid/sncid.h
 rename kernel/trace/rv/monitors/{sncid => sssw}/Kconfig (58%)
 create mode 100644 kernel/trace/rv/monitors/sssw/sssw.c
 create mode 100644 kernel/trace/rv/monitors/sssw/sssw.h
 create mode 100644 kernel/trace/rv/monitors/sssw/sssw_trace.h
 create mode 100644 kernel/trace/rv/monitors/sts/Kconfig
 create mode 100644 kernel/trace/rv/monitors/sts/sts.c
 create mode 100644 kernel/trace/rv/monitors/sts/sts.h
 rename kernel/trace/rv/monitors/{tss/tss_trace.h => sts/sts_trace.h} (67%)
 delete mode 100644 kernel/trace/rv/monitors/tss/tss.c
 delete mode 100644 kernel/trace/rv/monitors/tss/tss.h
 create mode 100644 tools/verification/models/sched/nrp.dot
 create mode 100644 tools/verification/models/sched/opid.dot
 delete mode 100644 tools/verification/models/sched/sncid.dot
 create mode 100644 tools/verification/models/sched/sssw.dot
 create mode 100644 tools/verification/models/sched/sts.dot
 delete mode 100644 tools/verification/models/sched/tss.dot

base-commit: b8a7fba39cd49eab343bfe561d85bb5dc57541af

Signature · 1 line
2.50.1