rt-tests: README: Remove obsolete -n option from cyclictest examples
PATCH 1/3 rt-tests: README: Remove obsolete -n option from cyclictest examples
The -n/--nanosleep option was removed from cyclictest in commit 9d2c9edd8694 ("cyclictest: make clock_nanosleep as default and add option for POSIX timers") in March 2017.
Since then, clock_nanosleep has been the default behavior, making the -n option redundant. The examples in README.markdown still referenced this obsolete option, causing the commands to fail with "invalid option".
Update all cyclictest example commands to remove the -n flag.
Also add SPDX-License-Identifier to match project conventions.
--- README.markdown | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
README.markdown +4 -2
@@ -1,3 +1,5 @@ +SPDX-License-Identifier: GPL-2.0-or-later + # RT-Tests This repository contains some programs that test various rt-linux features.
@@ -14,7 +16,7 @@ This repository contains some programs that test various rt-linux features. To run one test thread per CPU or per CPU core, each thread on a separate processor, type - sudo ./cyclictest -a -t -n -p99 + sudo ./cyclictest -a -t -p99 On a non-realtime system, you may see something like
@@ -62,7 +64,7 @@ tracer must be configured upfront. The resulting trace can be invastigated in `/sys/kernel/tracing/trace`, for example: echo function > /sys/kernel/tracing/current_tracer - cyclictest -a -t -n -p99 --tracemark --breaktrace 100 + cyclictest -a -t -p99 --tracemark --breaktrace 100 This causes the program to abort execution, if the latency value exceeds 100 microseconds; the culprit can then be found in the trace output at
Signature · 1 line
2.54.0
PATCH 2/3 rt-tests: Add SPDX identifiers and fix documentation issues
Add SPDX-License-Identifier: GPL-2.0-or-later to the following files that were missing license identifiers:
- get_pylib.py
- Dockerfile
- MAINTAINERS
- README-Dockerfile
- src/pi_tests/README
- src/queuelat/README
- src/queuelat/targeted-ipi/README
Additionally, fix documentation issues:
In README-Dockerfile:
- Fix spelling: "imbiguous" → "unambiguous"
- Correct project references: "rteval" → "rt-tests" in directory name, container image name, and section heading to accurately reflect that this is rt-tests documentation
In src/pi_tests/README:
- Fix spelling: "inheritence" → "inheritance"
- Fix typo: "one one processor" → "on one processor"
- Fix spacing: "fetchthe" → "fetch the"
- Fix option: "--duraton" → "--duration"
--- Dockerfile | 1 + MAINTAINERS | 2 ++ README-Dockerfile | 11 ++++++----- get_pylib.py | 1 + src/pi_tests/README | 10 ++++++---- src/queuelat/README | 3 ++- src/queuelat/targeted-ipi/README | 1 + 7 files changed, 19 insertions(+), 10 deletions(-)
Dockerfile +1 -0
@@ -1,3 +1,4 @@ +# SPDX-License-Identifier: GPL-2.0-or-later # Use CentOS Stream 9 as base image FROM centos:stream9
MAINTAINERS +2 -0
@@ -1,3 +1,5 @@ +SPDX-License-Identifier: GPL-2.0-or-later + Please send your patches to Clark Williams <[email protected]>
README-Dockerfile +6 -5
@@ -1,8 +1,9 @@ -1 +SPDX-License-Identifier: GPL-2.0-or-later + # README-Dockerfile-rt-tests ## Introduction -This README provides detailed instructions for setting up an rt-tests container using Docker or Podman. This Dockerfile will pull a current copy of your code (plus all of your changes) and compile it into a container image for you to test with. This will give you an imbiguous way to test your changes, assuming you are running on a semi-modern host OS. The rt-tests container can be used to test any of the subpackages found under the rt-tests project, but is mainly intended for hwlatdetect and cyclictest. +This README provides detailed instructions for setting up an rt-tests container using Docker or Podman. This Dockerfile will pull a current copy of your code (plus all of your changes) and compile it into a container image for you to test with. This will give you an unambiguous way to test your changes, assuming you are running on a semi-modern host OS. The rt-tests container can be used to test any of the subpackages found under the rt-tests project, but is mainly intended for hwlatdetect and cyclictest. ## Prerequisites - Docker or Podman installed on your system.
@@ -16,8 +17,8 @@ This README provides detailed instructions for setting up an rt-tests container - Alternatively, install Podman: `sudo apt-get update && sudo apt-get install podman -y` ### Build the Container -- Navigate to the root directory of the 'rteval' -- Build the container image named 'rteval-upstream': +- Navigate to the root directory of 'rt-tests' +- Build the container image named 'rt-tests-upstream': - For Podman: `sudo podman build -t rt-tests-upstream .` - For Docker: `sudo docker build -t rt-tests-upstream .`
@@ -40,7 +41,7 @@ This README provides detailed instructions for setting up an rt-tests container - For Podman: `sudo podman run -it --rm --privileged rt-tests-upstream` - For Docker: `sudo docker run -it --rm --privileged rt-tests-upstream` -### Test rteval Build +### Test rt-tests Build - Inside the container, test one of the rt-tests packages you just built: - Command: `cyclictest -D 1s` - Command: `hwlatdetect --duration 1`
get_pylib.py +1 -0
@@ -1,4 +1,5 @@ #!/usr/bin/python3 +# SPDX-License-Identifier: GPL-2.0-or-later # This file is used during the make process
src/pi_tests/README +6 -4
@@ -1,3 +1,5 @@ +SPDX-License-Identifier: GPL-2.0-or-later + DESCRIPTION The pi_stress program is a stress test that is intended to exercise
@@ -22,20 +24,20 @@ The threads run through a state machine designed to guarantee that a low-priority thread holds a mutex while a medium priority thread runs (keeping the low-priority thread from releasing the mutex). The high-priority thread attempts to acquire the mutex and is blocked -because of the low-priority thread holding it. If priority inheritence +because of the low-priority thread holding it. If priority inheritance is working, the low-priority thread will receive a priority boost (will inherit the high-priority thread's priority) and will then run and release the mutex, averting a deadlock. On a multi-processor system, the admin and reporter threads are run -one one processor while the inversion groups are run on another +on one processor while the inversion groups are run on another processor. STEPS TO RUN Install the pi_tests rpm. If you are running Fedora Core, the simplest -way do to that is to fetchthe rt-userspace.repo file from: +way do to that is to fetch the rt-userspace.repo file from: http://people.redhat.com/~williams/rt-userspace.repo
@@ -74,7 +76,7 @@ that have been performed. To stop the test, press any key and a summary will be printed to show how many inversions were performed and how long the test ran. -To run the test for a specifed time, use the --duraton=<seconds> +To run the test for a specifed time, use the --duration=<seconds> option.
src/queuelat/README +2 -1
@@ -1,5 +1,6 @@ +SPDX-License-Identifier: GPL-2.0-or-later -cyclictest does not catch all cases where packet forwarding +cyclictest does not catch all cases where packet forwarding latency can exceed a given threshold. Example:
src/queuelat/targeted-ipi/README +1 -0
@@ -1,3 +1,4 @@ +SPDX-License-Identifier: GPL-2.0-or-later To build:
Signature · 1 line
2.54.0
PATCH 3/3 rt-tests: Add AGENTS.md guide for AI coding assistants
In reply to John KacurThis guide is also useful for human developers new to the project.
Add comprehensive documentation for AI coding assistants working on the rt-tests codebase. This guide covers:
- Project overview and architecture
- Core programs (cyclictest, hwlatdetect, pi_stress, etc.)
- Shared library components and build system
- Development guidelines including code style and checkpatch usage
- Testing procedures and debugging tools
- Common gotchas for real-time testing
- Related projects in the RT ecosystem (rteval, tuna, rtla, rv)
- Kernel documentation references
The guide is designed to help AI assistants quickly understand the codebase structure, key patterns, and best practices when making changes to rt-tests.
Python requirement set to 3.8+ based on language features used in the codebase.
--- AGENTS.md | 416 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 416 insertions(+) create mode 100644 AGENTS.md
AGENTS.md new file +416 -0
@@ -0,0 +1,416 @@ +SPDX-License-Identifier: GPL-2.0-only + +Copyright 2026 John Kacur <[email protected]> + +# AGENTS.md - AI Coding Assistant Guide for rt-tests + +This document provides guidance for AI coding assistants working with the rt-tests codebase. + +## Project Overview + +**rt-tests** is a collection of programs for testing real-time Linux kernel behavior. These tools measure scheduling latency, test priority inheritance, validate deadline scheduling, and stress-test various real-time features of the Linux kernel. + +- **License**: GPL-2.0-only (some files GPL-2.0-or-later) +- **Language**: C (with some Python utilities) +- **Maintainers**: John Kacur <[email protected]>, Clark Williams <[email protected]> +- **Repository**: https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git +- **Mailing List**: [email protected] + +## Architecture + +### Directory Structure + +``` +rt-tests/ +├── src/ # Source code organized by program +│ ├── cyclictest/ # Cyclictest program (main latency tester) +│ ├── hackbench/ # Scheduler benchmark/stress test +│ ├── hwlatdetect/ # Hardware latency detector (Python) +│ ├── pi_tests/ # Priority inheritance tests +│ ├── pmqtest/ # POSIX message queue test +│ ├── ptsematest/ # POSIX semaphore test +│ ├── queuelat/ # Network queue latency detector +│ ├── rt-migrate-test/ # RT migration test +│ ├── sched_deadline/ # SCHED_DEADLINE tests +│ ├── signaltest/ # Signal latency test +│ ├── sigwaittest/ # Signal wait test +│ ├── ssdd/ # Stress, Strain, Duty, Duration test +│ ├── svsematest/ # SysV semaphore test +│ ├── oslat/ # OS latency detector +│ ├── backfire/ # Backfire test utility +│ ├── lib/ # Shared library code +│ │ ├── rt-utils.c # Common utilities (affinity, prio, tracing) +│ │ ├── rt-numa.c # NUMA support functions +│ │ ├── rt-error.c # Error handling +│ │ ├── rt-sched.c # Scheduler utilities +│ │ ├── rt-get_cpu.c # CPU detection +│ │ └── histogram.c # Histogram generation +│ ├── include/ # Common header files +│ │ ├── rt-utils.h # Utility function declarations +│ │ ├── rt-numa.h # NUMA declarations +│ │ ├── rt-error.h # Error handling declarations +│ │ ├── rt-sched.h # Scheduler declarations +│ │ ├── rt-get_cpu.h # CPU detection declarations +│ │ ├── histogram.h # Histogram declarations +│ │ ├── pi_stress.h # PI stress test definitions +│ │ ├── pip_stress.h # PIP stress test definitions +│ │ └── bionic.h # Android Bionic support +│ └── arch/ # Architecture-specific code +│ └── android/ # Android-specific support +├── Makefile # Build system +├── README.markdown # Usage documentation +├── MAINTAINERS # Maintainer contact information +├── COPYING # License text +├── Dockerfile # Container build configuration +├── get_pylib.py # Python library path helper +├── feature/ # Feature detection for build +└── bld/ # Build output directory +``` + +### Core Programs + +**cyclictest** - High-resolution timer latency tester (primary tool): +- Measures scheduling latency and timer jitter +- Multi-threaded with per-CPU thread support +- Integrates with kernel tracing (ftrace) +- Supports histogram output +- Can trigger trace on latency threshold (`--breaktrace`) +- Options for affinity, priority, interval, duration +- Man page: src/cyclictest/cyclictest.8 + +**hackbench** - Scheduler stress test and benchmark: +- Creates groups of processes/threads exchanging messages +- Stresses scheduler and IPC mechanisms +- Useful for finding worst-case latencies under load + +**pi_stress / pip_stress** - Priority inheritance tests: +- Tests kernel priority inheritance mechanisms +- Validates priority ceiling protocol +- Helps debug priority inversion issues + +**pmqtest / ptsematest / svsematest** - IPC latency tests: +- Test POSIX message queues, POSIX semaphores, SysV semaphores +- Measure wakeup latency through various IPC mechanisms + +**hwlatdetect** - Hardware/firmware latency detector (Python): +- Detects System Management Interrupts (SMIs) and firmware delays +- Uses kernel hwlat detector module +- Runs in debugfs environment + +**queuelat** - Network queue latency detector: +- Measures latency caused by network queue processing +- Tests impact of network load on real-time performance + +**deadline_test / cyclicdeadline** - SCHED_DEADLINE tests: +- Test and validate SCHED_DEADLINE scheduler +- Deadline-based real-time scheduling + +**oslat** - OS latency detector: +- Detects OS-induced latencies +- Measures worst-case OS interference + +**signaltest / sigwaittest** - Signal latency tests: +- Test signal delivery latency +- Measure real-time signal performance + +**ssdd** - Stress, Strain, Duty, Duration: +- Configurable stress test with various patterns +- Tests system under different workloads + +**rt-migrate-test** - RT task migration test: +- Tests real-time task migration behavior +- Validates scheduler migration logic + +### Shared Library Components + +**rt-utils.c / rt-utils.h** - Common utilities: +- `parse_cpumask()` - Parse CPU affinity masks +- `numa_initialize()` - Initialize NUMA support +- `enable_trace_mark()` - Enable kernel trace markers +- `tracemark()` - Write trace markers +- `get_tracers()` - Get available kernel tracers +- `setkfifo()` - Set SCHED_FIFO priority +- Priority, affinity, and tracing helpers + +**rt-numa.c / rt-numa.h** - NUMA support: +- CPU topology detection +- NUMA node management +- Per-node thread allocation + +**rt-error.c / rt-error.h** - Error handling: +- `fatal()` - Fatal error with exit +- `err_exit()` - Error with errno +- `warn()` - Warning messages +- `info()` - Informational messages + +**histogram.c / histogram.h** - Latency histogram generation: +- Collect latency samples into bins +- Format histogram output +- Calculate statistics (min, max, avg) + +**rt-sched.c / rt-sched.h** - Scheduler utilities: +- Scheduler policy helpers +- Priority validation + +**rt-get_cpu.c / rt-get_cpu.h** - CPU detection: +- Get current CPU number +- CPU affinity helpers + +### Python Utilities + +**hwlatdetect.py** - Hardware latency detector: +- Detects SMIs and firmware-induced delays +- Mounts/unmounts debugfs as needed +- Uses kernel's hwlat_detector module +- Supports duration, threshold, and window parameters + +**get_cyclictest_snapshot.py** - Cyclictest snapshot utility: +- Captures cyclictest state while running +- Reads /dev/shm/cyclictest-* files +- Provides statistics and thread status + +**get_pylib.py** - Python library path helper: +- Determines correct Python library installation path +- Used by Makefile for Python tool installation + +## Development Guidelines + +### Code Style + +- Follow Linux kernel coding style (K&R style) +- Use tabs for indentation (width 8) +- Maximum line length: 80 characters +- SPDX license identifier at top of each file: `// SPDX-License-Identifier: GPL-2.0-only` +- Function names: lowercase with underscores (e.g., `parse_cpumask`) +- Variable names: lowercase with underscores +- Constants: UPPERCASE with underscores +- **Checking code style**: Use the kernel's checkpatch.pl script to verify code style compliance: + ```bash + /path/to/kernel/scripts/checkpatch.pl --no-tree -f <file.c> + ``` + +### C Standards and Dependencies + +- **C Standard**: C99 or later +- **POSIX**: POSIX.1-2001 with real-time extensions +- **Required Libraries**: + - pthread (POSIX threads) + - rt (POSIX real-time extensions) + - numa (NUMA support, optional) + - dl (dynamic loading) +- **Optional Libraries**: + - cpupower (for deepest idle state support) + +### Python Code + +- **Python Version**: Python 3.8+ +- **Style**: PEP 8 Python style guide +- **Shebang**: `#!/usr/bin/python3` +- Use standard library modules when possible +- Include docstrings for modules, classes, and functions + +### Error Handling + +- Use `err_exit()` for fatal errors with errno +- Use `fatal()` for fatal errors without errno +- Use `warn()` for warnings that don't stop execution +- Check return values from all system calls +- Handle interrupted system calls (EINTR) +- Clean up resources (threads, memory, file descriptors) on error + +### Real-Time Programming Considerations + +1. **Memory Locking**: Lock memory with `mlockall(MCL_CURRENT | MCL_FUTURE)` to prevent page faults +2. **Priority**: Use SCHED_FIFO or SCHED_DEADLINE for real-time threads +3. **Affinity**: Pin threads to specific CPUs to avoid migration overhead +4. **Stack Prefault**: Touch stack pages before real-time operation +5. **Avoid Blocking**: Minimize system calls and library calls in RT critical sections +6. **Signal Handling**: Use real-time signals when possible +7. **Trace Integration**: Support ftrace integration for latency debugging + +## Build and Installation + +### Build Commands + +```bash +# Install build dependencies (Debian/Ubuntu) +sudo apt-get install build-essential libnuma-dev + +# Install build dependencies (Fedora/RHEL) +sudo dnf install gcc make numactl-devel kernel-tools-libs-devel + +# Build all programs +make + +# Build with debugging +make DEBUG=1 + +# Build specific program +make cyclictest + +# Clean build artifacts +make clean + +# Generate tags +make tags +``` + +### Installation + +```bash +# Install to /usr/local (default) +sudo make install + +# Install to custom prefix +sudo make prefix=/usr install + +# Install with compressed man pages +sudo make MAN_COMPRESSION=gzip install +``` + +### Cross-Compilation + +```bash +# Set cross-compiler +make CROSS_COMPILE=aarch64-linux-gnu- + +# Build for Android +make ostype=android +``` + +## Testing + +### Running cyclictest + +```bash +# Basic test: one thread per CPU, SCHED_FIFO priority 99 +sudo ./cyclictest -a -t -p99 + +# With histogram output +sudo ./cyclictest -a -t -p99 -h 400 -q + +# With trace on latency spike +sudo ./cyclictest -a -t -p99 --tracemark --breaktrace=100 + +# Long duration test (24 hours) +sudo ./cyclictest -a -t -p99 -D 24h -h 400 -q +``` + +### Running hwlatdetect + +```bash +# Detect hardware latency for 60 seconds +sudo ./hwlatdetect --duration=60 + +# With threshold +sudo ./hwlatdetect --duration=60 --threshold=10 +``` + +### Testing Priority Inheritance + +```bash +# Run pi_stress test +sudo ./pi_stress + +# Run with specific parameters +sudo ./pi_stress --inversions=100 --duration=60 +``` + +## Key Files + +- `src/cyclictest/cyclictest.c` - Main latency testing tool +- `src/lib/rt-utils.c` - Common utility functions +- `src/lib/rt-numa.c` - NUMA support +- `src/include/rt-utils.h` - Utility declarations +- `src/hwlatdetect/hwlatdetect.py` - Hardware latency detector +- `Makefile` - Build configuration +- `README.markdown` - Usage documentation + +## Common Gotchas + +1. **Root Privileges**: Most tests require root to set real-time priorities and lock memory +2. **Kernel Configuration**: RT tests require kernel with CONFIG_PREEMPT_RT or CONFIG_PREEMPT for best results +3. **CPU Isolation**: For accurate results, isolate CPUs from scheduler (isolcpus kernel parameter) +4. **Power Management**: Disable CPU frequency scaling and idle states for consistent results +5. **NUMA Systems**: Be aware of memory locality on NUMA systems +6. **Tracing Overhead**: Kernel tracing adds overhead; use only when needed +7. **Signal Handling**: Proper cleanup on SIGINT/SIGTERM is important +8. **Memory Locking**: Must lock memory before enabling real-time priority + +## Build System Notes + +The Makefile uses feature detection to conditionally enable features: + +- **libcpupower**: Detected via feature/test-feature.mak, enables `--deepest-idle-state` option +- **NUMA**: Detected at build time, links with libnuma if available +- **Python**: Detected via get_pylib.py, determines if Python utilities can be installed +- **Architecture**: Detected via `$(CC) -dumpmachine` for platform-specific code + +### Makefile Variables + +- `VERSION`: Current version number +- `DESTDIR`: Installation destination (for packaging) +- `prefix`: Installation prefix (default: /usr/local) +- `DEBUG`: Enable debug build (DEBUG=1) +- `CROSS_COMPILE`: Cross-compiler prefix +- `no_libcpupower`: Disable libcpupower (no_libcpupower=1) + +## Support and Contact + +- **Maintainers**: + - John Kacur <[email protected]> + - Clark Williams <[email protected]> +- **Mailing List**: [email protected] +- **Bug Reports**: Send to maintainers with mailing list CC'd +- **Patch Submissions**: Send to maintainers with mailing list CC'd + +## Git Workflow + +- **Main Branch**: `master` +- **Repository**: https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git +- **Patch Submission**: Use standard git format-patch/send-email workflow +- **Commit Format**: Follow kernel commit message format + +### Development Workflow + +1. Make changes to code +2. Build and test: `make && sudo ./cyclictest -a -t -p99` +3. Run relevant tests to verify changes +4. Update man pages if changing command-line options +5. Commit with descriptive message +6. Submit patches to maintainers with mailing list CC'd + +## Additional Notes for AI Assistants + +1. **Real-Time Focus**: All programs are designed for real-time system testing and analysis +2. **Kernel Integration**: Heavy integration with Linux kernel features (ftrace, debugfs, scheduler) +3. **Thread Safety**: Most programs are multi-threaded; be aware of synchronization +4. **Signal Handling**: Programs must handle SIGINT/SIGTERM for graceful shutdown +5. **Resource Cleanup**: Always clean up threads, memory, and file descriptors +6. **Man Page Updates**: Update man pages when changing options or behavior +7. **Backwards Compatibility**: Maintain compatibility with existing scripts and tools +8. **Performance Critical**: Some code paths are performance-critical; avoid adding overhead +9. **Architecture Support**: Code should work on x86_64, aarch64, ppc64le, s390x, RISC-V +10. **Android Support**: Some programs support Android via Bionic compatibility layer + +## Related Projects + +Projects in the RT ecosystem: + +- **rteval**: Real-time evaluation suite - https://git.kernel.org/pub/scm/utils/rteval/rteval.git +- **tuna**: Thread and IRQ tuning tool - https://git.kernel.org/pub/scm/utils/tuna/tuna.git +- **rtla**: Real-time Linux analysis tool - https://www.kernel.org/doc/html/latest/tools/rtla/ +- **rv**: Runtime verification tool - https://www.kernel.org/doc/html/latest/tools/rv/ +- **Linux RT Kernel**: PREEMPT_RT patches - https://kernel.org/pub/linux/kernel/projects/rt/ + +## References + +- **Linux Real-Time Wiki**: https://wiki.linuxfoundation.org/realtime/start +- **OSADL Latency Plots**: https://www.osadl.org/QA-Farm-Realtime.qa-farm-about.0.html +- **RT Kernel Documentation**: https://www.kernel.org/doc/Documentation/scheduler/sched-rt-group.txt +- **High-resolution timers**: https://www.kernel.org/doc/html/latest/timers/hrtimers.html + +--- + +**Last Updated**: 2026-06-17 +**Document Version**: 1.0
Signature · 1 line
2.54.0
Re: [PATCH 3/3] rt-tests: Add AGENTS.md guide for AI coding assistants
Quoted 435 lines · John Kacur
This guide is also useful for human developers new to the project.
Add comprehensive documentation for AI coding assistants working on the rt-tests codebase. This guide covers:
- Project overview and architecture
- Core programs (cyclictest, hwlatdetect,pi_stress, etc.)
- Shared library components and build system
- Development guidelines including code style and checkpatch usage
- Testing procedures and debugging tools
- Common gotchas for real-time testing
- Related projects in the RT ecosystem (rteval, tuna, rtla, rv)
- Kernel documentation referencesThe guide is designed to help AI assistants quickly understand the codebase structure, key patterns, and best practices when making changes to rt-tests.
Python requirement set to 3.8+ based on language features used in the codebase.
Assisted-by Claude Sonnet 4.5 <[email protected]>Signed-off-by John Kacur <[email protected]>--- AGENTS.md | 416 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 416 insertions(+) create mode 100644 AGENTS.mdAGENTS.md new file +401
-0@@ -0,0 +1,416 @@ +SPDX-License-Identifier: GPL-2.0-only + +Copyright 2026 John Kacur <[email protected]> + +# AGENTS.md - AI Coding Assistant Guide for rt-tests + +This document provides guidance for AI coding assistants working with the rt-tests codebase. + +## Project Overview + +**rt-tests** is a collection of programs for testing real-time Linux kernel behavior. These tools measure scheduling latency, test priority inheritance, validate deadline scheduling, and stress-test various real-time features of the Linux kernel. + +- **License**: GPL-2.0-only (some files GPL-2.0-or-later) +- **Language**: C (with some Python utilities) +- **Maintainers**: John Kacur <[email protected]>, Clark Williams <[email protected]> +- **Repository**: https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git +- **Mailing List**: [email protected] + +## Architecture + +### Directory Structure + +``` +rt-tests/ +├── src/ # Source code organized by program +│ ├── cyclictest/ # Cyclictest program (main latency tester) +│ ├── hackbench/ # Scheduler benchmark/stress test +│ ├── hwlatdetect/ # Hardware latency detector (Python) +│ ├── pi_tests/ # Priority inheritance tests +│ ├── pmqtest/ # POSIX message queue test +│ ├── ptsematest/ # POSIX semaphore test +│ ├── queuelat/ # Network queue latency detector +│ ├── rt-migrate-test/ # RT migration test +│ ├── sched_deadline/ # SCHED_DEADLINE tests +│ ├── signaltest/ # Signal latency test +│ ├── sigwaittest/ # Signal wait test +│ ├── ssdd/ # Stress, Strain, Duty, Duration test +│ ├── svsematest/ # SysV semaphore test +│ ├── oslat/ # OS latency detector +│ ├── backfire/ # Backfire test utility +│ ├── lib/ # Shared library code +│ │ ├── rt-utils.c # Common utilities (affinity, prio, tracing) +│ │ ├── rt-numa.c # NUMA support functions +│ │ ├── rt-error.c # Error handling +│ │ ├── rt-sched.c # Scheduler utilities +│ │ ├── rt-get_cpu.c # CPU detection +│ │ └── histogram.c # Histogram generation +│ ├── include/ # Common header files +│ │ ├── rt-utils.h # Utility function declarations +│ │ ├── rt-numa.h # NUMA declarations +│ │ ├── rt-error.h # Error handling declarations +│ │ ├── rt-sched.h # Scheduler declarations +│ │ ├── rt-get_cpu.h # CPU detection declarations +│ │ ├── histogram.h # Histogram declarations +│ │ ├── pi_stress.h # PI stress test definitions +│ │ ├── pip_stress.h # PIP stress test definitions +│ │ └── bionic.h # Android Bionic support +│ └── arch/ # Architecture-specific code +│ └── android/ # Android-specific support +├── Makefile # Build system +├── README.markdown # Usage documentation +├── MAINTAINERS # Maintainer contact information +├── COPYING # License text +├── Dockerfile # Container build configuration +├── get_pylib.py # Python library path helper +├── feature/ # Feature detection for build +└── bld/ # Build output directory +``` + +### Core Programs + +**cyclictest** - High-resolution timer latency tester (primary tool): +- Measures scheduling latency and timer jitter +- Multi-threaded with per-CPU thread support +- Integrates with kernel tracing (ftrace) +- Supports histogram output +- Can trigger trace on latency threshold (`--breaktrace`) +- Options for affinity, priority, interval, duration +- Man page: src/cyclictest/cyclictest.8 + +**hackbench** - Scheduler stress test and benchmark: +- Creates groups of processes/threads exchanging messages +- Stresses scheduler and IPC mechanisms +- Useful for finding worst-case latencies under load + +**pi_stress / pip_stress** - Priority inheritance tests: +- Tests kernel priority inheritance mechanisms +- Validates priority ceiling protocol +- Helps debug priority inversion issues + +**pmqtest / ptsematest / svsematest** - IPC latency tests: +- Test POSIX message queues, POSIX semaphores, SysV semaphores +- Measure wakeup latency through various IPC mechanisms + +**hwlatdetect** - Hardware/firmware latency detector (Python): +- Detects System Management Interrupts (SMIs) and firmware delays +- Uses kernel hwlat detector module +- Runs in debugfs environment + +**queuelat** - Network queue latency detector: +- Measures latency caused by network queue processing +- Tests impact of network load on real-time performance + +**deadline_test / cyclicdeadline** - SCHED_DEADLINE tests: +- Test and validate SCHED_DEADLINE scheduler +- Deadline-based real-time scheduling + +**oslat** - OS latency detector: +- Detects OS-induced latencies +- Measures worst-case OS interference + +**signaltest / sigwaittest** - Signal latency tests: +- Test signal delivery latency +- Measure real-time signal performance + +**ssdd** - Stress, Strain, Duty, Duration: +- Configurable stress test with various patterns +- Tests system under different workloads + +**rt-migrate-test** - RT task migration test: +- Tests real-time task migration behavior +- Validates scheduler migration logic + +### Shared Library Components + +**rt-utils.c / rt-utils.h** - Common utilities: +- `parse_cpumask()` - Parse CPU affinity masks +- `numa_initialize()` - Initialize NUMA support +- `enable_trace_mark()` - Enable kernel trace markers +- `tracemark()` - Write trace markers +- `get_tracers()` - Get available kernel tracers +- `setkfifo()` - Set SCHED_FIFO priority +- Priority, affinity, and tracing helpers + +**rt-numa.c / rt-numa.h** - NUMA support: +- CPU topology detection +- NUMA node management +- Per-node thread allocation + +**rt-error.c / rt-error.h** - Error handling: +- `fatal()` - Fatal error with exit +- `err_exit()` - Error with errno +- `warn()` - Warning messages +- `info()` - Informational messages + +**histogram.c / histogram.h** - Latency histogram generation: +- Collect latency samples into bins +- Format histogram output +- Calculate statistics (min, max, avg) + +**rt-sched.c / rt-sched.h** - Scheduler utilities: +- Scheduler policy helpers +- Priority validation + +**rt-get_cpu.c / rt-get_cpu.h** - CPU detection: +- Get current CPU number +- CPU affinity helpers + +### Python Utilities + +**hwlatdetect.py** - Hardware latency detector: +- Detects SMIs and firmware-induced delays +- Mounts/unmounts debugfs as needed +- Uses kernel's hwlat_detector module +- Supports duration, threshold, and window parameters + +**get_cyclictest_snapshot.py** - Cyclictest snapshot utility: +- Captures cyclictest state while running +- Reads /dev/shm/cyclictest-* files +- Provides statistics and thread status + +**get_pylib.py** - Python library path helper: +- Determines correct Python library installation path +- Used by Makefile for Python tool installation + +## Development Guidelines + +### Code Style + +- Follow Linux kernel coding style (K&R style) +- Use tabs for indentation (width 8) +- Maximum line length: 80 characters +- SPDX license identifier at top of each file: `// SPDX-License-Identifier: GPL-2.0-only` +- Function names: lowercase with underscores (e.g., `parse_cpumask`) +- Variable names: lowercase with underscores +- Constants: UPPERCASE with underscores +- **Checking code style**: Use the kernel's checkpatch.pl script to verify code style compliance: + ```bash + /path/to/kernel/scripts/checkpatch.pl --no-tree -f <file.c> + ``` + +### C Standards and Dependencies + +- **C Standard**: C99 or later +- **POSIX**: POSIX.1-2001 with real-time extensions +- **Required Libraries**: + - pthread (POSIX threads) + - rt (POSIX real-time extensions) + - numa (NUMA support, optional) + - dl (dynamic loading) +- **Optional Libraries**: + - cpupower (for deepest idle state support) + +### Python Code + +- **Python Version**: Python 3.8+ +- **Style**: PEP 8 Python style guide +- **Shebang**: `#!/usr/bin/python3` +- Use standard library modules when possible +- Include docstrings for modules, classes, and functions + +### Error Handling + +- Use `err_exit()` for fatal errors with errno +- Use `fatal()` for fatal errors without errno +- Use `warn()` for warnings that don't stop execution +- Check return values from all system calls +- Handle interrupted system calls (EINTR) +- Clean up resources (threads, memory, file descriptors) on error + +### Real-Time Programming Considerations + +1. **Memory Locking**: Lock memory with `mlockall(MCL_CURRENT | MCL_FUTURE)` to prevent page faults +2. **Priority**: Use SCHED_FIFO or SCHED_DEADLINE for real-time threads +3. **Affinity**: Pin threads to specific CPUs to avoid migration overhead +4. **Stack Prefault**: Touch stack pages before real-time operation +5. **Avoid Blocking**: Minimize system calls and library calls in RT critical sections +6. **Signal Handling**: Use real-time signals when possible +7. **Trace Integration**: Support ftrace integration for latency debugging + +## Build and Installation + +### Build Commands + +```bash +# Install build dependencies (Debian/Ubuntu) +sudo apt-get install build-essential libnuma-dev + +# Install build dependencies (Fedora/RHEL) +sudo dnf install gcc make numactl-devel kernel-tools-libs-devel + +# Build all programs +make + +# Build with debugging +make DEBUG=1 + +# Build specific program +make cyclictest + +# Clean build artifacts +make clean + +# Generate tags +make tags +``` + +### Installation + +```bash +# Install to /usr/local (default) +sudo make install + +# Install to custom prefix +sudo make prefix=/usr install + +# Install with compressed man pages +sudo make MAN_COMPRESSION=gzip install +``` + +### Cross-Compilation + +```bash +# Set cross-compiler +make CROSS_COMPILE=aarch64-linux-gnu- + +# Build for Android +make ostype=android +``` + +## Testing + +### Running cyclictest + +```bash +# Basic test: one thread per CPU, SCHED_FIFO priority 99 +sudo ./cyclictest -a -t -p99 + +# With histogram output +sudo ./cyclictest -a -t -p99 -h 400 -q + +# With trace on latency spike +sudo ./cyclictest -a -t -p99 --tracemark --breaktrace=100 + +# Long duration test (24 hours) +sudo ./cyclictest -a -t -p99 -D 24h -h 400 -q +``` + +### Running hwlatdetect + +```bash +# Detect hardware latency for 60 seconds +sudo ./hwlatdetect --duration=60 + +# With threshold +sudo ./hwlatdetect --duration=60 --threshold=10 +``` + +### Testing Priority Inheritance + +```bash +# Run pi_stress test +sudo ./pi_stress + +# Run with specific parameters +sudo ./pi_stress --inversions=100 --duration=60 +``` + +## Key Files + +- `src/cyclictest/cyclictest.c` - Main latency testing tool +- `src/lib/rt-utils.c` - Common utility functions +- `src/lib/rt-numa.c` - NUMA support +- `src/include/rt-utils.h` - Utility declarations +- `src/hwlatdetect/hwlatdetect.py` - Hardware latency detector +- `Makefile` - Build configuration +- `README.markdown` - Usage documentation + +## Common Gotchas + +1. **Root Privileges**: Most tests require root to set real-time priorities and lock memory +2. **Kernel Configuration**: RT tests require kernel with CONFIG_PREEMPT_RT or CONFIG_PREEMPT for best results +3. **CPU Isolation**: For accurate results, isolate CPUs from scheduler (isolcpus kernel parameter) +4. **Power Management**: Disable CPU frequency scaling and idle states for consistent results +5. **NUMA Systems**: Be aware of memory locality on NUMA systems +6. **Tracing Overhead**: Kernel tracing adds overhead; use only when needed +7. **Signal Handling**: Proper cleanup on SIGINT/SIGTERM is important +8. **Memory Locking**: Must lock memory before enabling real-time priority + +## Build System Notes + +The Makefile uses feature detection to conditionally enable features: + +- **libcpupower**: Detected via feature/test-feature.mak, enables `--deepest-idle-state` option +- **NUMA**: Detected at build time, links with libnuma if available +- **Python**: Detected via get_pylib.py, determines if Python utilities can be installed +- **Architecture**: Detected via `$(CC) -dumpmachine` for platform-specific code + +### Makefile Variables + +- `VERSION`: Current version number +- `DESTDIR`: Installation destination (for packaging) +- `prefix`: Installation prefix (default: /usr/local) +- `DEBUG`: Enable debug build (DEBUG=1) +- `CROSS_COMPILE`: Cross-compiler prefix +- `no_libcpupower`: Disable libcpupower (no_libcpupower=1) + +## Support and Contact + +- **Maintainers**: + - John Kacur <[email protected]> + - Clark Williams <[email protected]> +- **Mailing List**: [email protected] +- **Bug Reports**: Send to maintainers with mailing list CC'd +- **Patch Submissions**: Send to maintainers with mailing list CC'd + +## Git Workflow + +- **Main Branch**: `master` +- **Repository**: https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git +- **Patch Submission**: Use standard git format-patch/send-email workflow +- **Commit Format**: Follow kernel commit message format + +### Development Workflow + +1. Make changes to code +2. Build and test: `make && sudo ./cyclictest -a -t -p99` +3. Run relevant tests to verify changes +4. Update man pages if changing command-line options +5. Commit with descriptive message +6. Submit patches to maintainers with mailing list CC'd + +## Additional Notes for AI Assistants + +1. **Real-Time Focus**: All programs are designed for real-time system testing and analysis +2. **Kernel Integration**: Heavy integration with Linux kernel features (ftrace, debugfs, scheduler) +3. **Thread Safety**: Most programs are multi-threaded; be aware of synchronization +4. **Signal Handling**: Programs must handle SIGINT/SIGTERM for graceful shutdown +5. **Resource Cleanup**: Always clean up threads, memory, and file descriptors +6. **Man Page Updates**: Update man pages when changing options or behavior +7. **Backwards Compatibility**: Maintain compatibility with existing scripts and tools +8. **Performance Critical**: Some code paths are performance-critical; avoid adding overhead +9. **Architecture Support**: Code should work on x86_64, aarch64, ppc64le, s390x, RISC-V +10. **Android Support**: Some programs support Android via Bionic compatibility layer + +## Related Projects + +Projects in the RT ecosystem: + +- **rteval**: Real-time evaluation suite - https://git.kernel.org/pub/scm/utils/rteval/rteval.git +- **tuna**: Thread and IRQ tuning tool - https://git.kernel.org/pub/scm/utils/tuna/tuna.git
AGENTS.md line 411
Quoted 10 lines
AGENTS.md new file +10
-0+- **rtla**: Real-time Linux analysis tool - https://www.kernel.org/doc/html/latest/tools/rtla/ +- **rv**: Runtime verification tool - https://www.kernel.org/doc/html/latest/tools/rv/ +- **Linux RT Kernel**: PREEMPT_RT patches - https://kernel.org/pub/linux/kernel/projects/rt/ + +## References + +- **Linux Real-Time Wiki**: https://wiki.linuxfoundation.org/realtime/start +- **OSADL Latency Plots**: https://www.osadl.org/QA-Farm-Realtime.qa-farm-about.0.html +- **RT Kernel Documentation**: https://www.kernel.org/doc/Documentation/scheduler/sched-rt-group.txt +- **High-resolution timers**: https://www.kernel.org/doc/html/latest/timers/hrtimers.html
Sebastian also added a complete documentation guide for PREEMPT_RT:
https://docs.kernel.org/core-api/real-time/index.html
Thanks,
Kurt