remove old kernel-ci and enhance reboot-limit
cover remove old kernel-ci and enhance reboot-limit
The CONFIG_KERNEL_CI thing was used back in the hay day before we adopted fstests SOAK_DURATION. Although CONFIG_KERNEL_CI proved useful to test fstests in a loop -- the only reasonable way to scale is to just embrace SOAK_DURATION and extend any twitches we need within fstests.
However only one workflow uses CONFIG_KERNEL_CI still. So move that functionality directly into the reboot-limit workflow which uses it. While at it, as a drive by, let's enhance the reboot limit workflow with kexec and also graphing capabilities allowing us to do A/B testing for reboots and kexec regression testings.
The rationale for this work si to help fix any misunderatandings on what kdevops's kernel CI support consists of, and CONFIG_KERNEL_CI certainly was not it. It's evolved now to just the kdevops kernel-ci repo and the the .ci/ symlinks, along with optional kernel-kpd trees for subsystems.
Luis Chamberlain (23):
fstests: remove CONFIG_KERNEL_CI support
fstests: remove kernel-ci script symlinks
blktests: remove CONFIG_KERNEL_CI support
gitr: remove CONFIG_KERNEL_CI support
ltp: remove CONFIG_KERNEL_CI support
nfstest: remove CONFIG_KERNEL_CI support
pynfs: remove CONFIG_KERNEL_CI support
reboot-limit: convert CONFIG_KERNEL_CI to internal loop feature
kconfig: remove CONFIG_KERNEL_CI infrastructure
scripts: remove kernel-ci loop infrastructure
reboot-limit: simplify what gets selected
reboot-limit: add graph visualization support for results
reboot-limit: save graphs in organized results/graphs directory
docs: add comprehensive reboot-limit workflow documentation
reboot-limit: add kexec-tools dependency installation
reboot-limit: add A/B testing support targets
reboot-limit: fix kexec and reboot connection handling
reboot-limit: add COUNT parameter to override reboot count
reboot-limit: fix wait_for tasks using wrong host reference
reboot-limit: use ansible reboot module for all reboot types
reboot-limit: fix COUNT parameter to properly override reboot count
reboot-limit: handle empty dev group gracefully
reboot-limit: add kexec comparison feature
README.md | 19 +- defconfigs/kernel-testing | 1 - defconfigs/reboot-limit | 31 + defconfigs/reboot-limit-compare | 36 ++ defconfigs/reboot-limit-kexec | 32 + docs/fstests.md | 13 +- docs/images/reboot_limit_example.png | Bin 0 -> 735262 bytes docs/reboot-limit.md | 223 +++++++ kconfigs/Kconfig.kdevops | 3 - kconfigs/Kconfig.kernel-ci | 170 ----- playbooks/roles/gen_hosts/tasks/main.yml | 13 + playbooks/roles/gen_hosts/templates/hosts.j2 | 55 +- playbooks/roles/gen_nodes/tasks/main.yml | 32 + .../reboot-limit/tasks/do-reboot-compare.yml | 126 ++++ .../roles/reboot-limit/tasks/do-reboot.yml | 98 ++- .../reboot-limit/tasks/handle-reboot-data.yml | 94 +++ .../tasks/install-deps/debian/main.yml | 10 + .../reboot-limit/tasks/install-deps/main.yml | 12 + .../tasks/install-deps/redhat/main.yml | 10 + .../tasks/install-deps/suse/main.yml | 10 + playbooks/roles/reboot-limit/tasks/main.yml | 88 ++- scripts/workflows/blktests/run_kernel_ci.sh | 1 - .../workflows/blktests/run_kernel_ci_kotd.sh | 1 - scripts/workflows/blktests/run_loop.sh | 66 -- .../demos/reboot-limit/analyze_results.py | 609 ++++++++++++++++++ .../reboot-limit/generate_sample_data.py | 73 +++ .../demos/reboot-limit/run_kernel_ci.sh | 1 - .../demos/reboot-limit/run_kernel_ci_kotd.sh | 1 - .../workflows/demos/reboot-limit/run_loop.sh | 23 +- .../demos/reboot-limit/run_loop_kotd.sh | 25 + scripts/workflows/fstests/run_kernel_ci.sh | 1 - .../workflows/fstests/run_kernel_ci_kotd.sh | 1 - scripts/workflows/fstests/run_loop.sh | 98 --- scripts/workflows/generic/run_kernel_ci.sh | 295 --------- scripts/workflows/gitr/run_kernel_ci.sh | 1 - scripts/workflows/gitr/run_kernel_ci_kotd.sh | 1 - scripts/workflows/gitr/run_loop.sh | 58 -- scripts/workflows/kotd/run_kernel_ci_kotd.sh | 107 --- scripts/workflows/ltp/run_kernel_ci.sh | 1 - scripts/workflows/ltp/run_kernel_ci_kotd.sh | 1 - scripts/workflows/ltp/run_loop.sh | 58 -- scripts/workflows/nfstest/run_kernel_ci.sh | 1 - .../workflows/nfstest/run_kernel_ci_kotd.sh | 1 - scripts/workflows/nfstest/run_loop.sh | 58 -- scripts/workflows/pynfs/run_kernel_ci.sh | 1 - scripts/workflows/pynfs/run_kernel_ci_kotd.sh | 1 - scripts/workflows/pynfs/run_loop.sh | 79 --- workflows/blktests/Makefile | 13 - workflows/demos/reboot-limit/Kconfig | 64 +- workflows/demos/reboot-limit/Makefile | 117 +++- workflows/fstests/Makefile | 14 - workflows/gitr/Makefile | 19 - workflows/ltp/Makefile | 19 - workflows/nfstest/Makefile | 19 - workflows/pynfs/Makefile | 26 - 55 files changed, 1737 insertions(+), 1193 deletions(-) create mode 100644 defconfigs/reboot-limit create mode 100644 defconfigs/reboot-limit-compare create mode 100644 defconfigs/reboot-limit-kexec create mode 100644 docs/images/reboot_limit_example.png create mode 100644 docs/reboot-limit.md delete mode 100644 kconfigs/Kconfig.kernel-ci create mode 100644 playbooks/roles/reboot-limit/tasks/do-reboot-compare.yml create mode 100644 playbooks/roles/reboot-limit/tasks/handle-reboot-data.yml create mode 100644 playbooks/roles/reboot-limit/tasks/install-deps/debian/main.yml create mode 100644 playbooks/roles/reboot-limit/tasks/install-deps/main.yml create mode 100644 playbooks/roles/reboot-limit/tasks/install-deps/redhat/main.yml create mode 100644 playbooks/roles/reboot-limit/tasks/install-deps/suse/main.yml delete mode 120000 scripts/workflows/blktests/run_kernel_ci.sh delete mode 120000 scripts/workflows/blktests/run_kernel_ci_kotd.sh delete mode 100755 scripts/workflows/blktests/run_loop.sh create mode 100755 scripts/workflows/demos/reboot-limit/analyze_results.py create mode 100755 scripts/workflows/demos/reboot-limit/generate_sample_data.py delete mode 120000 scripts/workflows/demos/reboot-limit/run_kernel_ci.sh delete mode 120000 scripts/workflows/demos/reboot-limit/run_kernel_ci_kotd.sh create mode 100755 scripts/workflows/demos/reboot-limit/run_loop_kotd.sh delete mode 120000 scripts/workflows/fstests/run_kernel_ci.sh delete mode 120000 scripts/workflows/fstests/run_kernel_ci_kotd.sh delete mode 100755 scripts/workflows/fstests/run_loop.sh delete mode 100755 scripts/workflows/generic/run_kernel_ci.sh delete mode 120000 scripts/workflows/gitr/run_kernel_ci.sh delete mode 120000 scripts/workflows/gitr/run_kernel_ci_kotd.sh delete mode 100755 scripts/workflows/gitr/run_loop.sh delete mode 100755 scripts/workflows/kotd/run_kernel_ci_kotd.sh delete mode 120000 scripts/workflows/ltp/run_kernel_ci.sh delete mode 120000 scripts/workflows/ltp/run_kernel_ci_kotd.sh delete mode 100755 scripts/workflows/ltp/run_loop.sh delete mode 120000 scripts/workflows/nfstest/run_kernel_ci.sh delete mode 120000 scripts/workflows/nfstest/run_kernel_ci_kotd.sh delete mode 100755 scripts/workflows/nfstest/run_loop.sh delete mode 120000 scripts/workflows/pynfs/run_kernel_ci.sh delete mode 120000 scripts/workflows/pynfs/run_kernel_ci_kotd.sh delete mode 100755 scripts/workflows/pynfs/run_loop.sh
Signature · 1 line
2.47.2
PATCH 1/23 fstests: remove CONFIG_KERNEL_CI support
The CONFIG_KERNEL_CI feature was originally added for loop testing fstests, but SOAK_DURATION has proven to be a better architectural choice for long-term fstests testing.
Remove CONFIG_KERNEL_CI support from fstests by:
- Removing FSTEST_KERNEL_CI_LOOP variables
- Removing fstests-baseline-loop target
- Removing fstests-baseline-kernelci target
- Updating help menu to remove references
--- workflows/fstests/Makefile | 14 -------------- 1 file changed, 14 deletions(-)
workflows/fstests/Makefile +0 -14
@@ -73,12 +73,6 @@ WORKFLOW_ARGS += $(FSTESTS_ARGS) WORKFLOW_ARGS_SEPARATED += $(FSTESTS_ARGS_SEPARATED) WORKFLOW_ARGS_DIRECT += $(FSTESTS_ARGS_DIRECT) -FSTEST_KERNEL_CI_LOOP := false -FSTEST_KERNEL_CI_LOOP_KOTD := false -ifeq (y,$(CONFIG_KERNEL_CI)) -FSTEST_KERNEL_CI_LOOP := scripts/workflows/fstests/run_kernel_ci.sh -FSTEST_KERNEL_CI_LOOP_KOTD := scripts/workflows/fstests/run_kernel_ci_kotd.sh -endif # CONFIG_KERNEL_CI FSTESTS_DYNAMIC_RUNTIME_VARS := "kdevops_run_fstests": True ifneq (,$(RUN_FAILURES))
@@ -158,12 +152,6 @@ fstests-baseline: $(FSTESTS_BASELINE_EXTRA) --extra-vars '{ $(FSTESTS_DYNAMIC_RUNTIME_VARS) }' \ --extra-vars=@./extra_vars.yaml $(LIMIT_HOSTS) -fstests-baseline-loop: $(KDEVOPS_EXTRA_VARS) - $(Q)$(FSTEST_KERNEL_CI_LOOP) baseline $(LIMIT_HOSTS) - -fstests-baseline-kernelci: $(KDEVOPS_EXTRA_VARS) - $(Q)$(FSTEST_KERNEL_CI_LOOP_KOTD) baseline $(LIMIT_HOSTS) - fstests-baseline-skip-kdevops-update: $(KDEVOPS_EXTRA_VARS) $(Q)ansible-playbook $(ANSIBLE_VERBOSE) \ --limit 'baseline' \
@@ -271,8 +259,6 @@ fstests-help-menu: @echo "fstests-dev-results - Only collect results for dev hosts" @echo "fstests-baseline-skip-kdevops-update - Run fstests but skip updating kdevops" @echo "fstests-baseline-run-oscheck-only - Only run oscheck" - @echo "fstests-baseline-loop - Run fstests in a loop until a failure is found or steady state is reached" - @echo "fstests-baseline-kernelci - Run fstests kernel-ci loop" @echo "" HELP_TARGETS += fstests-help-menu
Signature · 1 line
2.47.2
PATCH 2/23 fstests: remove kernel-ci script symlinks
In reply to Luis ChamberlainRemove the fstests-specific symlinks to the generic kernel-ci scripts as they are no longer needed after removing CONFIG_KERNEL_CI support from fstests.
--- scripts/workflows/fstests/run_kernel_ci.sh | 1 - scripts/workflows/fstests/run_kernel_ci_kotd.sh | 1 - 2 files changed, 2 deletions(-) delete mode 120000 scripts/workflows/fstests/run_kernel_ci.sh delete mode 120000 scripts/workflows/fstests/run_kernel_ci_kotd.sh
scripts/workflows/fstests/run_kernel_ci.sh deleted +0 -1
@@ -1 +0,0 @@ -../generic/run_kernel_ci.sh
scripts/workflows/fstests/run_kernel_ci_kotd.sh deleted +0 -1
@@ -1 +0,0 @@ -../kotd/run_kernel_ci_kotd.sh
Signature · 1 line
2.47.2
PATCH 3/23 blktests: remove CONFIG_KERNEL_CI support
In reply to Luis ChamberlainRemove CONFIG_KERNEL_CI support from blktests workflow:
- Remove BLKTEST_KERNEL_CI_LOOP variables
- Remove blktests-baseline-loop target
- Remove blktests-baseline-kernelci target
- Update help menu to remove references
- Remove kernel-ci script symlinks
--- scripts/workflows/blktests/run_kernel_ci.sh | 1 - scripts/workflows/blktests/run_kernel_ci_kotd.sh | 1 - workflows/blktests/Makefile | 13 ------------- 3 files changed, 15 deletions(-) delete mode 120000 scripts/workflows/blktests/run_kernel_ci.sh delete mode 120000 scripts/workflows/blktests/run_kernel_ci_kotd.sh
scripts/workflows/blktests/run_kernel_ci.sh deleted +0 -1
@@ -1 +0,0 @@ -../generic/run_kernel_ci.sh
scripts/workflows/blktests/run_kernel_ci_kotd.sh deleted +0 -1
@@ -1 +0,0 @@ -../kotd/run_kernel_ci_kotd.sh
workflows/blktests/Makefile +0 -13
@@ -42,12 +42,6 @@ BLKTESTS_ARGS += blktests_data_target='$(BLKTESTS_DATA_TARGET)' WORKFLOW_ARGS += $(BLKTESTS_ARGS) -BLKTEST_KERNEL_CI_LOOP := false -BLKTEST_KERNEL_CI_LOOP_KOTD := false -ifeq (y,$(CONFIG_KERNEL_CI)) -BLKTEST_KERNEL_CI_LOOP := scripts/workflows/blktests/run_kernel_ci.sh -BLKTEST_KERNEL_CI_LOOP_KOTD := scripts/workflows/blktests/run_kernel_ci_kotd.sh -endif # CONFIG_KERNEL_CI EXTRA_VAR_INPUTS += extend-extra-args-blktests
@@ -78,11 +72,6 @@ blktests-baseline: --extra-vars '{ $(BLKTESTS_DYNAMIC_RUNTIME_VARS) }' \ --extra-vars=@./extra_vars.yaml $(LIMIT_HOSTS) -blktests-baseline-loop: - $(Q)$(BLKTEST_KERNEL_CI_LOOP) baseline $(LIMIT_HOSTS) - -blktests-baseline-kernelci: - $(Q)$(BLKTEST_KERNEL_CI_LOOP_KOTD) baseline $(LIMIT_HOSTS) blktests-baseline-skip-kdevops-update: $(Q)ansible-playbook $(ANSIBLE_VERBOSE) \
@@ -136,8 +125,6 @@ blktests-help-menu: @echo "blktests-dev-results - Only collect results for dev hosts" @echo "blktests-baseline-skip-kdevops-update - Run blktests but skip updating kdevops" @echo "blktests-baseline-run-oscheck-only - Only run oscheck" - @echo "blktests-baseline-loop - Run blktests in a loop until a failure is found or steady state is reached" - @echo "blktests-baseline-kernelci - Run blktests kernel-ci loop" @echo "" HELP_TARGETS += blktests-help-menu
Signature · 1 line
2.47.2
PATCH 4/23 gitr: remove CONFIG_KERNEL_CI support
In reply to Luis ChamberlainRemove CONFIG_KERNEL_CI support from gitr workflow:
- Remove GITR_KERNEL_CI_LOOP variables
- Remove gitr-baseline-loop and gitr-dev-loop targets
- Remove gitr-baseline-kernelci and gitr-dev-kernelci targets
- Update help menu to remove references
- Remove kernel-ci script symlinks
--- scripts/workflows/gitr/run_kernel_ci.sh | 1 - scripts/workflows/gitr/run_kernel_ci_kotd.sh | 1 - workflows/gitr/Makefile | 19 ------------------- 3 files changed, 21 deletions(-) delete mode 120000 scripts/workflows/gitr/run_kernel_ci.sh delete mode 120000 scripts/workflows/gitr/run_kernel_ci_kotd.sh
scripts/workflows/gitr/run_kernel_ci.sh deleted +0 -1
@@ -1 +0,0 @@ -../generic/run_kernel_ci.sh
scripts/workflows/gitr/run_kernel_ci_kotd.sh deleted +0 -1
@@ -1 +0,0 @@ -../kotd/run_kernel_ci_kotd.sh
workflows/gitr/Makefile +0 -19
@@ -62,13 +62,6 @@ ifeq (y,$(CONFIG_KDEVOPS_WORKFLOW_DEDICATE_GITR)) WORKFLOW_ARGS_SEPARATED += gitr_enabled_test_groups='$(subst $(space),+,$(GITR_ENABLED_TEST_GROUPS))' endif -GITR_KERNEL_CI_LOOP := false -GITR_KERNEL_CI_LOOP_KOTD := false - -ifeq (y,$(CONFIG_KERNEL_CI)) -GITR_KERNEL_CI_LOOP := scripts/workflows/gitr/run_kernel_ci.sh -GITR_KERNEL_CI_LOOP_KOTD := scripts/workflows/gitr/run_kernel_ci_kotd.sh -endif # CONFIG_KERNEL_CI ifndef LAST_KERNEL LAST_KERNEL := $(shell cat workflows/gitr/results/last-kernel.txt 2>/dev/null)
@@ -101,11 +94,6 @@ gitr-baseline: --extra-vars=@./extra_vars.yaml \ $(KDEVOPS_PLAYBOOKS_DIR)/gitr.yml -gitr-baseline-loop: - $(Q)$(GITR_KERNEL_CI_LOOP) baseline - -gitr-baseline-kernelci: - $(Q)$(GITR_KERNEL_CI_LOOP_KOTD) baseline gitr-dev-baseline: $(Q)ansible-playbook $(ANSIBLE_VERBOSE) \
@@ -114,11 +102,6 @@ gitr-dev-baseline: --extra-vars=@./extra_vars.yaml \ $(KDEVOPS_PLAYBOOKS_DIR)/gitr.yml -gitr-dev-loop: - $(Q)$(GITR_KERNEL_CI_LOOP) dev - -gitr-dev-kernelci: - $(Q)$(GITR_KERNEL_CI_LOOP_KOTD) dev gitr-dev-reset: $(Q)ansible-playbook $(ANSIBLE_VERBOSE) \
@@ -136,8 +119,6 @@ gitr-help-menu: @echo "gitr options:" @echo "gitr - Git clone git, build and install it" @echo "gitr-{baseline,dev} - Run the gitr test on baseline or dev hosts and collect results" - @echo "gitr-{baseline,dev}-loop" - Run gitr in a loop until error or steady state - @echo "gitr-{baseline,dev}-kernelci - Run gitr kernel-ci loop" @echo "" HELP_TARGETS += gitr-help-menu
Signature · 1 line
2.47.2
PATCH 5/23 ltp: remove CONFIG_KERNEL_CI support
In reply to Luis ChamberlainRemove CONFIG_KERNEL_CI support from ltp workflow:
- Remove LTP_KERNEL_CI_LOOP variables
- Remove ltp-baseline-loop and ltp-dev-loop targets
- Remove ltp-baseline-kernelci and ltp-dev-kernelci targets
- Update help menu to remove references
- Remove kernel-ci script symlinks
--- scripts/workflows/ltp/run_kernel_ci.sh | 1 - scripts/workflows/ltp/run_kernel_ci_kotd.sh | 1 - workflows/ltp/Makefile | 19 ------------------- 3 files changed, 21 deletions(-) delete mode 120000 scripts/workflows/ltp/run_kernel_ci.sh delete mode 120000 scripts/workflows/ltp/run_kernel_ci_kotd.sh
scripts/workflows/ltp/run_kernel_ci.sh deleted +0 -1
@@ -1 +0,0 @@ -../generic/run_kernel_ci.sh
scripts/workflows/ltp/run_kernel_ci_kotd.sh deleted +0 -1
@@ -1 +0,0 @@ -../kotd/run_kernel_ci_kotd.sh
workflows/ltp/Makefile +0 -19
@@ -88,13 +88,6 @@ endif # CONFIG_LTP_TESTS_TIRPC WORKFLOW_ARGS += $(LTP_ARGS) WORKFLOW_ARGS_SEPARATED += ltp_enabled_test_groups='$(subst $(space),+,$(LTP_ENABLED_TEST_GROUPS))' -LTP_KERNEL_CI_LOOP := false -LTP_KERNEL_CI_LOOP_KOTD := false - -ifeq (y,$(CONFIG_KERNEL_CI)) -LTP_KERNEL_CI_LOOP := scripts/workflows/ltp/run_kernel_ci.sh -LTP_KERNEL_CI_LOOP_KOTD := scripts/workflows/ltp/run_kernel_ci_kotd.sh -endif # CONFIG_KERNEL_CI ifndef LAST_KERNEL LAST_KERNEL := $(shell cat workflows/ltp/results/last-kernel.txt 2>/dev/null)
@@ -127,11 +120,6 @@ ltp-baseline: --extra-vars=@./extra_vars.yaml \ $(KDEVOPS_PLAYBOOKS_DIR)/ltp.yml -ltp-baseline-loop: - $(Q)$(LTP_KERNEL_CI_LOOP) baseline - -ltp-baseline-kernelci: - $(Q)$(LTP_KERNEL_CI_LOOP_KOTD) baseline ltp-dev-baseline: $(Q)ansible-playbook $(ANSIBLE_VERBOSE) \
@@ -140,11 +128,6 @@ ltp-dev-baseline: --extra-vars=@./extra_vars.yaml \ $(KDEVOPS_PLAYBOOKS_DIR)/ltp.yml -ltp-dev-loop: - $(Q)$(LTP_KERNEL_CI_LOOP) dev - -ltp-dev-kernelci: - $(Q)$(LTP_KERNEL_CI_LOOP_KOTD) dev ltp-dev-reset: $(Q)ansible-playbook $(ANSIBLE_VERBOSE) \
@@ -162,8 +145,6 @@ ltp-help-menu: @echo "ltp options:" @echo "ltp - Git clone ltp, build and install it" @echo "ltp-{baseline,dev} - Run the ltp test on baseline or dev hosts and collect results" - @echo "ltp-{baseline,dev}-loop" - Run ltp in a loop until error or steady state - @echo "ltp-{baseline,dev}-kernelci - Run ltp kernel-ci loop" @echo "" HELP_TARGETS += ltp-help-menu
Signature · 1 line
2.47.2
PATCH 6/23 nfstest: remove CONFIG_KERNEL_CI support
In reply to Luis ChamberlainRemove CONFIG_KERNEL_CI support from nfstest workflow:
- Remove NFSTEST_KERNEL_CI_LOOP variables
- Remove nfstest-baseline-loop and nfstest-dev-loop targets
- Remove nfstest-baseline-kernelci and nfstest-dev-kernelci targets
- Update help menu to remove references
- Remove kernel-ci script symlinks
--- scripts/workflows/nfstest/run_kernel_ci.sh | 1 - .../workflows/nfstest/run_kernel_ci_kotd.sh | 1 - workflows/nfstest/Makefile | 19 ------------------- 3 files changed, 21 deletions(-) delete mode 120000 scripts/workflows/nfstest/run_kernel_ci.sh delete mode 120000 scripts/workflows/nfstest/run_kernel_ci_kotd.sh
scripts/workflows/nfstest/run_kernel_ci.sh deleted +0 -1
@@ -1 +0,0 @@ -../generic/run_kernel_ci.sh
scripts/workflows/nfstest/run_kernel_ci_kotd.sh deleted +0 -1
@@ -1 +0,0 @@ -../kotd/run_kernel_ci_kotd.sh
workflows/nfstest/Makefile +0 -19
@@ -45,13 +45,6 @@ endif WORKFLOW_ARGS += $(NFSTEST_ARGS) WORKFLOW_ARGS_SEPARATED += nfstest_enabled_test_groups='$(subst $(space),+,$(NFSTEST_ENABLED_TEST_GROUPS))' -NFSTEST_KERNEL_CI_LOOP := false -NFSTEST_KERNEL_CI_LOOP_KOTD := false - -ifeq (y,$(CONFIG_KERNEL_CI)) -NFSTEST_KERNEL_CI_LOOP := scripts/workflows/nfstest/run_kernel_ci.sh -NFSTEST_KERNEL_CI_LOOP_KOTD := scripts/workflows/nfstest/run_kernel_ci_kotd.sh -endif # CONFIG_KERNEL_CI ifndef LAST_KERNEL LAST_KERNEL := $(shell cat workflows/nfstest/results/last-kernel.txt 2>/dev/null)
@@ -84,11 +77,6 @@ nfstest-baseline: --extra-vars=@./extra_vars.yaml \ $(KDEVOPS_PLAYBOOKS_DIR)/nfstest.yml -nfstest-baseline-loop: - $(Q)$(NFSTEST_KERNEL_CI_LOOP) baseline - -nfstest-baseline-kernelci: - $(Q)$(NFSTEST_KERNEL_CI_LOOP_KOTD) baseline nfstest-dev-baseline: $(Q)ansible-playbook $(ANSIBLE_VERBOSE) \
@@ -97,11 +85,6 @@ nfstest-dev-baseline: --extra-vars=@./extra_vars.yaml \ $(KDEVOPS_PLAYBOOKS_DIR)/nfstest.yml -nfstest-dev-loop: - $(Q)$(NFSTEST_KERNEL_CI_LOOP) dev - -nfstest-dev-kernelci: - $(Q)$(NFSTEST_KERNEL_CI_LOOP_KOTD) dev nfstest-dev-reset: $(Q)ansible-playbook $(ANSIBLE_VERBOSE) \
@@ -119,8 +102,6 @@ nfstest-help-menu: @echo "nfstest options:" @echo "nfstest - Git clone nfstest and install it" @echo "nfstest-{baseline,dev} - Run selected nfstests on baseline or dev hosts and collect results" - @echo "nfstest-{baseline,dev}-loop" - Run nfstest in a loop until error or steady state - @echo "nfstest-{baseline,dev}-kernelci - Run nfstest kernel-ci loop" @echo "" HELP_TARGETS += nfstest-help-menu
Signature · 1 line
2.47.2
PATCH 7/23 pynfs: remove CONFIG_KERNEL_CI support
In reply to Luis ChamberlainRemove CONFIG_KERNEL_CI support from pynfs workflow:
- Remove PYNFS_KERNEL_CI_LOOP variables and associated comments
- Remove pynfs-baseline-loop and pynfs-dev-loop targets
- Remove pynfs-baseline-kernelci and pynfs-dev-kernelci targets
- Update help menu to remove references
- Remove kernel-ci script symlinks
--- scripts/workflows/pynfs/run_kernel_ci.sh | 1 - scripts/workflows/pynfs/run_kernel_ci_kotd.sh | 1 - workflows/pynfs/Makefile | 26 ------------------- 3 files changed, 28 deletions(-) delete mode 120000 scripts/workflows/pynfs/run_kernel_ci.sh delete mode 120000 scripts/workflows/pynfs/run_kernel_ci_kotd.sh
scripts/workflows/pynfs/run_kernel_ci.sh deleted +0 -1
@@ -1 +0,0 @@ -../generic/run_kernel_ci.sh
scripts/workflows/pynfs/run_kernel_ci_kotd.sh deleted +0 -1
@@ -1 +0,0 @@ -../kotd/run_kernel_ci_kotd.sh
workflows/pynfs/Makefile +0 -26
@@ -14,15 +14,6 @@ endif # CONFIG_PYNFS_PNFS_BLOCK WORKFLOW_ARGS += $(PYNFS_ARGS) -# The default is our workflow does not have kernel-ci enabled -PYNFS_KERNEL_CI_LOOP := false -PYNFS_KERNEL_CI_LOOP_KOTD := false - -# If kdevops was configured to enable kernel-ci we define our scripts -ifeq (y,$(CONFIG_KERNEL_CI)) -PYNFS_KERNEL_CI_LOOP := scripts/workflows/pynfs/run_kernel_ci.sh -PYNFS_KERNEL_CI_LOOP_KOTD := scripts/workflows/pynfs/run_kernel_ci_kotd.sh -endif # CONFIG_KERNEL_CI ifndef LAST_KERNEL LAST_KERNEL := $(shell cat workflows/pynfs/results/last-kernel.txt 2>/dev/null)
@@ -61,16 +52,6 @@ pynfs-baseline: --extra-vars=@./extra_vars.yaml \ $(KDEVOPS_PLAYBOOKS_DIR)/pynfs.yml -# Once you know the baseline works, you may want to run the baseline in a loop -# up to the number of times you define. We define this goal in kdevops as -# the CONFIG_KERNEL_CI_STEADY_STATE_GOAL. -pynfs-baseline-loop: - $(Q)$(PYNFS_KERNEL_CI_LOOP) baseline - -# The kdevops kernel-ci target will try to upgrade your kernel to the latest -# and then run the baseline-loop. -pynfs-baseline-kernelci: - $(Q)$(PYNFS_KERNEL_CI_LOOP_KOTD) baseline # Below are the corresponding dev targets pynfs-dev-baseline:
@@ -85,11 +66,6 @@ pynfs-dev-baseline: --extra-vars=@./extra_vars.yaml \ $(KDEVOPS_PLAYBOOKS_DIR)/pynfs.yml -pynfs-dev-loop: - $(Q)$(PYNFS_KERNEL_CI_LOOP) dev - -pynfs-dev-kernelci: - $(Q)$(PYNFS_KERNEL_CI_LOOP_KOTD) dev pynfs-dev-reset: $(Q)ansible-playbook $(ANSIBLE_VERBOSE) \
@@ -107,8 +83,6 @@ pynfs-help-menu: @echo "pynfs options:" @echo "pynfs - Git clone pynfs, build and install it" @echo "pynfs-{baseline,dev} - Run the pynfs test on baseline or dev hosts and collect results" - @echo "pynfs-{baseline,dev}-loop" - Run pynfs in a loop until error or steady state - @echo "pynfs-{baseline,dev}-kernelci - Run pynfs kernel-ci loop" @echo "" HELP_TARGETS += pynfs-help-menu
Signature · 1 line
2.47.2
PATCH 8/23 reboot-limit: convert CONFIG_KERNEL_CI to internal loop feature
In reply to Luis ChamberlainConvert reboot-limit workflow to use internal loop configuration instead of the global CONFIG_KERNEL_CI feature:
- Remove CONFIG_KERNEL_CI selects from reboot-limit Kconfig
- Add internal REBOOT_LIMIT_ENABLE_LOOP configuration
- Add REBOOT_LIMIT_LOOP_STEADY_STATE_GOAL for loop count control
- Add REBOOT_LIMIT_LOOP_STEADY_STATE_INCREMENTAL for resumable loops
- Update Makefile to use internal variables instead of kernel-ci
- Update run_loop.sh to use reboot-limit specific configs
- Rename targets from -kernelci to -kotd (kernel-of-the-day)
- Remove kernel-ci script symlinks and add loop scripts
- Update help menu to reflect new target names
This makes reboot-limit self-contained with its own loop testing configuration rather than relying on the global kernel-ci feature that's being removed.
--- .../demos/reboot-limit/run_kernel_ci.sh | 1 - .../demos/reboot-limit/run_kernel_ci_kotd.sh | 1 - .../workflows/demos/reboot-limit/run_loop.sh | 23 +++++++----- .../demos/reboot-limit/run_loop_kotd.sh | 25 +++++++++++++ workflows/demos/reboot-limit/Kconfig | 29 +++++++++++++-- workflows/demos/reboot-limit/Makefile | 35 ++++++++++--------- 6 files changed, 85 insertions(+), 29 deletions(-) delete mode 120000 scripts/workflows/demos/reboot-limit/run_kernel_ci.sh delete mode 120000 scripts/workflows/demos/reboot-limit/run_kernel_ci_kotd.sh create mode 100755 scripts/workflows/demos/reboot-limit/run_loop_kotd.sh
scripts/workflows/demos/reboot-limit/run_kernel_ci.sh deleted +0 -1
@@ -1 +0,0 @@ -../../generic/run_kernel_ci.sh
scripts/workflows/demos/reboot-limit/run_kernel_ci_kotd.sh deleted +0 -1
@@ -1 +0,0 @@ -../../kotd/run_kernel_ci_kotd.sh
scripts/workflows/demos/reboot-limit/run_loop.sh +15 -8
@@ -1,18 +1,25 @@ #!/bin/bash # SPDX-License-Identifier: copyleft-next-0.3.1 -# Part of kdevops kernel-ci, this is the script which will run the test workflow -# as many times as indicated up to CONFIG_KERNEL_CI_STEADY_STATE_GOAL +# Part of kdevops reboot-limit loop testing, this script will run the test workflow +# as many times as indicated up to CONFIG_REBOOT_LIMIT_LOOP_STEADY_STATE_GOAL source ${TOPDIR}/.config source ${TOPDIR}/scripts/lib.sh -COUNT=1 +# Check if we're resuming from a previous run +if [[ "$CONFIG_REBOOT_LIMIT_LOOP_STEADY_STATE_INCREMENTAL" == "y" && -f $KERNEL_CI_OK_FILE ]]; then + COUNT=$(cat $KERNEL_CI_OK_FILE) + let COUNT=$COUNT+1 +else + COUNT=1 + rm -f $KERNEL_CI_OK_FILE +fi run_loop() { while true; do - echo "== kernel-ci reboot-limit test loop $COUNT start: $(date)" > $KERNEL_CI_FAIL_LOG + echo "== reboot-limit test loop $COUNT start: $(date)" > $KERNEL_CI_FAIL_LOG echo "/usr/bin/time -f %E make reboot-limit-baseline" >> $KERNEL_CI_FAIL_LOG /usr/bin/time -p -o $KERNEL_CI_LOGTIME make reboot-limit-baseline >> $KERNEL_CI_FAIL_LOG ANSIBLE_CALL_RET=$?
@@ -43,14 +50,14 @@ run_loop() echo $COUNT > $KERNEL_CI_OK_FILE let COUNT=$COUNT+1 - if [[ "$CONFIG_KERNEL_CI_ENABLE_STEADY_STATE" == "y" && - "$COUNT" -gt "$CONFIG_KERNEL_CI_STEADY_STATE_GOAL" ]]; then + if [[ "$CONFIG_REBOOT_LIMIT_ENABLE_LOOP" == "y" && + "$COUNT" -gt "$CONFIG_REBOOT_LIMIT_LOOP_STEADY_STATE_GOAL" ]]; then exit 0 fi sleep 1 done } -rm -f $KERNEL_CI_FAIL_FILE $KERNEL_CI_OK_FILE -echo "= kernel-ci full log" > $KERNEL_CI_FULL_LOG +rm -f $KERNEL_CI_FAIL_FILE +echo "= reboot-limit loop full log" > $KERNEL_CI_FULL_LOG run_loop
scripts/workflows/demos/reboot-limit/run_loop_kotd.sh new file +25 -0
@@ -0,0 +1,25 @@ +#!/bin/bash +# SPDX-License-Identifier: copyleft-next-0.3.1 + +# Part of kdevops reboot-limit loop testing with kernel-of-the-day +# This script would normally update to the latest kernel before running the loop, +# but since CONFIG_KERNEL_CI is being removed, this is now just a placeholder +# that runs the regular loop. + +if [[ "$TOPDIR" == "" ]]; then + TOPDIR=$PWD +fi + +source ${TOPDIR}/.config +source ${TOPDIR}/scripts/lib.sh + +TARGET_HOSTS="baseline" +if [[ "$1" != "" ]]; then + TARGET_HOSTS=$1 +fi + +echo "Note: kernel-of-the-day updates are not yet implemented for reboot-limit" +echo "Running regular reboot-limit loop instead..." + +# For now, just run the regular loop +${TOPDIR}/scripts/workflows/demos/reboot-limit/run_loop.sh
workflows/demos/reboot-limit/Kconfig +27 -2
@@ -4,8 +4,6 @@ config WORKFLOWS_REBOOT_LIMIT select KDEVOPS_TRY_INSTALL_KDEV_TOOLS select KDEVOPS_DEVCONFIG_ENABLE_CONSOLE select KDEVOPS_DEVCONFIG_ENABLE_SYSTEMD_WATCHDOG - select KERNEL_CI - select KERNEL_CI_ENABLE_STEADY_STATE help If you are doing kernel tests which includes a reboot, along the way, or if you do a reboot after a series of tests, you may want to first
@@ -71,6 +69,33 @@ config REBOOT_LIMIT_WATCHDOG_CHECK_TIME endif +config REBOOT_LIMIT_ENABLE_LOOP + bool "Enable loop testing for reboot-limit" + default y + help + Enable this to run reboot-limit tests in a loop until a failure + is found or steady state is reached. + +if REBOOT_LIMIT_ENABLE_LOOP + +config REBOOT_LIMIT_LOOP_STEADY_STATE_GOAL + int "Steady-state goal for reboot-limit loop testing" + default 100 + help + The maximum number of successful test loops to run before + declaring steady state achieved. Each loop runs REBOOT_LIMIT_BOOT_MAX + reboots, so the total number of reboots is this value multiplied + by REBOOT_LIMIT_BOOT_MAX. + +config REBOOT_LIMIT_LOOP_STEADY_STATE_INCREMENTAL + bool "Use incremental steady-state counting" + default y + help + Enable this so that the loop will resume from the last success + counter after failures rather than resetting to zero. + +endif # REBOOT_LIMIT_ENABLE_LOOP + config REBOOT_LIMIT_BOOT_COUNT_CRASH_ENABLE bool "Force a crash after certain period of reboots" help
workflows/demos/reboot-limit/Makefile +18 -17
@@ -70,15 +70,15 @@ endif # the extra_vars.yaml file. WORKFLOW_ARGS += $(REBOOT_LIMIT_ARGS) -# The default is our workflow does not have kernel-ci enabled -REBOOT_LIMIT_KERNEL_CI_LOOP := false -REBOOT_LIMIT_KERNEL_CI_LOOP_KOTD := false +# The default is our workflow does not have loop testing enabled +REBOOT_LIMIT_LOOP := false +REBOOT_LIMIT_LOOP_KOTD := false -# If kdevops was configured to enable kernel-ci we define our scripts -ifeq (y,$(CONFIG_KERNEL_CI)) -REBOOT_LIMIT_KERNEL_CI_LOOP := scripts/workflows/demos/reboot-limit/run_kernel_ci.sh -REBOOT_LIMIT_KERNEL_CI_LOOP_KOTD := scripts/workflows/demos/reboot-limit/run_kernel_ci_kotd.sh -endif # CONFIG_KERNEL_CI +# If reboot-limit was configured to enable loop testing we define our scripts +ifeq (y,$(CONFIG_REBOOT_LIMIT_ENABLE_LOOP)) +REBOOT_LIMIT_LOOP := scripts/workflows/demos/reboot-limit/run_loop.sh +REBOOT_LIMIT_LOOP_KOTD := scripts/workflows/demos/reboot-limit/run_loop_kotd.sh +endif # CONFIG_REBOOT_LIMIT_ENABLE_LOOP # By default KDEVOPS_HOSTS_TEMPLATE is defined to be hosts.in, if you want an # alternative you can define one, or even have a dynamic one. For an example
@@ -146,14 +146,14 @@ reboot-limit-baseline: # Once you know the baseline works, you may want to run the baseline in a loop # up to the number of times you define. We define this goal in kdevops as -# the CONFIG_KERNEL_CI_STEADY_STATE_GOAL. +# the CONFIG_REBOOT_LIMIT_LOOP_STEADY_STATE_GOAL. reboot-limit-baseline-loop: - $(Q)$(REBOOT_LIMIT_KERNEL_CI_LOOP) baseline + $(Q)$(REBOOT_LIMIT_LOOP) baseline -# The kdevops kernel-ci target will try to upgrade your kernel to the latest +# The kdevops kotd (kernel-of-the-day) target will try to upgrade your kernel to the latest # and then run the baseline-loop. -reboot-limit-baseline-kernelci: - $(Q)$(REBOOT_LIMIT_KERNEL_CI_LOOP_KOTD) baseline +reboot-limit-baseline-kotd: + $(Q)$(REBOOT_LIMIT_LOOP_KOTD) baseline # Resets the boot counters so we start from scratch reboot-limit-baseline-reset:
@@ -177,10 +177,10 @@ reboot-limit-dev-baseline: --extra-vars=@./extra_vars.yaml reboot-limit-dev-loop: - $(Q)$(REBOOT_LIMIT_KERNEL_CI_LOOP) dev + $(Q)$(REBOOT_LIMIT_LOOP) dev -reboot-limit-dev-kernelci: - $(Q)$(REBOOT_LIMIT_KERNEL_CI_LOOP_KOTD) dev +reboot-limit-dev-kotd: + $(Q)$(REBOOT_LIMIT_LOOP_KOTD) dev reboot-limit-dev-reset: $(Q)ansible-playbook $(ANSIBLE_VERBOSE) \
@@ -195,7 +195,8 @@ reboot-limit-help-menu: @echo "reboot-limit-baseline - Run the reboot-linit test on baseline hosts and collect results" @echo "reboot-limit-baseline-reset - Reset the test boot counter for baseline" @echo "reboot-limit-dev - Run the reboot-limti test on dev hosts and collect results" - @echo "reboot-limit-baseline-kernelci - Run the reboot-limit kernel-ci loop" + @echo "reboot-limit-baseline-loop - Run the reboot-limit test in a loop until failure or steady state" + @echo "reboot-limit-baseline-kotd - Run the reboot-limit kotd (kernel-of-the-day) loop" @echo "" HELP_TARGETS += reboot-limit-help-menu
Signature · 1 line
2.47.2
PATCH 9/23 kconfig: remove CONFIG_KERNEL_CI infrastructure
In reply to Luis ChamberlainRemove the CONFIG_KERNEL_CI Kconfig infrastructure now that all workflows have been converted to use their own internal loop configurations or SOAK_DURATION:
- Remove Kconfig.kernel-ci file with all kernel-ci options
- Remove kernel-ci menu and source from Kconfig.kdevops
- Remove CONFIG_KERNEL_CI from kernel-testing defconfig
- Update fstests documentation to mention SOAK_DURATION instead
This completes the removal of the confusing CONFIG_KERNEL_CI feature that was originally added for loop testing but has been superseded by better architectural choices.
--- defconfigs/kernel-testing | 1 - docs/fstests.md | 13 +-- kconfigs/Kconfig.kdevops | 3 - kconfigs/Kconfig.kernel-ci | 170 ------------------------------------- 4 files changed, 4 insertions(+), 183 deletions(-) delete mode 100644 kconfigs/Kconfig.kernel-ci
defconfigs/kernel-testing +0 -1
@@ -2,5 +2,4 @@ CONFIG_KDEVOPS_TRY_REFRESH_REPOS=y CONFIG_KDEVOPS_TRY_UPDATE_SYSTEMS=y CONFIG_KDEVOPS_TRY_INSTALL_KDEV_TOOLS=y CONFIG_GIT_ALTERNATIVES=y -CONFIG_KERNEL_CI=y CONFIG_WORKFLOW_KOTD_ENABLE=y
docs/fstests.md +4 -9
@@ -137,15 +137,10 @@ of a baseline by running fstests in a loop 100 times. If no new failures have been found we consider that we have a high confidence in a baseline and then it can be used to verify if new development changes are not causing a regression. -The amount of loops you choose to test with fstests is configurable -with `CONFIG_KERNEL_CI_STEADY_STATE_GOAL`. We default to 100. This value -is chosen for a few reasons: - - * Running fstests 100 times takes about 1 week - * This is a sensible litmus test to ensure no regressions are - introduced with a new delta of patches, if you want to build confidence - in some possibly intrusive changes to Linux, or backport some patches - without regressing the kernel. +The preferred method for long-term testing with fstests is to use the +SOAK_DURATION feature, which allows specific tests designed for soak +testing to run for extended periods. This provides a more targeted approach +to stress testing compared to simply running all tests in a loop. The value originally comes from kdevops use at SUSE Linux enterprise distributions, as a litmus test to validate kernel releases, so to ensure
kconfigs/Kconfig.kdevops +0 -3
@@ -187,9 +187,6 @@ menu "Ansible post-bring up provisioning configuration" source "kconfigs/Kconfig.ansible_provisioning" endmenu -menu "Kernel continous integration configuration" -source "kconfigs/Kconfig.kernel-ci" -endmenu menu "Ansible Configuration" source "kconfigs/Kconfig.ansible_cfg"
kconfigs/Kconfig.kernel-ci deleted +0 -170
170 lines not on this page · raw message
Signature · 1 line
2.47.2
PATCH 10/23 scripts: remove kernel-ci loop infrastructure
In reply to Luis ChamberlainRemove the now-unused kernel-ci loop testing scripts:
- Remove run_loop.sh from all workflows except reboot-limit
- Remove generic run_kernel_ci.sh script
- Remove kotd/run_kernel_ci_kotd.sh script and empty kotd directory
These scripts were part of the CONFIG_KERNEL_CI infrastructure that has been replaced with workflow-specific implementations or better testing approaches like SOAK_DURATION.
The reboot-limit workflow keeps its run_loop.sh as it now uses internal loop configuration.
--- scripts/workflows/blktests/run_loop.sh | 66 ----- scripts/workflows/fstests/run_loop.sh | 98 ------ scripts/workflows/generic/run_kernel_ci.sh | 295 ------------------- scripts/workflows/gitr/run_loop.sh | 58 ---- scripts/workflows/kotd/run_kernel_ci_kotd.sh | 107 ------- scripts/workflows/ltp/run_loop.sh | 58 ---- scripts/workflows/nfstest/run_loop.sh | 58 ---- scripts/workflows/pynfs/run_loop.sh | 79 ----- 8 files changed, 819 deletions(-) delete mode 100755 scripts/workflows/blktests/run_loop.sh delete mode 100755 scripts/workflows/fstests/run_loop.sh delete mode 100755 scripts/workflows/generic/run_kernel_ci.sh delete mode 100755 scripts/workflows/gitr/run_loop.sh delete mode 100755 scripts/workflows/kotd/run_kernel_ci_kotd.sh delete mode 100755 scripts/workflows/ltp/run_loop.sh delete mode 100755 scripts/workflows/nfstest/run_loop.sh delete mode 100755 scripts/workflows/pynfs/run_loop.sh
scripts/workflows/blktests/run_loop.sh deleted +0 -66
@@ -1,66 +0,0 @@ -#!/bin/bash -# SPDX-License-Identifier: copyleft-next-0.3.1 - -source ${TOPDIR}/.config -source ${TOPDIR}/scripts/lib.sh - -COUNT=1 - -run_loop() -{ - while true; do - echo "== kernel-ci blktests test loop $COUNT start: $(date)" > $KERNEL_CI_FAIL_LOG - echo "/usr/bin/time -f %E make blktests-baseline" >> $KERNEL_CI_FAIL_LOG - /usr/bin/time -p -o $KERNEL_CI_LOGTIME make blktests-baseline >> $KERNEL_CI_FAIL_LOG - echo "End $COUNT: $(date)" >> $KERNEL_CI_FAIL_LOG - cat $KERNEL_CI_LOGTIME >> $KERNEL_CI_FAIL_LOG - echo "git status:" >> $KERNEL_CI_FAIL_LOG - git status >> $KERNEL_CI_FAIL_LOG - echo "Results:" >> $KERNEL_CI_FAIL_LOG - - rm -f $KERNEL_CI_DIFF_LOG - - DIFF_COUNT=$(git diff workflows/blktests/expunges/ | wc -l) - if [[ "$DIFF_COUNT" -ne 0 ]]; then - echo "Detected a failure as reported by differences in our expunge list" >> $KERNEL_CI_DIFF_LOG - fi - - NEW_EXPUNGE_FILES="no" - ${TOPDIR}/playbooks/python/workflows/blktests/get_new_expunge_files.py workflows/blktests/expunges/ > .tmp.new_expunges - NEW_EXPUNGE_FILE_COUNT=$(cat .tmp.new_expunges | wc -l | awk '{print $1}') - if [[ $NEW_EXPUNGE_FILE_COUNT -ne 0 ]]; then - NEW_EXPUNGE_FILES="yes" - echo "Detected a failure since new expunge files were found which are not commited into git" >> $KERNEL_CI_DIFF_LOG - echo "New expunge file found, listing output below:" >> $KERNEL_CI_DIFF_LOG - cat .tmp.new_expunges >> $KERNEL_CI_DIFF_LOG - rm -f .tmp.new_expunges - fi - rm -f .tmp.new_expunges - - if [[ "$DIFF_COUNT" -ne 0 || "$NEW_EXPUNGE_FILES" == "yes" ]]; then - echo "Test $COUNT: FAILED!" >> $KERNEL_CI_DIFF_LOG - echo "== Test loop count $COUNT" >> $KERNEL_CI_DIFF_LOG - echo "$(git describe)" >> $KERNEL_CI_DIFF_LOG - git diff workflows/blktests/expunges/ >> $KERNEL_CI_DIFF_LOG - cat $KERNEL_CI_DIFF_LOG >> $KERNEL_CI_FAIL_LOG - cat $KERNEL_CI_FAIL_LOG >> $KERNEL_CI_FULL_LOG - echo $COUNT > $KERNEL_CI_FAIL_FILE - exit 1 - else - echo "Test $COUNT: OK!" >> $KERNEL_CI_FAIL_LOG - echo "----------------------------------------------------------------" >> $KERNEL_CI_FAIL_LOG - cat $KERNEL_CI_FAIL_LOG >> $KERNEL_CI_FULL_LOG - fi - echo $COUNT > $KERNEL_CI_OK_FILE - let COUNT=$COUNT+1 - if [[ "$CONFIG_KERNEL_CI_ENABLE_STEADY_STATE" == "y" && - "$COUNT" -gt "$CONFIG_KERNEL_CI_STEADY_STATE_GOAL" ]]; then - exit 0 - fi - sleep 1 - done -} - -rm -f $KERNEL_CI_FAIL_FILE $KERNEL_CI_OK_FILE -echo "= kernel-ci full log" > $KERNEL_CI_FULL_LOG -run_loop
scripts/workflows/fstests/run_loop.sh deleted +0 -98
98 lines not on this page · raw message
scripts/workflows/generic/run_kernel_ci.sh deleted +0 -295
295 lines not on this page · raw message
scripts/workflows/gitr/run_loop.sh deleted +0 -58
58 lines not on this page · raw message
scripts/workflows/kotd/run_kernel_ci_kotd.sh deleted +0 -107
107 lines not on this page · raw message
scripts/workflows/ltp/run_loop.sh deleted +0 -58
58 lines not on this page · raw message
scripts/workflows/nfstest/run_loop.sh deleted +0 -58
58 lines not on this page · raw message
scripts/workflows/pynfs/run_loop.sh deleted +0 -79
79 lines not on this page · raw message
Signature · 1 line
2.47.2
PATCH 11/23 reboot-limit: simplify what gets selected
In reply to Luis ChamberlainIf you try to take the reboot-limit workflow for a spin right now you will run into a conflict with networkd.service and the bringup we have on debian-testing. I have narrowed this down to the few extra selects we do, and the reason is that we have not seen these issues before on our CI. Testing with these disabled fixes the issue so just disable those selects as we already do enough of a good job of enabling sensible defaults.
--- workflows/demos/reboot-limit/Kconfig | 4 ---- 1 file changed, 4 deletions(-)
workflows/demos/reboot-limit/Kconfig +0 -4
@@ -1,9 +1,5 @@ config WORKFLOWS_REBOOT_LIMIT bool "Reboot-limit" - select KDEVOPS_TRY_REFRESH_REPOS - select KDEVOPS_TRY_INSTALL_KDEV_TOOLS - select KDEVOPS_DEVCONFIG_ENABLE_CONSOLE - select KDEVOPS_DEVCONFIG_ENABLE_SYSTEMD_WATCHDOG help If you are doing kernel tests which includes a reboot, along the way, or if you do a reboot after a series of tests, you may want to first
Signature · 1 line
2.47.2
PATCH 12/23 reboot-limit: add graph visualization support for results
In reply to Luis ChamberlainAdd support to analyze and visualize reboot-limit workflow results. This helps users understand boot performance trends and identify anomalies across multiple reboots.
The implementation includes:
- analyze_results.py: Parses systemd-analyze output and generates graphs showing boot time trends, component breakdown (kernel/initrd/userspace), and statistical analysis
- generate_sample_data.py: Creates realistic test data for development
- New Makefile targets:
- reboot-limit-results: Analyze and display summary statistics
- reboot-limit-graph: Generate visualization graphs
The visualization provides:
- Stacked area charts showing boot component times
- Total boot time trends with statistical indicators (mean, median, stddev)
- Summary statistics for each host including min/max/range
- Support for multiple hosts (baseline and dev)
--- .../demos/reboot-limit/analyze_results.py | 304 ++++++++++++++++++ .../reboot-limit/generate_sample_data.py | 73 +++++ workflows/demos/reboot-limit/Makefile | 13 +- 3 files changed, 388 insertions(+), 2 deletions(-) create mode 100755 scripts/workflows/demos/reboot-limit/analyze_results.py create mode 100755 scripts/workflows/demos/reboot-limit/generate_sample_data.py
scripts/workflows/demos/reboot-limit/analyze_results.py new file +304 -0
@@ -0,0 +1,304 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: copyleft-next-0.3.1 + +""" +Analyze and visualize reboot-limit workflow results. + +This script parses systemd-analyze output and generates graphs showing: +- Boot time trends across reboots +- Individual component times (kernel, initrd, userspace) +- Statistical analysis of boot performance +""" + +import os +import sys +import re +import argparse +import statistics +from pathlib import Path +import matplotlib.pyplot as plt +import matplotlib.ticker as ticker +from typing import Dict, List, Tuple, Optional + + +class RebootLimitAnalyzer: + """Analyzes reboot-limit workflow results.""" + + def __init__(self, results_dir: str): + self.results_dir = Path(results_dir) + self.hosts_data: Dict[str, Dict] = {} + + def parse_systemd_analyze_line(self, line: str) -> Optional[Dict[str, float]]: + """ + Parse a systemd-analyze output line. + + Example line: + Startup finished in 2.345s (kernel) + 1.234s (initrd) + 5.678s (userspace) = 9.257s + + Returns dict with times in seconds or None if parse fails. + """ + # Pattern for systemd-analyze output + pattern = r"Startup finished in ([\d.]+)s \(kernel\) \+ ([\d.]+)s \(initrd\) \+ ([\d.]+)s \(userspace\) = ([\d.]+)s" + + # Alternative pattern without initrd (for systems without initrd) + pattern_no_initrd = r"Startup finished in ([\d.]+)s \(kernel\) \+ ([\d.]+)s \(userspace\) = ([\d.]+)s" + + match = re.search(pattern, line) + if match: + return { + "kernel": float(match.group(1)), + "initrd": float(match.group(2)), + "userspace": float(match.group(3)), + "total": float(match.group(4)), + } + + match = re.search(pattern_no_initrd, line) + if match: + return { + "kernel": float(match.group(1)), + "initrd": 0.0, + "userspace": float(match.group(2)), + "total": float(match.group(3)), + } + + return None + + def load_host_data(self, host_dir: Path) -> Dict: + """Load and parse data for a single host.""" + data = {"boot_count": 0, "boot_times": []} + + # Read boot count + count_file = host_dir / "reboot-count.txt" + if count_file.exists(): + with open(count_file, "r") as f: + content = f.read().strip() + if content: + data["boot_count"] = int(content) + + # Read systemd-analyze results + analyze_file = host_dir / "systemctl-analyze.txt" + if analyze_file.exists(): + with open(analyze_file, "r") as f: + for line in f: + parsed = self.parse_systemd_analyze_line(line.strip()) + if parsed: + data["boot_times"].append(parsed) + + return data + + def load_all_data(self): + """Load data for all hosts in the results directory.""" + # Look for host directories + for item in self.results_dir.iterdir(): + if item.is_dir(): + self.hosts_data[item.name] = self.load_host_data(item) + + def calculate_statistics(self, times: List[float]) -> Dict[str, float]: + """Calculate statistical measures for a list of times.""" + if not times: + return {} + + return { + "min": min(times), + "max": max(times), + "mean": statistics.mean(times), + "median": statistics.median(times), + "stdev": statistics.stdev(times) if len(times) > 1 else 0, + } + + def plot_boot_times(self, output_file: str = "reboot_limit_analysis.png"): + """Generate plots for boot time analysis.""" + if not self.hosts_data: + print("No data to plot") + return + + # Create figure with subplots + num_hosts = len(self.hosts_data) + fig, axes = plt.subplots(num_hosts, 2, figsize=(14, 6 * num_hosts)) + + if num_hosts == 1: + axes = axes.reshape(1, -1) + + for idx, (host, data) in enumerate(self.hosts_data.items()): + boot_times = data["boot_times"] + if not boot_times: + continue + + # Extract time series + boot_numbers = list(range(1, len(boot_times) + 1)) + kernel_times = [bt["kernel"] for bt in boot_times] + initrd_times = [bt["initrd"] for bt in boot_times] + userspace_times = [bt["userspace"] for bt in boot_times] + total_times = [bt["total"] for bt in boot_times] + + # Plot 1: Stacked area chart of boot components + ax1 = axes[idx, 0] + ax1.fill_between(boot_numbers, 0, kernel_times, alpha=0.7, label="Kernel") + ax1.fill_between( + boot_numbers, + kernel_times, + [k + i for k, i in zip(kernel_times, initrd_times)], + alpha=0.7, + label="Initrd", + ) + ax1.fill_between( + boot_numbers, + [k + i for k, i in zip(kernel_times, initrd_times)], + total_times, + alpha=0.7, + label="Userspace", + ) + + ax1.set_xlabel("Boot Number") + ax1.set_ylabel("Time (seconds)") + ax1.set_title(f"{host}: Boot Component Times") + ax1.legend() + ax1.grid(True, alpha=0.3) + ax1.xaxis.set_major_locator(ticker.MaxNLocator(integer=True)) + + # Plot 2: Total boot time with statistics + ax2 = axes[idx, 1] + ax2.plot( + boot_numbers, total_times, "b-", linewidth=2, label="Total Boot Time" + ) + + # Add statistical lines + stats = self.calculate_statistics(total_times) + if stats: + ax2.axhline( + y=stats["mean"], + color="r", + linestyle="--", + label=f"Mean: {stats['mean']:.2f}s", + ) + ax2.axhline( + y=stats["median"], + color="g", + linestyle="--", + label=f"Median: {stats['median']:.2f}s", + ) + + # Add standard deviation band + if stats["stdev"] > 0: + ax2.fill_between( + boot_numbers, + stats["mean"] - stats["stdev"], + stats["mean"] + stats["stdev"], + alpha=0.2, + color="gray", + label=f"±1 StdDev: {stats['stdev']:.2f}s", + ) + + ax2.set_xlabel("Boot Number") + ax2.set_ylabel("Time (seconds)") + ax2.set_title(f"{host}: Total Boot Time Analysis") + ax2.legend() + ax2.grid(True, alpha=0.3) + ax2.xaxis.set_major_locator(ticker.MaxNLocator(integer=True)) + + # Add text box with statistics + stats_text = f"Boots: {data['boot_count']}\n" + if stats: + stats_text += f"Min: {stats['min']:.2f}s\n" + stats_text += f"Max: {stats['max']:.2f}s\n" + stats_text += f"Range: {stats['max'] - stats['min']:.2f}s" + + ax2.text( + 0.02, + 0.98, + stats_text, + transform=ax2.transAxes, + verticalalignment="top", + bbox=dict(boxstyle="round", facecolor="wheat", alpha=0.5), + ) + + plt.tight_layout() + plt.savefig(output_file, dpi=300, bbox_inches="tight") + print(f"Saved plot to {output_file}") + + def print_summary(self): + """Print a summary of the analysis to stdout.""" + for host, data in self.hosts_data.items(): + print(f"\n{'=' * 60}") + print(f"Host: {host}") + print(f"Total boots: {data['boot_count']}") + + if data["boot_times"]: + total_times = [bt["total"] for bt in data["boot_times"]] + stats = self.calculate_statistics(total_times) + + print(f"\nBoot time statistics:") + print(f" Samples analyzed: {len(total_times)}") + if stats: + print(f" Minimum: {stats['min']:.2f}s") + print(f" Maximum: {stats['max']:.2f}s") + print(f" Mean: {stats['mean']:.2f}s") + print(f" Median: {stats['median']:.2f}s") + print(f" StdDev: {stats['stdev']:.2f}s") + print(f" Range: {stats['max'] - stats['min']:.2f}s") + + # Component breakdown + kernel_times = [bt["kernel"] for bt in data["boot_times"]] + initrd_times = [bt["initrd"] for bt in data["boot_times"]] + userspace_times = [bt["userspace"] for bt in data["boot_times"]] + + print(f"\nComponent averages:") + print(f" Kernel: {statistics.mean(kernel_times):.2f}s") + if any(t > 0 for t in initrd_times): + print(f" Initrd: {statistics.mean(initrd_times):.2f}s") + print(f" Userspace: {statistics.mean(userspace_times):.2f}s") + else: + print(" No boot time data available") + + +def main(): + parser = argparse.ArgumentParser( + description="Analyze reboot-limit workflow results" + ) + parser.add_argument( + "results_dir", + nargs="?", + default="workflows/demos/reboot-limit/results", + help="Path to results directory (default: workflows/demos/reboot-limit/results)", + ) + parser.add_argument( + "-o", + "--output", + default="reboot_limit_analysis.png", + help="Output filename for plot (default: reboot_limit_analysis.png)", + ) + parser.add_argument( + "--no-plot", action="store_true", help="Skip plotting, only show summary" + ) + + args = parser.parse_args() + + # Check if results directory exists + if not os.path.exists(args.results_dir): + print(f"Error: Results directory '{args.results_dir}' not found") + sys.exit(1) + + # Create analyzer and load data + analyzer = RebootLimitAnalyzer(args.results_dir) + analyzer.load_all_data() + + if not analyzer.hosts_data: + print(f"No host data found in '{args.results_dir}'") + print("Make sure you've run 'make reboot-limit-baseline' first") + sys.exit(1) + + # Print summary + analyzer.print_summary() + + # Generate plots + if not args.no_plot: + try: + analyzer.plot_boot_times(args.output) + except ImportError: + print("\nWarning: matplotlib not installed. Install with:") + print(" pip install matplotlib") + print("Skipping plot generation.") + + +if __name__ == "__main__": + main()
scripts/workflows/demos/reboot-limit/generate_sample_data.py new file +73 -0
73 lines not on this page · raw message
workflows/demos/reboot-limit/Makefile +11 -2
@@ -189,14 +189,23 @@ reboot-limit-dev-reset: --tags vars,reset \ --extra-vars=@./extra_vars.yaml +reboot-limit-results: + $(Q)echo "Analyzing reboot-limit results..." + $(Q)python3 scripts/workflows/demos/reboot-limit/analyze_results.py + +reboot-limit-graph: reboot-limit-results + $(Q)echo "Graph saved to reboot_limit_analysis.png" + reboot-limit-help-menu: @echo "reboot-limit options:" @echo "reboot-limit - Sets up the /data/reboot-limit directory" - @echo "reboot-limit-baseline - Run the reboot-linit test on baseline hosts and collect results" + @echo "reboot-limit-baseline - Run the reboot-limit test on baseline hosts and collect results" @echo "reboot-limit-baseline-reset - Reset the test boot counter for baseline" - @echo "reboot-limit-dev - Run the reboot-limti test on dev hosts and collect results" + @echo "reboot-limit-dev - Run the reboot-limit test on dev hosts and collect results" @echo "reboot-limit-baseline-loop - Run the reboot-limit test in a loop until failure or steady state" @echo "reboot-limit-baseline-kotd - Run the reboot-limit kotd (kernel-of-the-day) loop" + @echo "reboot-limit-results - Analyze and summarize reboot-limit test results" + @echo "reboot-limit-graph - Generate graphs from reboot-limit test results" @echo "" HELP_TARGETS += reboot-limit-help-menu
Signature · 1 line
2.47.2
PATCH 13/23 reboot-limit: save graphs in organized results/graphs directory
In reply to Luis ChamberlainUpdate the graph visualization to save output files in a more organized location under workflows/demos/reboot-limit/results/graphs/. This keeps the results directory better organized by separating raw data from generated visualizations.
The analyze_results.py script now automatically creates the graphs subdirectory if it doesn't exist, ensuring a clean directory structure for the workflow results.
--- scripts/workflows/demos/reboot-limit/analyze_results.py | 9 +++++++-- workflows/demos/reboot-limit/Makefile | 2 +- 2 files changed, 8 insertions(+), 3 deletions(-)
scripts/workflows/demos/reboot-limit/analyze_results.py +7 -2
@@ -112,6 +112,11 @@ class RebootLimitAnalyzer: print("No data to plot") return + # Ensure the output directory exists + output_path = Path(output_file) + if output_path.parent != Path("."): + output_path.parent.mkdir(parents=True, exist_ok=True) + # Create figure with subplots num_hosts = len(self.hosts_data) fig, axes = plt.subplots(num_hosts, 2, figsize=(14, 6 * num_hosts))
@@ -264,8 +269,8 @@ def main(): parser.add_argument( "-o", "--output", - default="reboot_limit_analysis.png", - help="Output filename for plot (default: reboot_limit_analysis.png)", + default="workflows/demos/reboot-limit/results/graphs/reboot_limit_analysis.png", + help="Output filename for plot (default: workflows/demos/reboot-limit/results/graphs/reboot_limit_analysis.png)", ) parser.add_argument( "--no-plot", action="store_true", help="Skip plotting, only show summary"
workflows/demos/reboot-limit/Makefile +1 -1
@@ -194,7 +194,7 @@ reboot-limit-results: $(Q)python3 scripts/workflows/demos/reboot-limit/analyze_results.py reboot-limit-graph: reboot-limit-results - $(Q)echo "Graph saved to reboot_limit_analysis.png" + $(Q)echo "Graph saved to workflows/demos/reboot-limit/results/graphs/reboot_limit_analysis.png" reboot-limit-help-menu: @echo "reboot-limit options:"
Signature · 1 line
2.47.2
PATCH 14/23 docs: add comprehensive reboot-limit workflow documentation
In reply to Luis ChamberlainAdd detailed documentation for the reboot-limit workflow including:
- Purpose and use cases for continuous reboot testing
- Configuration options and key features
- Step-by-step usage instructions
- Data collection and analysis explanation
- Example visualization showing boot performance trends
- Troubleshooting guide and best practices
- Integration with CI/CD workflows
Also includes:
- Default configuration file (defconfig-reboot-limit)
- Example graph showing typical boot time analysis
- Links in main README for discoverability
- Fixed typo in README (blktets -> blktests)
The documentation helps users understand how to use the reboot-limit workflow for system stability testing, boot performance analysis, and kernel regression detection.
--- README.md | 19 ++- defconfigs/reboot-limit | 31 ++++ docs/images/reboot_limit_example.png | Bin 0 -> 735262 bytes docs/reboot-limit.md | 223 +++++++++++++++++++++++++++ 4 files changed, 272 insertions(+), 1 deletion(-) create mode 100644 defconfigs/reboot-limit create mode 100644 docs/images/reboot_limit_example.png create mode 100644 docs/reboot-limit.md
README.md +18 -1
@@ -11,6 +11,7 @@ Table of Contents * [Start testing NFS with in 2 commands](#start-testing-nfs-with-in-2-commands) * [Runs some kernel selftests in a parallel manner](#runs-some-kernel-selftests-in-a-parallel-manner) * [CXL](#cxl) + * [reboot-limit](#reboot-limit) * [sysbench](#sysbench) * [kdevops chats](#kdevops-chats) * [kdevops on discord](#kdevops-on-discord)
@@ -222,6 +223,21 @@ to guests and create custom topologies. kdevops let you build and install the latest CXL enabled qemu version as well for you. For more details refer to [kdevops cxl docs](docs/cxl.md) +### reboot-limit + +Test system stability and boot performance with continuous reboots: + +```bash +make defconfig-reboot-limit +make +make bringup +make reboot-limit-baseline +make reboot-limit-graph +``` + +The reboot-limit workflow helps identify boot regressions, hardware issues, +and kernel stability problems. For more details see [kdevops reboot-limit docs](docs/reboot-limit.md). + ### sysbench kdevops supports automation of sysbench tests on VMs with or without
@@ -275,10 +291,11 @@ want to just use the kernel that comes with your Linux distribution. ## kdevops workflows which may be dedicated * [kdevops fstests docs](docs/fstests.md) - * [kdevops blktests docs](docs/blktets.md) + * [kdevops blktests docs](docs/blktests.md) * [kdevops CXL docs](docs/cxl.md) * [kdevops NFS docs](docs/nfs.md) * [kdevops selftests docs](docs/selftests.md) + * [kdevops reboot-limit docs](docs/reboot-limit.md) # kdevops general documentation
defconfigs/reboot-limit new file +31 -0
@@ -0,0 +1,31 @@ +# Demo defconfig for reboot-limit workflow +# +# This enables the reboot-limit workflow for system stability testing +# through continuous reboots with performance monitoring. + +# Enable reboot-limit workflow +CONFIG_WORKFLOWS_TESTS=y +CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y +CONFIG_WORKFLOWS_TESTS_DEMOS=y +CONFIG_WORKFLOWS_REBOOT_LIMIT=y + +# Enable data collection for analysis +CONFIG_REBOOT_LIMIT_ENABLE_DATA_COLLECTION=y +CONFIG_REBOOT_LIMIT_ENABLE_SYSTEMD_ANALYZE=y + +# Enable loop testing for continuous operation +CONFIG_REBOOT_LIMIT_ENABLE_LOOP=y +CONFIG_REBOOT_LIMIT_LOOP_STEADY_STATE_GOAL=100 + +# Set a reasonable number of reboots per test run +CONFIG_REBOOT_LIMIT_BOOT_MAX=100 + +# Use default reboot method (Ansible) +CONFIG_REBOOT_LIMIT_TYPE_ANSIBLE=y + +# Enable baseline and development nodes for A/B testing +CONFIG_KDEVOPS_BASELINE_AND_DEV=y + +# Storage configuration +CONFIG_LIBVIRT_EXTRA_STORAGE_DRIVE_SIZE_SET_CUSTOM=y +CONFIG_LIBVIRT_EXTRA_STORAGE_DRIVE_SIZE=20
docs/images/reboot_limit_example.png new file binary +0 -0
docs/reboot-limit.md new file +223 -0
223 lines not on this page · raw message
Signature · 1 line
2.47.2
PATCH 15/23 reboot-limit: add kexec-tools dependency installation
In reply to Luis ChamberlainAdd kexec-tools installation support for the reboot-limit workflow, following the same pattern used by fstests. This ensures kexec is properly installed across all supported distributions (Debian, Red Hat, and SUSE) when the systemctl_kexec test type is enabled.
The dependency installation is integrated into the reboot-limit role's main tasks and will be automatically executed during 'make reboot-limit'.
--- .../reboot-limit/tasks/install-deps/debian/main.yml | 10 ++++++++++ .../roles/reboot-limit/tasks/install-deps/main.yml | 12 ++++++++++++ .../reboot-limit/tasks/install-deps/redhat/main.yml | 10 ++++++++++ .../reboot-limit/tasks/install-deps/suse/main.yml | 10 ++++++++++ playbooks/roles/reboot-limit/tasks/main.yml | 4 ++++ 5 files changed, 46 insertions(+) create mode 100644 playbooks/roles/reboot-limit/tasks/install-deps/debian/main.yml create mode 100644 playbooks/roles/reboot-limit/tasks/install-deps/main.yml create mode 100644 playbooks/roles/reboot-limit/tasks/install-deps/redhat/main.yml create mode 100644 playbooks/roles/reboot-limit/tasks/install-deps/suse/main.yml
playbooks/roles/reboot-limit/tasks/install-deps/debian/main.yml new file +10 -0
@@ -0,0 +1,10 @@ +--- +- name: Install kexec-tools and dependencies for reboot-limit on Debian + become: yes + become_method: sudo + ansible.builtin.apt: + name: + - kexec-tools + state: present + update_cache: yes + tags: [ 'install', 'kexec' ]
playbooks/roles/reboot-limit/tasks/install-deps/main.yml new file +12 -0
@@ -0,0 +1,12 @@ +--- +- include_role: + name: pkg + +# tasks to install dependencies for reboot-limit +- name: reboot-limit distribution specific setup + import_tasks: tasks/install-deps/debian/main.yml + when: ansible_facts['os_family']|lower == 'debian' +- import_tasks: tasks/install-deps/suse/main.yml + when: ansible_facts['os_family']|lower == 'suse' +- import_tasks: tasks/install-deps/redhat/main.yml + when: ansible_facts['os_family']|lower == 'redhat'
playbooks/roles/reboot-limit/tasks/install-deps/redhat/main.yml new file +10 -0
@@ -0,0 +1,10 @@ +--- +- name: Install kexec-tools and dependencies for reboot-limit on Red Hat + become: yes + become_method: sudo + ansible.builtin.yum: + name: + - kexec-tools + state: present + update_cache: yes + tags: [ 'install', 'kexec' ]
playbooks/roles/reboot-limit/tasks/install-deps/suse/main.yml new file +10 -0
@@ -0,0 +1,10 @@ +--- +- name: Install kexec-tools and dependencies for reboot-limit on SUSE + become: yes + become_method: sudo + ansible.builtin.zypper: + name: + - kexec-tools + state: present + update_cache: yes + tags: [ 'install', 'kexec' ]
playbooks/roles/reboot-limit/tasks/main.yml +4 -0
@@ -14,6 +14,10 @@ name: create_data_partition tags: [ 'data_partition' ] +- name: Install dependencies for reboot-limit + include_tasks: install-deps/main.yml + tags: [ 'install', 'deps' ] + - name: Create the reboot-limit data collection directory for each host become: yes become_method: sudo
Signature · 1 line
2.47.2
PATCH 16/23 reboot-limit: add A/B testing support targets
In reply to Luis ChamberlainAdd missing targets to support A/B testing workflow:
- reboot-limit-tests: runs tests on both baseline and dev groups
- reboot-limit-dev: runs tests on dev group only
Rename the confusing reboot-limit-dev-baseline target to just reboot-limit-dev. The "dev-baseline" naming was nonsensical as it mixed two opposing concepts - "dev" refers to development/testing nodes while "baseline" refers to the control group. Following the pattern of other workflows like fstests, the dev target should simply be named reboot-limit-dev.
Also updated help menu to document all available targets including the previously undocumented dev-loop and dev-kotd targets.
This enables proper A/B testing with linux-ab workflow where users can test different kernels on baseline and dev groups simultaneously.
--- workflows/demos/reboot-limit/Makefile | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
workflows/demos/reboot-limit/Makefile +7 -1
@@ -163,8 +163,11 @@ reboot-limit-baseline-reset: --tags vars,reset \ --extra-vars=@./extra_vars.yaml +# This runs tests on both baseline and dev groups - useful for A/B testing +reboot-limit-tests: reboot-limit-baseline reboot-limit-dev + # Below are the corresponding dev targets -reboot-limit-dev-baseline: +reboot-limit-dev: $(Q)ansible-playbook $(ANSIBLE_VERBOSE) \ --limit 'dev' \ playbooks/reboot-limit.yml \
@@ -202,8 +205,11 @@ reboot-limit-help-menu: @echo "reboot-limit-baseline - Run the reboot-limit test on baseline hosts and collect results" @echo "reboot-limit-baseline-reset - Reset the test boot counter for baseline" @echo "reboot-limit-dev - Run the reboot-limit test on dev hosts and collect results" + @echo "reboot-limit-tests - Run the reboot-limit test on both baseline and dev hosts (A/B testing)" @echo "reboot-limit-baseline-loop - Run the reboot-limit test in a loop until failure or steady state" @echo "reboot-limit-baseline-kotd - Run the reboot-limit kotd (kernel-of-the-day) loop" + @echo "reboot-limit-dev-loop - Run the reboot-limit test in a loop on dev hosts" + @echo "reboot-limit-dev-kotd - Run the reboot-limit kotd loop on dev hosts" @echo "reboot-limit-results - Analyze and summarize reboot-limit test results" @echo "reboot-limit-graph - Generate graphs from reboot-limit test results" @echo ""
Signature · 1 line
2.47.2
PATCH 17/23 reboot-limit: fix kexec and reboot connection handling
In reply to Luis ChamberlainReplace shell-based kexec implementation with proper Ansible tasks and fix connection handling for both systemctl reboot and kexec.
For kexec:
- Break down into discrete tasks to find kernel and initrd files
- Use stat module to check multiple possible locations
- Load kernel with kexec -l before executing systemctl kexec
- Use async execution to avoid connection failures
For both reboot types:
- Use async: 1 and poll: 0 to fire-and-forget the reboot command
- Add proper wait_for tasks to handle connection loss and recovery
- Wait for system to go down then come back up
This approach is more idiomatic Ansible and handles the SSH connection properly during system reboots.
--- .../roles/reboot-limit/tasks/do-reboot.yml | 126 ++++++++++++++++++ 1 file changed, 126 insertions(+)
playbooks/roles/reboot-limit/tasks/do-reboot.yml +126 -0
@@ -21,14 +21,140 @@ become: yes become_method: sudo command: "systemctl reboot" + async: 1 + poll: 0 when: - reboot_limit_test_type == "systemctl_reboot" tags: [ 'run_tests' ] +- name: Wait for system to go down for reboot + wait_for: + host: "{{ ansible_host }}" + port: 22 + state: stopped + delay: 5 + timeout: 60 + delegate_to: localhost + when: + - reboot_limit_test_type == "systemctl_reboot" + tags: [ 'run_tests' ] + +- name: Wait for system to come back after reboot + wait_for: + host: "{{ ansible_host }}" + port: 22 + state: started + delay: 10 + timeout: 300 + delegate_to: localhost + when: + - reboot_limit_test_type == "systemctl_reboot" + tags: [ 'run_tests' ] + +# kexec tasks - prepare the kernel for kexec before rebooting +- name: Get current kernel version for kexec + command: uname -r + register: current_kernel_version + when: + - reboot_limit_test_type == "systemctl_kexec" + tags: [ 'run_tests' ] + +- name: Check for kernel image locations for kexec + stat: + path: "{{ item }}" + register: kernel_stat + with_items: + - "/boot/vmlinuz-{{ current_kernel_version.stdout }}" + - "/boot/vmlinux-{{ current_kernel_version.stdout }}" + - "/boot/kernel-{{ current_kernel_version.stdout }}" + when: + - reboot_limit_test_type == "systemctl_kexec" + - current_kernel_version is defined + tags: [ 'run_tests' ] + +- name: Set kernel path for kexec + set_fact: + kexec_kernel_path: "{{ item.stat.path }}" + with_items: "{{ kernel_stat.results }}" + when: + - reboot_limit_test_type == "systemctl_kexec" + - item.stat.exists + tags: [ 'run_tests' ] + +- name: Check for initrd/initramfs locations for kexec + stat: + path: "{{ item }}" + register: initrd_stat + with_items: + - "/boot/initrd.img-{{ current_kernel_version.stdout }}" + - "/boot/initramfs-{{ current_kernel_version.stdout }}.img" + - "/boot/initrd-{{ current_kernel_version.stdout }}" + - "/boot/initrd-{{ current_kernel_version.stdout }}.img" + when: + - reboot_limit_test_type == "systemctl_kexec" + - current_kernel_version is defined + tags: [ 'run_tests' ] + +- name: Set initrd path for kexec + set_fact: + kexec_initrd_path: "{{ item.stat.path }}" + with_items: "{{ initrd_stat.results }}" + when: + - reboot_limit_test_type == "systemctl_kexec" + - item.stat.exists + tags: [ 'run_tests' ] + +- name: Read current kernel command line for kexec + slurp: + src: /proc/cmdline + register: cmdline_content + when: + - reboot_limit_test_type == "systemctl_kexec" + tags: [ 'run_tests' ] + +- name: Load kernel into kexec + become: yes + become_method: sudo + command: > + kexec -l {{ kexec_kernel_path }} + --initrd={{ kexec_initrd_path }} + --command-line="{{ cmdline_content.content | b64decode | trim }}" + when: + - reboot_limit_test_type == "systemctl_kexec" + - kexec_kernel_path is defined + - kexec_initrd_path is defined + tags: [ 'run_tests' ] + - name: Run the reboot test using systemctl kexec become: yes become_method: sudo command: "systemctl kexec" + async: 1 + poll: 0 + when: + - reboot_limit_test_type == "systemctl_kexec" + tags: [ 'run_tests' ] + +- name: Wait for system to go down for kexec + wait_for: + host: "{{ ansible_host }}" + port: 22 + state: stopped + delay: 5 + timeout: 60 + delegate_to: localhost + when: + - reboot_limit_test_type == "systemctl_kexec" + tags: [ 'run_tests' ] + +- name: Wait for system to come back after kexec + wait_for: + host: "{{ ansible_host }}" + port: 22 + state: started + delay: 10 + timeout: 300 + delegate_to: localhost when: - reboot_limit_test_type == "systemctl_kexec" tags: [ 'run_tests' ]
Signature · 1 line
2.47.2
PATCH 18/23 reboot-limit: add COUNT parameter to override reboot count
In reply to Luis ChamberlainAdd support for overriding the reboot count at runtime using the COUNT parameter. This allows for quick testing without modifying the configuration:
make reboot-limit-tests COUNT=2
This is useful for development and testing when you want to quickly verify the workflow with a small number of reboots instead of the configured maximum.
The COUNT parameter works with all reboot-limit targets that respect the reboot_limit_max variable.
--- workflows/demos/reboot-limit/Makefile | 8 ++++++++ 1 file changed, 8 insertions(+)
workflows/demos/reboot-limit/Makefile +8 -0
@@ -57,7 +57,12 @@ endif # it works OK. Note that if enabling kernel-ci the steady state there # is separate. And so if this reboot_limit_max is 100 and steady state # is 100, we'd have to reboot 10,000 times before reaching steady state. +# The COUNT parameter can override this value at runtime. +ifdef COUNT +REBOOT_LIMIT_MAX:=$(COUNT) +else REBOOT_LIMIT_MAX:=$(subst ",,$(CONFIG_REBOOT_LIMIT_BOOT_MAX)) +endif REBOOT_LIMIT_ARGS += reboot_limit_max='$(REBOOT_LIMIT_MAX)' ifeq (y,$(CONFIG_REBOOT_LIMIT_BOOT_COUNT_CRASH_ENABLE))
@@ -213,5 +218,8 @@ reboot-limit-help-menu: @echo "reboot-limit-results - Analyze and summarize reboot-limit test results" @echo "reboot-limit-graph - Generate graphs from reboot-limit test results" @echo "" + @echo "Optional parameters:" + @echo " COUNT=N - Override the number of reboots (e.g., make reboot-limit-tests COUNT=2)" + @echo "" HELP_TARGETS += reboot-limit-help-menu
Signature · 1 line
2.47.2
PATCH 19/23 reboot-limit: fix wait_for tasks using wrong host reference
In reply to Luis ChamberlainThe wait_for tasks were failing because they were trying to connect to localhost:22 instead of the actual target host when delegated to localhost.
Fix by using a proper fallback chain for the host reference:
- ansible_ssh_host (preferred, used elsewhere in the role)
- ansible_host (fallback)
- inventory_hostname (last resort)
This ensures the wait_for module connects to the correct host IP/name when checking if the system has gone down or come back up after reboot.
--- playbooks/roles/reboot-limit/tasks/do-reboot.yml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
playbooks/roles/reboot-limit/tasks/do-reboot.yml +4 -4
@@ -29,7 +29,7 @@ - name: Wait for system to go down for reboot wait_for: - host: "{{ ansible_host }}" + host: "{{ ansible_ssh_host | default(ansible_host) | default(inventory_hostname) }}" port: 22 state: stopped delay: 5
@@ -41,7 +41,7 @@ - name: Wait for system to come back after reboot wait_for: - host: "{{ ansible_host }}" + host: "{{ ansible_ssh_host | default(ansible_host) | default(inventory_hostname) }}" port: 22 state: started delay: 10
@@ -137,7 +137,7 @@ - name: Wait for system to go down for kexec wait_for: - host: "{{ ansible_host }}" + host: "{{ ansible_ssh_host | default(ansible_host) | default(inventory_hostname) }}" port: 22 state: stopped delay: 5
@@ -149,7 +149,7 @@ - name: Wait for system to come back after kexec wait_for: - host: "{{ ansible_host }}" + host: "{{ ansible_ssh_host | default(ansible_host) | default(inventory_hostname) }}" port: 22 state: started delay: 10
Signature · 1 line
2.47.2
PATCH 20/23 reboot-limit: use ansible reboot module for all reboot types
In reply to Luis ChamberlainReplace the complex async command execution and wait_for logic with the ansible reboot module, which properly handles SSH connections during system reboots.
The reboot module:
- Supports custom reboot commands via reboot_command parameter
- Handles connection loss and recovery automatically
- Works correctly with kdevops SSH configuration
- Avoids the localhost host resolution issues with wait_for
This simplifies the code and makes all three reboot methods (ansible, systemctl reboot, systemctl kexec) use the same robust reboot handling mechanism.
--- .../roles/reboot-limit/tasks/do-reboot.yml | 68 ++++--------------- 1 file changed, 12 insertions(+), 56 deletions(-)
playbooks/roles/reboot-limit/tasks/do-reboot.yml +12 -56
@@ -17,36 +17,14 @@ - reboot_limit_test_type == "ansible" tags: [ 'run_tests' ] -- name: Run the reboot test using systemctl reboot +- name: Reboot using systemctl reboot with proper handling become: yes become_method: sudo - command: "systemctl reboot" - async: 1 - poll: 0 - when: - - reboot_limit_test_type == "systemctl_reboot" - tags: [ 'run_tests' ] - -- name: Wait for system to go down for reboot - wait_for: - host: "{{ ansible_ssh_host | default(ansible_host) | default(inventory_hostname) }}" - port: 22 - state: stopped - delay: 5 - timeout: 60 - delegate_to: localhost - when: - - reboot_limit_test_type == "systemctl_reboot" - tags: [ 'run_tests' ] - -- name: Wait for system to come back after reboot - wait_for: - host: "{{ ansible_ssh_host | default(ansible_host) | default(inventory_hostname) }}" - port: 22 - state: started - delay: 10 - timeout: 300 - delegate_to: localhost + reboot: + msg: "Rebooting system via systemctl reboot for reboot-limit test" + reboot_command: "systemctl reboot" + post_reboot_delay: 10 + reboot_timeout: 300 when: - reboot_limit_test_type == "systemctl_reboot" tags: [ 'run_tests' ]
@@ -125,36 +103,14 @@ - kexec_initrd_path is defined tags: [ 'run_tests' ] -- name: Run the reboot test using systemctl kexec +- name: Reboot using systemctl kexec with proper handling become: yes become_method: sudo - command: "systemctl kexec" - async: 1 - poll: 0 - when: - - reboot_limit_test_type == "systemctl_kexec" - tags: [ 'run_tests' ] - -- name: Wait for system to go down for kexec - wait_for: - host: "{{ ansible_ssh_host | default(ansible_host) | default(inventory_hostname) }}" - port: 22 - state: stopped - delay: 5 - timeout: 60 - delegate_to: localhost - when: - - reboot_limit_test_type == "systemctl_kexec" - tags: [ 'run_tests' ] - -- name: Wait for system to come back after kexec - wait_for: - host: "{{ ansible_ssh_host | default(ansible_host) | default(inventory_hostname) }}" - port: 22 - state: started - delay: 10 - timeout: 300 - delegate_to: localhost + reboot: + msg: "Rebooting system via systemctl kexec for reboot-limit test" + reboot_command: "systemctl kexec" + post_reboot_delay: 10 + reboot_timeout: 300 when: - reboot_limit_test_type == "systemctl_kexec" tags: [ 'run_tests' ]
Signature · 1 line
2.47.2
PATCH 21/23 reboot-limit: fix COUNT parameter to properly override reboot count
In reply to Luis ChamberlainThe COUNT parameter was not being passed to ansible-playbook, causing it to always use the configured value from extra_vars.yaml instead of the runtime override.
Fix by:
- Adding $(KDEVOPS_EXTRA_VARS) dependency to ensure extra_vars.yaml exists
- Passing reboot_limit_max as an additional --extra-vars parameter which overrides the value from extra_vars.yaml
This ensures that make reboot-limit-tests COUNT=2 properly limits the test to 2 reboots instead of using the configured maximum.
--- workflows/demos/reboot-limit/Makefile | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-)
workflows/demos/reboot-limit/Makefile +13 -8
@@ -107,10 +107,11 @@ endif # CONFIG_REBOOT_LIMIT_ENABLE_LOOP # # The playbook reboot-limit.yml is used to run all tasks defined except those # with tags "run_tests" or "copy_results". -reboot-limit: +reboot-limit: $(KDEVOPS_EXTRA_VARS) $(Q)ansible-playbook $(ANSIBLE_VERBOSE) \ playbooks/reboot-limit.yml \ - --skip-tags run_tests,copy_results + --skip-tags run_tests,copy_results \ + --extra-vars="reboot_limit_max=$(REBOOT_LIMIT_MAX)" # kdevops supports generic workflows such as the enabling you to then use the # /data/ partition to git clone linux, compile linux, install it and reboot
@@ -137,17 +138,19 @@ reboot-limit: # tags "first_run" first and then "run_tests" and "copy_results". We always # use the vars tag to ensure we get variables defined which we construct on the # fly. -reboot-limit-baseline: +reboot-limit-baseline: $(KDEVOPS_EXTRA_VARS) $(Q)ansible-playbook $(ANSIBLE_VERBOSE) \ --limit 'baseline' \ playbooks/reboot-limit.yml \ --tags vars,first_run,reset \ - --extra-vars=@./extra_vars.yaml + --extra-vars=@./extra_vars.yaml \ + --extra-vars="reboot_limit_max=$(REBOOT_LIMIT_MAX)" $(Q)ansible-playbook $(ANSIBLE_VERBOSE) \ --limit 'baseline' \ playbooks/reboot-limit.yml \ --tags vars,run_tests,copy_results \ - --extra-vars=@./extra_vars.yaml + --extra-vars=@./extra_vars.yaml \ + --extra-vars="reboot_limit_max=$(REBOOT_LIMIT_MAX)" # Once you know the baseline works, you may want to run the baseline in a loop # up to the number of times you define. We define this goal in kdevops as
@@ -172,17 +175,19 @@ reboot-limit-baseline-reset: reboot-limit-tests: reboot-limit-baseline reboot-limit-dev # Below are the corresponding dev targets -reboot-limit-dev: +reboot-limit-dev: $(KDEVOPS_EXTRA_VARS) $(Q)ansible-playbook $(ANSIBLE_VERBOSE) \ --limit 'dev' \ playbooks/reboot-limit.yml \ --tags vars,first_run,reset \ - --extra-vars=@./extra_vars.yaml + --extra-vars=@./extra_vars.yaml \ + --extra-vars="reboot_limit_max=$(REBOOT_LIMIT_MAX)" $(Q)ansible-playbook $(ANSIBLE_VERBOSE) \ --limit 'dev' \ playbooks/reboot-limit.yml \ --tags vars,run_tests,copy_results \ - --extra-vars=@./extra_vars.yaml + --extra-vars=@./extra_vars.yaml \ + --extra-vars="reboot_limit_max=$(REBOOT_LIMIT_MAX)" reboot-limit-dev-loop: $(Q)$(REBOOT_LIMIT_LOOP) dev
Signature · 1 line
2.47.2
PATCH 22/23 reboot-limit: handle empty dev group gracefully
In reply to Luis ChamberlainThe reboot-limit-dev target was failing when the dev group had no hosts, which is a common configuration when not using A/B testing.
Fix by checking if the dev group has any hosts before attempting to run ansible-playbook. This allows reboot-limit-tests to work correctly even when only baseline hosts are configured.
The check uses a simple grep to see if there are any hostnames listed after the [dev] group header in the hosts file.
--- workflows/demos/reboot-limit/Makefile | 29 ++++++++++++++++----------- 1 file changed, 17 insertions(+), 12 deletions(-)
workflows/demos/reboot-limit/Makefile +17 -12
@@ -175,19 +175,24 @@ reboot-limit-baseline-reset: reboot-limit-tests: reboot-limit-baseline reboot-limit-dev # Below are the corresponding dev targets +# Check if dev group has hosts before running reboot-limit-dev: $(KDEVOPS_EXTRA_VARS) - $(Q)ansible-playbook $(ANSIBLE_VERBOSE) \ - --limit 'dev' \ - playbooks/reboot-limit.yml \ - --tags vars,first_run,reset \ - --extra-vars=@./extra_vars.yaml \ - --extra-vars="reboot_limit_max=$(REBOOT_LIMIT_MAX)" - $(Q)ansible-playbook $(ANSIBLE_VERBOSE) \ - --limit 'dev' \ - playbooks/reboot-limit.yml \ - --tags vars,run_tests,copy_results \ - --extra-vars=@./extra_vars.yaml \ - --extra-vars="reboot_limit_max=$(REBOOT_LIMIT_MAX)" + $(Q)if grep -A1 '^\[dev\]' hosts | grep -qE '^[a-zA-Z0-9]'; then \ + ansible-playbook $(ANSIBLE_VERBOSE) \ + --limit 'dev' \ + playbooks/reboot-limit.yml \ + --tags vars,first_run,reset \ + --extra-vars=@./extra_vars.yaml \ + --extra-vars="reboot_limit_max=$(REBOOT_LIMIT_MAX)" && \ + ansible-playbook $(ANSIBLE_VERBOSE) \ + --limit 'dev' \ + playbooks/reboot-limit.yml \ + --tags vars,run_tests,copy_results \ + --extra-vars=@./extra_vars.yaml \ + --extra-vars="reboot_limit_max=$(REBOOT_LIMIT_MAX)"; \ + else \ + echo "No hosts found in dev group, skipping dev tests"; \ + fi reboot-limit-dev-loop: $(Q)$(REBOOT_LIMIT_LOOP) dev
Signature · 1 line
2.47.2
PATCH 23/23 reboot-limit: add kexec comparison feature
In reply to Luis ChamberlainAdd comprehensive support for comparing regular reboots vs kexec reboots in the reboot-limit workflow. This enables users to test both reboot types sequentially and compare their performance characteristics.
New defconfigs:
- reboot-limit-kexec: Configure for kexec-only testing
- reboot-limit-compare: Configure for comparison mode testing
New Kconfig options:
- REBOOT_LIMIT_TYPE_COMPARE_BOTH: Test both regular and kexec reboots
- REBOOT_LIMIT_DATA_REGULAR/KEXEC: Separate data paths for comparison
Workflow enhancements:
- Sequential testing: regular reboot first, then kexec
- Separate statistics collection for each reboot type
- Enhanced data organization with regular/ and kexec/ subdirectories
Analysis improvements:
- Automatic comparison mode detection
- Side-by-side performance analysis
- Speedup calculation and visualization
- Comprehensive comparative graphs showing both boot types
Usage:
make defconfig-reboot-limit-compare
make bringup
make reboot-limit-tests
make reboot-limit-graph
--- defconfigs/reboot-limit-compare | 36 ++ defconfigs/reboot-limit-kexec | 32 ++ playbooks/roles/gen_hosts/tasks/main.yml | 13 + playbooks/roles/gen_hosts/templates/hosts.j2 | 55 ++- playbooks/roles/gen_nodes/tasks/main.yml | 32 ++ .../reboot-limit/tasks/do-reboot-compare.yml | 126 +++++++ .../roles/reboot-limit/tasks/do-reboot.yml | 28 +- .../reboot-limit/tasks/handle-reboot-data.yml | 94 ++++++ playbooks/roles/reboot-limit/tasks/main.yml | 84 ++++- .../demos/reboot-limit/analyze_results.py | 314 +++++++++++++++++- workflows/demos/reboot-limit/Kconfig | 31 ++ workflows/demos/reboot-limit/Makefile | 13 + 12 files changed, 828 insertions(+), 30 deletions(-) create mode 100644 defconfigs/reboot-limit-compare create mode 100644 defconfigs/reboot-limit-kexec create mode 100644 playbooks/roles/reboot-limit/tasks/do-reboot-compare.yml create mode 100644 playbooks/roles/reboot-limit/tasks/handle-reboot-data.yml
defconfigs/reboot-limit-compare new file +36 -0
@@ -0,0 +1,36 @@ +# Demo defconfig for reboot-limit workflow with comparison testing +# +# This enables the reboot-limit workflow to compare regular reboots +# vs kexec reboots with separate performance monitoring and analysis. + +# Enable workflows and reboot-limit workflow +CONFIG_WORKFLOWS=y +CONFIG_WORKFLOWS_TESTS=y +CONFIG_WORKFLOWS_TESTS_DEMOS=y +CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y +CONFIG_WORKFLOWS_REBOOT_LIMIT=y + +# Enable baseline and dev testing +CONFIG_KDEVOPS_BASELINE_AND_DEV=y + +# Enable data collection for analysis +CONFIG_REBOOT_LIMIT_ENABLE_DATA_COLLECTION=y +CONFIG_REBOOT_LIMIT_ENABLE_SYSTEMD_ANALYZE=y + +# Enable loop testing for continuous operation +CONFIG_REBOOT_LIMIT_ENABLE_LOOP=y +CONFIG_REBOOT_LIMIT_LOOP_STEADY_STATE_GOAL=100 + +# Set a reasonable number of reboots per test run +CONFIG_REBOOT_LIMIT_BOOT_MAX=100 + +# Use comparison mode to test both regular and kexec reboots +CONFIG_REBOOT_LIMIT_TYPE_COMPARE_BOTH=y + +# We'll build our kernel +CONFIG_WORKFLOW_LINUX_CUSTOM=y +CONFIG_BOOTLINUX=y +CONFIG_BOOTLINUX_9P=y + +# Enable A/B testing with different kernel references +CONFIG_BOOTLINUX_AB_DIFFERENT_REF=y
defconfigs/reboot-limit-kexec new file +32 -0
@@ -0,0 +1,32 @@ +# Demo defconfig for reboot-limit workflow with kexec +# +# This enables the reboot-limit workflow for system stability testing +# through continuous kexec reboots with performance monitoring. + +# Enable workflows and reboot-limit workflow +CONFIG_WORKFLOWS=y +CONFIG_WORKFLOWS_TESTS=y +CONFIG_WORKFLOWS_TESTS_DEMOS=y +CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y +CONFIG_WORKFLOWS_REBOOT_LIMIT=y + +# Enable data collection for analysis +CONFIG_REBOOT_LIMIT_ENABLE_DATA_COLLECTION=y +CONFIG_REBOOT_LIMIT_ENABLE_SYSTEMD_ANALYZE=y + +# Enable loop testing for continuous operation +CONFIG_REBOOT_LIMIT_ENABLE_LOOP=y +CONFIG_REBOOT_LIMIT_LOOP_STEADY_STATE_GOAL=100 + +# Set a reasonable number of reboots per test run +CONFIG_REBOOT_LIMIT_BOOT_MAX=100 + +# Use kexec reboot method for faster rebooting +CONFIG_REBOOT_LIMIT_TYPE_SYSTEMD_KEXEC=y + +# Enable baseline and development nodes for A/B testing +CONFIG_KDEVOPS_BASELINE_AND_DEV=y + +# Storage configuration +CONFIG_LIBVIRT_EXTRA_STORAGE_DRIVE_SIZE_SET_CUSTOM=y +CONFIG_LIBVIRT_EXTRA_STORAGE_DRIVE_SIZE=20
playbooks/roles/gen_hosts/tasks/main.yml +13 -0
@@ -351,6 +351,19 @@ - kdevops_workflow_enable_mmtests - ansible_hosts_template.stat.exists +- name: Generate the Ansible hosts file for a dedicated reboot-limit setup + tags: [ 'hosts' ] + template: + src: "{{ kdevops_hosts_template }}" + dest: "{{ ansible_cfg_inventory }}" + force: yes + trim_blocks: True + lstrip_blocks: True + when: + - kdevops_workflows_dedicated_workflow + - workflows_reboot_limit + - ansible_hosts_template.stat.exists + - name: Verify if final host file exists stat: path: "{{ ansible_cfg_inventory }}"
playbooks/roles/gen_hosts/templates/hosts.j2 +45 -10
97 lines not on this page · raw message
playbooks/roles/gen_nodes/tasks/main.yml +32 -0
@@ -558,6 +558,38 @@ - kdevops_workflow_enable_mmtests - ansible_nodes_template.stat.exists +- name: Generate the reboot-limit kdevops nodes file using {{ kdevops_nodes_template }} as jinja2 source template + tags: [ 'hosts' ] + vars: + node_template: "{{ kdevops_nodes_template | basename }}" + nodes: "{{ [kdevops_host_prefix + '-reboot-limit'] }}" + all_generic_nodes: "{{ [kdevops_host_prefix + '-reboot-limit'] }}" + template: + src: "{{ node_template }}" + dest: "{{ topdir_path }}/{{ kdevops_nodes }}" + force: yes + when: + - kdevops_workflows_dedicated_workflow + - workflows_reboot_limit + - ansible_nodes_template.stat.exists + - not kdevops_baseline_and_dev + +- name: Generate the reboot-limit kdevops nodes file with dev hosts using {{ kdevops_nodes_template }} as jinja2 source template + tags: [ 'hosts' ] + vars: + node_template: "{{ kdevops_nodes_template | basename }}" + nodes: "{{ [kdevops_host_prefix + '-reboot-limit', kdevops_host_prefix + '-reboot-limit-dev'] }}" + all_generic_nodes: "{{ [kdevops_host_prefix + '-reboot-limit', kdevops_host_prefix + '-reboot-limit-dev'] }}" + template: + src: "{{ node_template }}" + dest: "{{ topdir_path }}/{{ kdevops_nodes }}" + force: yes + when: + - kdevops_workflows_dedicated_workflow + - workflows_reboot_limit + - ansible_nodes_template.stat.exists + - kdevops_baseline_and_dev + - name: Get the control host's timezone ansible.builtin.command: "timedatectl show -p Timezone --value" register: kdevops_host_timezone
playbooks/roles/reboot-limit/tasks/do-reboot-compare.yml new file +126 -0
126 lines not on this page · raw message
playbooks/roles/reboot-limit/tasks/do-reboot.yml +18 -10
@@ -39,12 +39,14 @@ - name: Check for kernel image locations for kexec stat: - path: "{{ item }}" + path: "{{ kernel_path }}" register: kernel_stat - with_items: + loop: - "/boot/vmlinuz-{{ current_kernel_version.stdout }}" - "/boot/vmlinux-{{ current_kernel_version.stdout }}" - "/boot/kernel-{{ current_kernel_version.stdout }}" + loop_control: + loop_var: kernel_path when: - reboot_limit_test_type == "systemctl_kexec" - current_kernel_version is defined
@@ -52,22 +54,26 @@ - name: Set kernel path for kexec set_fact: - kexec_kernel_path: "{{ item.stat.path }}" - with_items: "{{ kernel_stat.results }}" + kexec_kernel_path: "{{ kernel_item.stat.path }}" + loop: "{{ kernel_stat.results }}" + loop_control: + loop_var: kernel_item when: - reboot_limit_test_type == "systemctl_kexec" - - item.stat.exists + - kernel_item.stat.exists tags: [ 'run_tests' ] - name: Check for initrd/initramfs locations for kexec stat: - path: "{{ item }}" + path: "{{ initrd_path }}" register: initrd_stat - with_items: + loop: - "/boot/initrd.img-{{ current_kernel_version.stdout }}" - "/boot/initramfs-{{ current_kernel_version.stdout }}.img" - "/boot/initrd-{{ current_kernel_version.stdout }}" - "/boot/initrd-{{ current_kernel_version.stdout }}.img" + loop_control: + loop_var: initrd_path when: - reboot_limit_test_type == "systemctl_kexec" - current_kernel_version is defined
@@ -75,11 +81,13 @@ - name: Set initrd path for kexec set_fact: - kexec_initrd_path: "{{ item.stat.path }}" - with_items: "{{ initrd_stat.results }}" + kexec_initrd_path: "{{ initrd_item.stat.path }}" + loop: "{{ initrd_stat.results }}" + loop_control: + loop_var: initrd_item when: - reboot_limit_test_type == "systemctl_kexec" - - item.stat.exists + - initrd_item.stat.exists tags: [ 'run_tests' ] - name: Read current kernel command line for kexec
playbooks/roles/reboot-limit/tasks/handle-reboot-data.yml new file +94 -0
94 lines not on this page · raw message
playbooks/roles/reboot-limit/tasks/main.yml +81 -3
115 lines not on this page · raw message
scripts/workflows/demos/reboot-limit/analyze_results.py +307 -7
353 lines not on this page · raw message
workflows/demos/reboot-limit/Kconfig +31 -0
@@ -32,6 +32,14 @@ config REBOOT_LIMIT_TYPE_SYSTEMD_KEXEC help This will try to reboot instead using systemctl kexec. +config REBOOT_LIMIT_TYPE_COMPARE_BOTH + bool "Compare regular reboot vs kexec" + help + This will test both regular reboot (ansible) and kexec reboot + sequentially to compare their performance characteristics. + Regular reboot will be tested first, followed by kexec reboot. + Statistics will be collected separately for each reboot type. + endchoice config REBOOT_LIMIT_TEST_TYPE
@@ -39,6 +47,13 @@ config REBOOT_LIMIT_TEST_TYPE default "ansible" if REBOOT_LIMIT_TYPE_ANSIBLE default "systemctl_reboot" if REBOOT_LIMIT_TYPE_SYSTEMD_REBOOT default "systemctl_kexec" if REBOOT_LIMIT_TYPE_SYSTEMD_KEXEC + default "compare_both" if REBOOT_LIMIT_TYPE_COMPARE_BOTH + +config REBOOT_LIMIT_COMPARE_BOTH_ENABLED + bool + default y if REBOOT_LIMIT_TYPE_COMPARE_BOTH + help + Internal configuration variable to track when comparison mode is enabled. config REBOOT_LIMIT_BOOT_MAX int "How many reboots should we do to consider reboots OK?"
@@ -119,6 +134,22 @@ config REBOOT_LIMIT_DATA of each node. Note that {{data_path}} corresponds to the location set by the configuration option CONFIG_WORKFLOW_DATA_PATH. +config REBOOT_LIMIT_DATA_REGULAR + string "Where to place regular reboot statistics in comparison mode" + default "{{data_path}}/reboot-limit/regular" + depends on REBOOT_LIMIT_COMPARE_BOTH_ENABLED + help + This is the target location for regular (ansible) reboot statistics + when running in comparison mode. + +config REBOOT_LIMIT_DATA_KEXEC + string "Where to place kexec reboot statistics in comparison mode" + default "{{data_path}}/reboot-limit/kexec" + depends on REBOOT_LIMIT_COMPARE_BOTH_ENABLED + help + This is the target location for kexec reboot statistics + when running in comparison mode. + config REBOOT_LIMIT_ENABLE_SYSTEMD_ANALYZE bool "Enable data collection of systemd-analyze results" default y
workflows/demos/reboot-limit/Makefile +13 -0
@@ -47,6 +47,14 @@ ifeq (y,$(CONFIG_REBOOT_LIMIT_ENABLE_DATA_COLLECTION)) REBOOT_LIMIT_DATA :=$(subst ",,$(CONFIG_REBOOT_LIMIT_DATA)) REBOOT_LIMIT_ARGS += reboot_limit_data=\"$(REBOOT_LIMIT_DATA)\" +ifeq (y,$(CONFIG_REBOOT_LIMIT_COMPARE_BOTH_ENABLED)) +REBOOT_LIMIT_DATA_REGULAR :=$(subst ",,$(CONFIG_REBOOT_LIMIT_DATA_REGULAR)) +REBOOT_LIMIT_DATA_KEXEC :=$(subst ",,$(CONFIG_REBOOT_LIMIT_DATA_KEXEC)) +REBOOT_LIMIT_ARGS += reboot_limit_data_regular=\"$(REBOOT_LIMIT_DATA_REGULAR)\" +REBOOT_LIMIT_ARGS += reboot_limit_data_kexec=\"$(REBOOT_LIMIT_DATA_KEXEC)\" +REBOOT_LIMIT_ARGS += reboot_limit_compare_both_enabled=True +endif + # This is an example of how to map a boolean from kconfig into ansible ifeq (y,$(CONFIG_REBOOT_LIMIT_ENABLE_SYSTEMD_ANALYZE)) REBOOT_LIMIT_ARGS += reboot_limit_enable_systemd_analyze=True
@@ -75,6 +83,11 @@ endif # the extra_vars.yaml file. WORKFLOW_ARGS += $(REBOOT_LIMIT_ARGS) +# Export workflow enabled flag for ansible +ifeq (y,$(CONFIG_WORKFLOWS_REBOOT_LIMIT)) +WORKFLOW_ARGS += workflows_reboot_limit=True +endif + # The default is our workflow does not have loop testing enabled REBOOT_LIMIT_LOOP := false REBOOT_LIMIT_LOOP_KOTD := false
Signature · 1 line
2.47.2
Quoted 126 lines · Luis Chamberlain
The
CONFIG_KERNEL_CIthing was used back in the hay day before we adopted fstestsSOAK_DURATION. AlthoughCONFIG_KERNEL_CIproved useful to test fstests in a loop -- the only reasonable way to scale is to just embraceSOAK_DURATIONand extend any twitches we need within fstests.However only one workflow uses
CONFIG_KERNEL_CIstill. So move that functionality directly into the reboot-limit workflow which uses it. While at it, as a drive by, let's enhance the reboot limit workflow with kexec and also graphing capabilities allowing us to do A/B testing for reboots and kexec regression testings.The rationale for this work si to help fix any misunderatandings on what kdevops's kernel CI support consists of, and
CONFIG_KERNEL_CIcertainly was not it. It's evolved now to just the kdevops kernel-ci repo and the the .ci/ symlinks, along with optional kernel-kpd trees for subsystems.Luis Chamberlain (23):
fstests: removeCONFIG_KERNEL_CIsupport
fstests: remove kernel-ci script symlinks
blktests: removeCONFIG_KERNEL_CIsupport
gitr: removeCONFIG_KERNEL_CIsupport
ltp: removeCONFIG_KERNEL_CIsupport
nfstest: removeCONFIG_KERNEL_CIsupport
pynfs: removeCONFIG_KERNEL_CIsupport
reboot-limit: convertCONFIG_KERNEL_CIto internal loop feature
kconfig: removeCONFIG_KERNEL_CIinfrastructure
scripts: remove kernel-ci loop infrastructure
reboot-limit: simplify what gets selected
reboot-limit: add graph visualization support for results
reboot-limit: save graphs in organized results/graphs directory
docs: add comprehensive reboot-limit workflow documentation
reboot-limit: add kexec-tools dependency installation
reboot-limit: add A/B testing support targets
reboot-limit: fix kexec and reboot connection handling
reboot-limit: add COUNT parameter to override reboot count
reboot-limit: fixwait_fortasks using wrong host reference
reboot-limit: use ansible reboot module for all reboot types
reboot-limit: fix COUNT parameter to properly override reboot count
reboot-limit: handle empty dev group gracefully
reboot-limit: add kexec comparison featureREADME.md | 19 +- defconfigs/kernel-testing | 1 - defconfigs/reboot-limit | 31 + defconfigs/reboot-limit-compare | 36 ++ defconfigs/reboot-limit-kexec | 32 + docs/fstests.md | 13 +- docs/images/reboot_limit_example.png | Bin 0 -> 735262 bytes docs/reboot-limit.md | 223 +++++++ kconfigs/Kconfig.kdevops | 3 - kconfigs/Kconfig.kernel-ci | 170 ----- playbooks/roles/gen_hosts/tasks/main.yml | 13 + playbooks/roles/gen_hosts/templates/hosts.j2 | 55 +- playbooks/roles/gen_nodes/tasks/main.yml | 32 + .../reboot-limit/tasks/do-reboot-compare.yml | 126 ++++ .../roles/reboot-limit/tasks/do-reboot.yml | 98 ++- .../reboot-limit/tasks/handle-reboot-data.yml | 94 +++ .../tasks/install-deps/debian/main.yml | 10 + .../reboot-limit/tasks/install-deps/main.yml | 12 + .../tasks/install-deps/redhat/main.yml | 10 + .../tasks/install-deps/suse/main.yml | 10 + playbooks/roles/reboot-limit/tasks/main.yml | 88 ++- scripts/workflows/blktests/run_kernel_ci.sh | 1 - .../workflows/blktests/run_kernel_ci_kotd.sh | 1 - scripts/workflows/blktests/run_loop.sh | 66 -- .../demos/reboot-limit/analyze_results.py | 609 ++++++++++++++++++ .../reboot-limit/generate_sample_data.py | 73 +++ .../demos/reboot-limit/run_kernel_ci.sh | 1 - .../demos/reboot-limit/run_kernel_ci_kotd.sh | 1 - .../workflows/demos/reboot-limit/run_loop.sh | 23 +- .../demos/reboot-limit/run_loop_kotd.sh | 25 + scripts/workflows/fstests/run_kernel_ci.sh | 1 - .../workflows/fstests/run_kernel_ci_kotd.sh | 1 - scripts/workflows/fstests/run_loop.sh | 98 --- scripts/workflows/generic/run_kernel_ci.sh | 295 --------- scripts/workflows/gitr/run_kernel_ci.sh | 1 - scripts/workflows/gitr/run_kernel_ci_kotd.sh | 1 - scripts/workflows/gitr/run_loop.sh | 58 -- scripts/workflows/kotd/run_kernel_ci_kotd.sh | 107 --- scripts/workflows/ltp/run_kernel_ci.sh | 1 - scripts/workflows/ltp/run_kernel_ci_kotd.sh | 1 - scripts/workflows/ltp/run_loop.sh | 58 -- scripts/workflows/nfstest/run_kernel_ci.sh | 1 - .../workflows/nfstest/run_kernel_ci_kotd.sh | 1 - scripts/workflows/nfstest/run_loop.sh | 58 -- scripts/workflows/pynfs/run_kernel_ci.sh | 1 - scripts/workflows/pynfs/run_kernel_ci_kotd.sh | 1 - scripts/workflows/pynfs/run_loop.sh | 79 --- workflows/blktests/Makefile | 13 - workflows/demos/reboot-limit/Kconfig | 64 +- workflows/demos/reboot-limit/Makefile | 117 +++- workflows/fstests/Makefile | 14 - workflows/gitr/Makefile | 19 - workflows/ltp/Makefile | 19 - workflows/nfstest/Makefile | 19 - workflows/pynfs/Makefile | 26 - 55 files changed, 1737 insertions(+), 1193 deletions(-) create mode 100644 defconfigs/reboot-limit create mode 100644 defconfigs/reboot-limit-compare create mode 100644 defconfigs/reboot-limit-kexec create mode 100644 docs/images/reboot_limit_example.png create mode 100644 docs/reboot-limit.md delete mode 100644 kconfigs/Kconfig.kernel-ci create mode 100644 playbooks/roles/reboot-limit/tasks/do-reboot-compare.yml create mode 100644 playbooks/roles/reboot-limit/tasks/handle-reboot-data.yml create mode 100644 playbooks/roles/reboot-limit/tasks/install-deps/debian/main.yml create mode 100644 playbooks/roles/reboot-limit/tasks/install-deps/main.yml create mode 100644 playbooks/roles/reboot-limit/tasks/install-deps/redhat/main.yml create mode 100644 playbooks/roles/reboot-limit/tasks/install-deps/suse/main.yml delete mode 120000 scripts/workflows/blktests/run_kernel_ci.sh delete mode 120000 scripts/workflows/blktests/run_kernel_ci_kotd.sh delete mode 100755 scripts/workflows/blktests/run_loop.sh create mode 100755 scripts/workflows/demos/reboot-limit/analyze_results.py create mode 100755 scripts/workflows/demos/reboot-limit/generate_sample_data.py delete mode 120000 scripts/workflows/demos/reboot-limit/run_kernel_ci.sh delete mode 120000 scripts/workflows/demos/reboot-limit/run_kernel_ci_kotd.sh create mode 100755 scripts/workflows/demos/reboot-limit/run_loop_kotd.sh delete mode 120000 scripts/workflows/fstests/run_kernel_ci.sh delete mode 120000 scripts/workflows/fstests/run_kernel_ci_kotd.sh delete mode 100755 scripts/workflows/fstests/run_loop.sh delete mode 100755 scripts/workflows/generic/run_kernel_ci.sh delete mode 120000 scripts/workflows/gitr/run_kernel_ci.sh delete mode 120000 scripts/workflows/gitr/run_kernel_ci_kotd.sh delete mode 100755 scripts/workflows/gitr/run_loop.sh delete mode 100755 scripts/workflows/kotd/run_kernel_ci_kotd.sh delete mode 120000 scripts/workflows/ltp/run_kernel_ci.sh
Quoted 9 lines
delete mode 120000 scripts/workflows/ltp/run_kernel_ci_kotd.sh delete mode 100755 scripts/workflows/ltp/run_loop.sh delete mode 120000 scripts/workflows/nfstest/run_kernel_ci.sh delete mode 120000 scripts/workflows/nfstest/run_kernel_ci_kotd.sh delete mode 100755 scripts/workflows/nfstest/run_loop.sh delete mode 120000 scripts/workflows/pynfs/run_kernel_ci.sh delete mode 120000 scripts/workflows/pynfs/run_kernel_ci_kotd.sh delete mode 100755 scripts/workflows/pynfs/run_loop.sh
Signature · 1 line
Chuck Lever
Quoted 1 line · Chuck Lever
Reviewed-by Chuck Lever <[email protected]>
Pushed with what I hope is the CI fix which was defining workflows_reboot_limit as False.
Luis