declared hosts support
cover declared hosts support
It should be easy for us to add bare metal support because well, the biggest pain is bring up. If bringup and ssh is already configured, then all we have to do is just skip a bunch of things. So the question to ask, is how do we generate the hosts file? The easiest interface I could think of is to allow you to specify it at make menuconfig time or defconfig time with a DECLARE_HOSTS= specifier with either a space or comma separated lists of hosts.
I tested this with a new workflow I am testing, minio WARP testing, but I also tested this simply with a random host I can ssh to with just:
make defconfig-xfs_reflink_4k DECLARE_HOSTS=foo ; make AV=2 make uname
And that works. For my other workflow I am testing I got full test restults and mkfs'd a drive. And so each workflow needs a bit of love for further DECLARE_HOSTS support. Things like the data partition, and so on. I suspect we can just specify some of these in the command line as CLI variables.
But this goes with just basic raw support. Maybe we only enable workflows for KDEVOPS_USE_DECLARED_HOSTS if we declare the workflow as supporting it (maybe select SUPPORTS_DECLARED_HOSTS)?
Luis Chamberlain (2):
gen_hosts: refactor monolithic hosts.j2 template into per-workflow
templates
declared_hosts: add support for pre-existing infrastructure
Makefile | 7 + kconfigs/Kconfig.bringup | 7 + kconfigs/Kconfig.declared_hosts | 65 +++++ playbooks/roles/devconfig/tasks/main.yml | 60 ++++ playbooks/roles/gen_hosts/tasks/main.yml | 40 ++- playbooks/roles/gen_hosts/templates/hosts.j2 | 264 +++--------------- .../roles/gen_hosts/templates/workflows/ai.j2 | 99 +++++++ .../gen_hosts/templates/workflows/blktests.j2 | 58 ++++ .../gen_hosts/templates/workflows/cxl.j2 | 7 + .../templates/workflows/declared-hosts.j2 | 219 +++++++++++++++ .../gen_hosts/templates/workflows/default.j2 | 62 ++++ .../templates/workflows/fio-tests.j2 | 38 +++ .../gen_hosts/templates/workflows/fstests.j2 | 72 +++++ .../gen_hosts/templates/workflows/gitr.j2 | 41 +++ .../gen_hosts/templates/workflows/ltp.j2 | 41 +++ .../gen_hosts/templates/workflows/mmtests.j2 | 77 +++++ .../gen_hosts/templates/workflows/nfstest.j2 | 41 +++ .../gen_hosts/templates/workflows/pynfs.j2 | 7 + .../templates/workflows/reboot-limit.j2 | 33 +++ .../templates/workflows/selftests.j2 | 53 ++++ .../gen_hosts/templates/workflows/sysbench.j2 | 53 ++++ playbooks/roles/gen_nodes/tasks/main.yml | 5 + workflows/ai/Makefile | 3 +- workflows/blktests/Makefile | 3 +- workflows/cxl/Makefile | 3 +- workflows/fio-tests/Makefile | 3 +- workflows/fstests/Makefile | 3 +- workflows/gitr/Makefile | 3 +- workflows/linux/Makefile | 3 +- workflows/ltp/Makefile | 3 +- workflows/mmtests/Makefile | 3 +- workflows/nfstest/Makefile | 3 +- workflows/pynfs/Makefile | 3 +- workflows/selftests/Makefile | 3 +- workflows/sysbench/Makefile | 3 +- 35 files changed, 1143 insertions(+), 245 deletions(-) create mode 100644 kconfigs/Kconfig.declared_hosts create mode 100644 playbooks/roles/gen_hosts/templates/workflows/ai.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/blktests.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/cxl.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/declared-hosts.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/default.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/fio-tests.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/fstests.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/gitr.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/ltp.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/mmtests.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/nfstest.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/pynfs.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/reboot-limit.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/selftests.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/sysbench.j2
Signature · 1 line
2.50.1
PATCH 1/2 gen_hosts: refactor monolithic hosts.j2 template into per-workflow templates
The hosts.j2 template had become unwieldy spaghetti code with all workflows mixed together in a single file, making it difficult to read, maintain, and extend. This was particularly problematic when attempting to add new features.
This refactoring:
- Splits the monolithic hosts.j2 into individual workflow templates under playbooks/roles/gen_hosts/templates/workflows/
- Creates dedicated templates for each workflow (fstests, blktests, selftests, mmtests, sysbench, ai, fio-tests, etc.)
- Converts the main hosts.j2 to use Jinja2 includes for cleaner organization
- Maintains exact same functionality with improved maintainability
- Removes workflow-specific template overrides from all workflow Makefiles as they now all use the unified hosts.j2 template
Benefits:
- Each workflow template is now self-contained and easy to understand
- Adding new workflows no longer requires navigating through hundreds of lines of unrelated configuration
- Template logic is easier to debug and test
- All workflows now use a consistent template system
This is a pure refactoring with no functional changes. The generated hosts files remain identical to the previous implementation.
--- playbooks/roles/gen_hosts/tasks/main.yml | 40 ++- playbooks/roles/gen_hosts/templates/hosts.j2 | 264 +++--------------- .../roles/gen_hosts/templates/workflows/ai.j2 | 99 +++++++ .../gen_hosts/templates/workflows/blktests.j2 | 58 ++++ .../gen_hosts/templates/workflows/cxl.j2 | 7 + .../gen_hosts/templates/workflows/default.j2 | 62 ++++ .../templates/workflows/fio-tests.j2 | 38 +++ .../gen_hosts/templates/workflows/fstests.j2 | 72 +++++ .../gen_hosts/templates/workflows/gitr.j2 | 41 +++ .../gen_hosts/templates/workflows/ltp.j2 | 41 +++ .../gen_hosts/templates/workflows/mmtests.j2 | 77 +++++ .../gen_hosts/templates/workflows/nfstest.j2 | 41 +++ .../gen_hosts/templates/workflows/pynfs.j2 | 7 + .../templates/workflows/reboot-limit.j2 | 33 +++ .../templates/workflows/selftests.j2 | 53 ++++ .../gen_hosts/templates/workflows/sysbench.j2 | 53 ++++ workflows/ai/Makefile | 3 +- workflows/blktests/Makefile | 3 +- workflows/cxl/Makefile | 3 +- workflows/fio-tests/Makefile | 3 +- workflows/fstests/Makefile | 3 +- workflows/gitr/Makefile | 3 +- workflows/linux/Makefile | 3 +- workflows/ltp/Makefile | 3 +- workflows/mmtests/Makefile | 3 +- workflows/nfstest/Makefile | 3 +- workflows/pynfs/Makefile | 3 +- workflows/selftests/Makefile | 3 +- workflows/sysbench/Makefile | 3 +- 29 files changed, 780 insertions(+), 245 deletions(-) create mode 100644 playbooks/roles/gen_hosts/templates/workflows/ai.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/blktests.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/cxl.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/default.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/fio-tests.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/fstests.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/gitr.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/ltp.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/mmtests.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/nfstest.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/pynfs.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/reboot-limit.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/selftests.j2 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/sysbench.j2
playbooks/roles/gen_hosts/tasks/main.yml +39 -1
@@ -10,6 +10,7 @@ skip: true tags: vars + - name: Get our user ansible.builtin.command: "whoami" register: my_user
@@ -39,6 +40,21 @@ path: "{{ kdevops_hosts_template_full_path }}" register: ansible_hosts_template +- name: Generate the Ansible hosts file for declared hosts configuration + tags: ['hosts'] + ansible.builtin.template: + src: "{{ kdevops_hosts_template }}" + dest: "{{ ansible_cfg_inventory }}" + force: true + trim_blocks: True + lstrip_blocks: True + mode: '0644' + when: + - kdevops_use_declared_hosts|default(false)|bool + - kdevops_declared_hosts is defined + - kdevops_declared_hosts|length > 0 + - ansible_hosts_template.stat.exists + - name: Set fstests config file variable for {{ fstests_fstyp }} ansible.builtin.set_fact: is_fstests: true
@@ -67,6 +83,7 @@ when: - bootlinux_builder - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - name: Generate the Ansible inventory file tags: ["hosts"]
@@ -80,6 +97,7 @@ - not kdevops_workflows_dedicated_workflow - ansible_hosts_template.stat.exists - not kdevops_enable_nixos|default(false)|bool + - not kdevops_use_declared_hosts|default(false)|bool - name: Generate the Ansible inventory file for NixOS tags: ['hosts']
@@ -93,6 +111,7 @@ - not kdevops_workflows_dedicated_workflow - ansible_hosts_template.stat.exists - kdevops_enable_nixos|default(false)|bool + - not kdevops_use_declared_hosts|default(false)|bool - name: Update Ansible inventory access modification time so make sees it updated ansible.builtin.file:
@@ -112,6 +131,7 @@ - kdevops_workflows_dedicated_workflow - kdevops_workflow_enable_cxl - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - name: Generate the Ansible inventory file for dedicated pynfs work tags: ["hosts"]
@@ -125,6 +145,7 @@ - kdevops_workflows_dedicated_workflow - kdevops_workflow_enable_pynfs - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - name: Generate the Ansible inventory file for dedicated gitr workflow tags: ["hosts"]
@@ -140,6 +161,7 @@ - kdevops_workflows_dedicated_workflow - kdevops_workflow_enable_gitr - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - name: Generate an Ansible inventory file for a dedicated ltp workflow tags: ["hosts"]
@@ -155,6 +177,7 @@ - kdevops_workflows_dedicated_workflow - kdevops_workflow_enable_ltp - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - name: Generate the Ansible inventory file for dedicated nfstest workflow tags: ["hosts"]
@@ -170,6 +193,7 @@ - kdevops_workflows_dedicated_workflow - kdevops_workflow_enable_nfstest - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - name: Set empty enabled test types list for fstests tags: ["hosts"]
@@ -178,6 +202,7 @@ when: - is_fstests - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - name: Check which fstests test types are enabled register: fstests_enabled_test_types_reg
@@ -202,6 +227,7 @@ when: - is_fstests - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - name: Now expand the list of enabled fstests for valid configuration sections tags: ["hosts"]
@@ -213,6 +239,7 @@ when: - is_fstests - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - item.changed - name: Generate the Ansible inventory file for a dedicated fstests setup
@@ -233,6 +260,7 @@ when: - is_fstests - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - name: Infer enabled blktests test section types ansible.builtin.set_fact:
@@ -247,6 +275,7 @@ - kdevops_workflows_dedicated_workflow - kdevops_workflow_enable_blktests - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - name: Debug inferring block test types ansible.builtin.debug:
@@ -269,6 +298,7 @@ - kdevops_workflows_dedicated_workflow - kdevops_workflow_enable_blktests - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - name: Infer enabled selftests test section types ansible.builtin.set_fact:
@@ -283,6 +313,7 @@ - kdevops_workflows_dedicated_workflow - kdevops_workflow_enable_selftests - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - name: Generate the Ansible inventory file for a dedicated selftests setup tags: ["hosts"]
@@ -296,6 +327,7 @@ - kdevops_workflows_dedicated_workflow - kdevops_workflow_enable_selftests - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - name: Collect dynamically supported filesystems vars:
@@ -338,6 +370,7 @@ - kdevops_workflows_dedicated_workflow - kdevops_workflow_enable_sysbench - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - name: Generate the Ansible hosts file for a dedicated fio-tests setup
@@ -354,6 +387,7 @@ - kdevops_workflow_enable_fio_tests - ansible_hosts_template.stat.exists - not kdevops_enable_nixos|default(false)|bool + - not kdevops_use_declared_hosts|default(false)|bool - name: Infer enabled mmtests test types
@@ -369,6 +403,7 @@ - kdevops_workflows_dedicated_workflow - kdevops_workflow_enable_mmtests - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - name: Generate the Ansible hosts file for a dedicated mmtests setup tags: ["hosts"]
@@ -382,6 +417,7 @@ - kdevops_workflows_dedicated_workflow - kdevops_workflow_enable_mmtests - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - name: Generate the Ansible hosts file for a dedicated reboot-limit setup tags: ["hosts"]
@@ -395,6 +431,7 @@ - kdevops_workflows_dedicated_workflow - workflows_reboot_limit - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - name: Load AI nodes configuration for multi-filesystem setup include_vars:
@@ -426,8 +463,9 @@ mode: '0644' when: - kdevops_workflows_dedicated_workflow - - kdevops_workflow_enable_ai + - kdevops_workflow_enable_ai|default(false)|bool - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - name: Verify if final host file exists ansible.builtin.stat:
playbooks/roles/gen_hosts/templates/hosts.j2 +33 -231
276 lines not on this page · raw message
playbooks/roles/gen_hosts/templates/workflows/ai.j2 new file +99 -0
99 lines not on this page · raw message
playbooks/roles/gen_hosts/templates/workflows/blktests.j2 new file +58 -0
58 lines not on this page · raw message
playbooks/roles/gen_hosts/templates/workflows/cxl.j2 new file +7 -0
@@ -0,0 +1,7 @@ +{# Workflow template for CXL #} +[all] +localhost ansible_connection=local +write-your-own-template-for-cxl-workflow + +[all:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}"
playbooks/roles/gen_hosts/templates/workflows/default.j2 new file +62 -0
62 lines not on this page · raw message
playbooks/roles/gen_hosts/templates/workflows/fio-tests.j2 new file +38 -0
@@ -0,0 +1,38 @@ +{# Workflow template for fio-tests #} +[all] +localhost ansible_connection=local +{{ kdevops_host_prefix }}-fio-tests +{% if kdevops_baseline_and_dev %} +{{ kdevops_host_prefix }}-fio-tests-dev +{% endif %} + +[all:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}" + +[baseline] +{{ kdevops_host_prefix }}-fio-tests + +[baseline:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}" + +{% if kdevops_baseline_and_dev %} +[dev] +{{ kdevops_host_prefix }}-fio-tests-dev + +[dev:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}" + +{% endif %} +[fio_tests] +{{ kdevops_host_prefix }}-fio-tests +{% if kdevops_baseline_and_dev %} +{{ kdevops_host_prefix }}-fio-tests-dev +{% endif %} + +[fio_tests:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}" + +[service] + +[service:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}"
playbooks/roles/gen_hosts/templates/workflows/fstests.j2 new file +72 -0
72 lines not on this page · raw message
playbooks/roles/gen_hosts/templates/workflows/gitr.j2 new file +41 -0
41 lines not on this page · raw message
playbooks/roles/gen_hosts/templates/workflows/ltp.j2 new file +41 -0
41 lines not on this page · raw message
playbooks/roles/gen_hosts/templates/workflows/mmtests.j2 new file +77 -0
77 lines not on this page · raw message
playbooks/roles/gen_hosts/templates/workflows/nfstest.j2 new file +41 -0
41 lines not on this page · raw message
playbooks/roles/gen_hosts/templates/workflows/pynfs.j2 new file +7 -0
@@ -0,0 +1,7 @@ +{# Workflow template for pynfs #} +[all] +localhost ansible_connection=local +write-your-own-template-for-pynfs-workflow + +[all:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}"
playbooks/roles/gen_hosts/templates/workflows/reboot-limit.j2 new file +33 -0
@@ -0,0 +1,33 @@ +{# Workflow template for reboot-limit #} +[all] +localhost ansible_connection=local +{{ kdevops_host_prefix }}-reboot-limit +{% if kdevops_baseline_and_dev %} +{{ kdevops_host_prefix }}-reboot-limit-dev +{% endif %} + +[all:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}" + +[baseline] +{{ kdevops_host_prefix }}-reboot-limit + +[baseline:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}" + +{% if kdevops_baseline_and_dev %} +[dev] +{{ kdevops_host_prefix }}-reboot-limit-dev + +[dev:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}" + +{% endif %} +[reboot-limit] +{{ kdevops_host_prefix }}-reboot-limit +{% if kdevops_baseline_and_dev %} +{{ kdevops_host_prefix }}-reboot-limit-dev +{% endif %} + +[reboot-limit:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}"
playbooks/roles/gen_hosts/templates/workflows/selftests.j2 new file +53 -0
53 lines not on this page · raw message
playbooks/roles/gen_hosts/templates/workflows/sysbench.j2 new file +53 -0
53 lines not on this page · raw message
workflows/ai/Makefile +2 -1
@@ -4,7 +4,8 @@ PHONY += ai-tests ai-tests-baseline ai-tests-dev PHONY += ai-tests-results ifeq (y,$(CONFIG_WORKFLOWS_DEDICATED_WORKFLOW)) -export KDEVOPS_HOSTS_TEMPLATE := hosts.j2 +# After refactoring, all workflows use hosts.j2 which includes workflow-specific templates +# export KDEVOPS_HOSTS_TEMPLATE := hosts.j2 endif export AI_DATA_TARGET := $(subst ",,$(CONFIG_AI_BENCHMARK_RESULTS_DIR))
workflows/blktests/Makefile +2 -1
@@ -12,7 +12,8 @@ ID=$(shell id -u) BLKTESTS_ARGS := ifeq (y,$(CONFIG_WORKFLOWS_DEDICATED_WORKFLOW)) -export KDEVOPS_HOSTS_TEMPLATE := blktests.j2 +# After refactoring, all workflows use hosts.j2 which includes workflow-specific templates +# export KDEVOPS_HOSTS_TEMPLATE := blktests.j2 endif BLKTESTS_GIT:=$(subst ",,$(CONFIG_BLKTESTS_GIT))
workflows/cxl/Makefile +2 -1
@@ -1,6 +1,7 @@ # SPDX-License-Identifier: copyleft-next-0.3.1 -export KDEVOPS_HOSTS_TEMPLATE := cxl.j2 +# After refactoring, all workflows use hosts.j2 which includes workflow-specific templates +# export KDEVOPS_HOSTS_TEMPLATE := cxl.j2 CXL_ARGS := CXL_ARGS += ndctl_git='$(subst ",,$(CONFIG_NDCTL_GIT))'
workflows/fio-tests/Makefile +2 -1
@@ -1,5 +1,6 @@ ifeq (y,$(CONFIG_WORKFLOWS_DEDICATED_WORKFLOW)) -export KDEVOPS_HOSTS_TEMPLATE := fio-tests.j2 +# After refactoring, all workflows use hosts.j2 which includes workflow-specific templates +# export KDEVOPS_HOSTS_TEMPLATE := fio-tests.j2 endif fio-tests:
workflows/fstests/Makefile +2 -1
@@ -9,7 +9,8 @@ FSTESTS_BASELINE_EXTRA := export FSTYP:=$(subst ",,$(CONFIG_FSTESTS_FSTYP)) ifeq (y,$(CONFIG_WORKFLOWS_DEDICATED_WORKFLOW)) -export KDEVOPS_HOSTS_TEMPLATE := fstests.j2 +# After refactoring, all workflows use hosts.j2 which includes workflow-specific templates +# export KDEVOPS_HOSTS_TEMPLATE := fstests.j2 endif FSTESTS_ARGS += fstests_fstyp='$(FSTYP)'
workflows/gitr/Makefile +2 -1
@@ -1,5 +1,6 @@ ifeq (y,$(CONFIG_WORKFLOWS_DEDICATED_WORKFLOW)) -export KDEVOPS_HOSTS_TEMPLATE := gitr.j2 +# After refactoring, all workflows use hosts.j2 which includes workflow-specific templates +# export KDEVOPS_HOSTS_TEMPLATE := gitr.j2 endif GITR_MNT:=$(subst ",,$(CONFIG_GITR_MNT))
workflows/linux/Makefile +2 -1
@@ -14,7 +14,8 @@ TREE_CONFIG:=config-$(TREE_REF)-pure-iomap endif ifeq (y,$(CONFIG_BOOTLINUX_BUILDER)) -KDEVOPS_HOSTS_TEMPLATE=builder.j2 +# After refactoring, all workflows use hosts.j2 which includes workflow-specific templates +# KDEVOPS_HOSTS_TEMPLATE=builder.j2 endif # Describes the Linux clone
workflows/ltp/Makefile +2 -1
@@ -1,5 +1,6 @@ ifeq (y,$(CONFIG_WORKFLOWS_DEDICATED_WORKFLOW)) -export KDEVOPS_HOSTS_TEMPLATE := ltp.j2 +# After refactoring, all workflows use hosts.j2 which includes workflow-specific templates +# export KDEVOPS_HOSTS_TEMPLATE := ltp.j2 endif LTP_REPO:=$(subst ",,$(CONFIG_LTP_REPO))
workflows/mmtests/Makefile +2 -1
@@ -1,7 +1,8 @@ MMTESTS_ARGS := ifeq (y,$(CONFIG_WORKFLOWS_DEDICATED_WORKFLOW)) -export KDEVOPS_HOSTS_TEMPLATE := mmtests.j2 +# After refactoring, all workflows use hosts.j2 which includes workflow-specific templates +# export KDEVOPS_HOSTS_TEMPLATE := mmtests.j2 endif mmtests:
workflows/nfstest/Makefile +2 -1
@@ -1,5 +1,6 @@ ifeq (y,$(CONFIG_WORKFLOWS_DEDICATED_WORKFLOW)) -export KDEVOPS_HOSTS_TEMPLATE := nfstest.j2 +# After refactoring, all workflows use hosts.j2 which includes workflow-specific templates +# export KDEVOPS_HOSTS_TEMPLATE := nfstest.j2 endif # CONFIG_WORKFLOWS_DEDICATED_WORKFLOW ifeq (y,$(CONFIG_NFSTEST_USE_KDEVOPS_NFSD))
workflows/pynfs/Makefile +2 -1
@@ -1,5 +1,6 @@ ifeq (y,$(CONFIG_WORKFLOWS_DEDICATED_WORKFLOW)) -export KDEVOPS_HOSTS_TEMPLATE := pynfs.j2 +# After refactoring, all workflows use hosts.j2 which includes workflow-specific templates +# export KDEVOPS_HOSTS_TEMPLATE := pynfs.j2 endif PYNFS_GIT:=$(subst ",,$(CONFIG_PYNFS_GIT))
workflows/selftests/Makefile +2 -1
@@ -3,7 +3,8 @@ SELFTESTS_ARGS := ifeq (y,$(CONFIG_WORKFLOWS_DEDICATED_WORKFLOW)) -export KDEVOPS_HOSTS_TEMPLATE := selftests.j2 +# After refactoring, all workflows use hosts.j2 which includes workflow-specific templates +# export KDEVOPS_HOSTS_TEMPLATE := selftests.j2 endif SELFTESTS_DYNAMIC_RUNTIME_VARS := "kdevops_run_selftests": True
workflows/sysbench/Makefile +2 -1
@@ -1,7 +1,8 @@ PHONY += sysbench sysbench-test sysbench-telemetry sysbench-help-menu ifeq (y,$(CONFIG_WORKFLOWS_DEDICATED_WORKFLOW)) -export KDEVOPS_HOSTS_TEMPLATE := sysbench.j2 +# After refactoring, all workflows use hosts.j2 which includes workflow-specific templates +# export KDEVOPS_HOSTS_TEMPLATE := sysbench.j2 endif TAGS_SYSBENCH_RUN := db_start
Signature · 1 line
2.50.1
PATCH 2/2 declared_hosts: add support for pre-existing infrastructure
In reply to Luis ChamberlainThis adds support for using pre-existing infrastructure (bare metal servers, pre-provisioned VMs) that already have SSH access configured. This is useful for environments where kdevops bringup/teardown is not desired or possible.
Key features:
- New DECLARE_HOSTS CLI parameter to specify existing hosts
- Automatic detection of CLI override to enable declared hosts mode
- SKIP_BRINGUP is automatically selected when using declared hosts
- Hosts can be specified as space or comma-separated list
- For A/B testing workflows, odd-numbered hosts become baseline nodes and even-numbered hosts become dev nodes
- New Kconfig.declared_hosts for configuration options
- New declared-hosts.j2 template for generating inventory
Usage:
make defconfig-xfs_reflink_4k DECLARE_HOSTS='server1 server2 server3' make # Will skip bringup and use the declared hosts
The implementation assumes SSH access is already configured in the user's ~/.ssh/config or via SSH keys. No SSH configuration is generated by kdevops when using declared hosts.
This feature integrates cleanly with the refactored template system, using the same hosts.j2 entry point with a dedicated declared-hosts.j2 template for this mode.
--- Makefile | 7 + kconfigs/Kconfig.bringup | 7 + kconfigs/Kconfig.declared_hosts | 65 ++++++ playbooks/roles/devconfig/tasks/main.yml | 60 +++++ .../templates/workflows/declared-hosts.j2 | 219 ++++++++++++++++++ playbooks/roles/gen_nodes/tasks/main.yml | 5 + 6 files changed, 363 insertions(+) create mode 100644 kconfigs/Kconfig.declared_hosts create mode 100644 playbooks/roles/gen_hosts/templates/workflows/declared-hosts.j2
Makefile +7 -0
@@ -19,6 +19,13 @@ export KDEVOPS_NODES := export PYTHONUNBUFFERED=1 export TOPDIR=./ export TOPDIR_PATH = $(shell readlink -f $(TOPDIR)) + +# Export CLI override variables for Kconfig to detect them +# Note: We accept DECLARE_HOSTS but export as DECLARED_HOSTS for consistency +ifdef DECLARE_HOSTS +export DECLARED_HOSTS := $(DECLARE_HOSTS) +endif + include scripts/refs.Makefile KDEVOPS_NODES_ROLE_TEMPLATE_DIR := $(KDEVOPS_PLAYBOOKS_DIR)/roles/gen_nodes/templates
kconfigs/Kconfig.bringup +7 -0
@@ -9,8 +9,14 @@ config KDEVOPS_ENABLE_NIXOS bool output yaml +# CLI override detection for DECLARED_HOSTS which should enable SKIP_BRINGUP +config SKIP_BRINGUP_SET_BY_CLI + bool + default $(shell, scripts/check-cli-set-var.sh DECLARED_HOSTS) + choice prompt "Node bring up method" + default SKIP_BRINGUP if SKIP_BRINGUP_SET_BY_CLI default GUESTFS config GUESTFS
@@ -80,6 +86,7 @@ config LIBVIRT source "kconfigs/Kconfig.guestfs" source "kconfigs/Kconfig.nixos" source "terraform/Kconfig" +source "kconfigs/Kconfig.declared_hosts" if LIBVIRT source "kconfigs/Kconfig.libvirt" endif
kconfigs/Kconfig.declared_hosts new file +65 -0
65 lines not on this page · raw message
playbooks/roles/devconfig/tasks/main.yml +60 -0
66 lines not on this page · raw message
playbooks/roles/gen_hosts/templates/workflows/declared-hosts.j2 new file +219 -0
219 lines not on this page · raw message
playbooks/roles/gen_nodes/tasks/main.yml +5 -0
@@ -20,6 +20,11 @@ register: my_group changed_when: false +- name: Skip node generation if using declared hosts + meta: end_play + when: + - kdevops_use_declared_hosts|default(false)|bool + - name: Create guestfs directory ansible.builtin.file: path: "{{ guestfs_path }}"
Signature · 1 line
2.50.1
Re: [PATCH 1/2] gen_hosts: refactor monolithic hosts.j2 template into per-workflow templates
In reply to Luis ChamberlainQuoted 169 lines · Luis Chamberlain
playbooks/roles/gen_hosts/tasks/main.yml +21
-0@@ -67,6 +83,7 @@ when: - bootlinux_builder - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - name: Generate the Ansible inventory file tags: ["hosts"]@@ -80,6 +97,7 @@ - not kdevops_workflows_dedicated_workflow - ansible_hosts_template.stat.exists - not kdevops_enable_nixos|default(false)|bool + - not kdevops_use_declared_hosts|default(false)|bool - name: Generate the Ansible inventory file for NixOS tags: ['hosts']@@ -93,6 +111,7 @@ - not kdevops_workflows_dedicated_workflow - ansible_hosts_template.stat.exists - kdevops_enable_nixos|default(false)|bool + - not kdevops_use_declared_hosts|default(false)|bool - name: Update Ansible inventory access modification time so make sees it updated ansible.builtin.file:@@ -112,6 +131,7 @@ - kdevops_workflows_dedicated_workflow - kdevops_workflow_enable_cxl - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - name: Generate the Ansible inventory file for dedicated pynfs work tags: ["hosts"]@@ -125,6 +145,7 @@ - kdevops_workflows_dedicated_workflow - kdevops_workflow_enable_pynfs - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - name: Generate the Ansible inventory file for dedicated gitr workflow tags: ["hosts"]@@ -140,6 +161,7 @@ - kdevops_workflows_dedicated_workflow - kdevops_workflow_enable_gitr - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - name: Generate an Ansible inventory file for a dedicated ltp workflow tags: ["hosts"]@@ -155,6 +177,7 @@ - kdevops_workflows_dedicated_workflow - kdevops_workflow_enable_ltp - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - name: Generate the Ansible inventory file for dedicated nfstest workflow tags: ["hosts"]@@ -170,6 +193,7 @@ - kdevops_workflows_dedicated_workflow - kdevops_workflow_enable_nfstest - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - name: Set empty enabled test types list for fstests tags: ["hosts"]@@ -178,6 +202,7 @@ when: - is_fstests - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - name: Check which fstests test types are enabled register: fstests_enabled_test_types_reg@@ -202,6 +227,7 @@ when: - is_fstests - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - name: Now expand the list of enabled fstests for valid configuration sections tags: ["hosts"]@@ -213,6 +239,7 @@ when: - is_fstests - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - item.changed - name: Generate the Ansible inventory file for a dedicated fstests setup@@ -233,6 +260,7 @@ when: - is_fstests - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - name: Infer enabled blktests test section types ansible.builtin.set_fact:@@ -247,6 +275,7 @@ - kdevops_workflows_dedicated_workflow - kdevops_workflow_enable_blktests - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - name: Debug inferring block test types ansible.builtin.debug:@@ -269,6 +298,7 @@ - kdevops_workflows_dedicated_workflow - kdevops_workflow_enable_blktests - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - name: Infer enabled selftests test section types ansible.builtin.set_fact:@@ -283,6 +313,7 @@ - kdevops_workflows_dedicated_workflow - kdevops_workflow_enable_selftests - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - name: Generate the Ansible inventory file for a dedicated selftests setup tags: ["hosts"]@@ -296,6 +327,7 @@ - kdevops_workflows_dedicated_workflow - kdevops_workflow_enable_selftests - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - name: Collect dynamically supported filesystems vars:@@ -338,6 +370,7 @@ - kdevops_workflows_dedicated_workflow - kdevops_workflow_enable_sysbench - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - name: Generate the Ansible hosts file for a dedicated fio-tests setup@@ -354,6 +387,7 @@ - kdevops_workflow_enable_fio_tests - ansible_hosts_template.stat.exists - not kdevops_enable_nixos|default(false)|bool + - not kdevops_use_declared_hosts|default(false)|bool - name: Infer enabled mmtests test types@@ -369,6 +403,7 @@ - kdevops_workflows_dedicated_workflow - kdevops_workflow_enable_mmtests - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - name: Generate the Ansible hosts file for a dedicated mmtests setup tags: ["hosts"]@@ -382,6 +417,7 @@ - kdevops_workflows_dedicated_workflow - kdevops_workflow_enable_mmtests - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - name: Generate the Ansible hosts file for a dedicated reboot-limit setup tags: ["hosts"]@@ -395,6 +431,7 @@ - kdevops_workflows_dedicated_workflow - workflows_reboot_limit - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool - name: Load AI nodes configuration for multi-filesystem setup include_vars:
playbooks/roles/gen_hosts/tasks/main.yml line 469
Quoted 8 lines
playbooks/roles/gen_hosts/tasks/main.yml +2
-1@@ -426,8 +463,9 @@ mode: '0644' when: - kdevops_workflows_dedicated_workflow - - kdevops_workflow_enable_ai + - kdevops_workflow_enable_ai|default(false)|bool - ansible_hosts_template.stat.exists + - not kdevops_use_declared_hosts|default(false)|bool
Sorry, these hunks should have been part of the second patch.
Luis