kdevops: use linux-kdevops for the main tree
Subsystems (2)
cover kdevops: use linux-kdevops for the main tree
More just like a heads up as to the next steps from discussions at LSFMM with regards to kdevops. Those using kdevops in some environemnts will want to read the last commit.
This work starts off by making the linux-kdevops organization the default for the source of commits for the kdevops projects. We can now share efforts, and commits. The last commit will contain more details as to the motivation for this.
Luis Chamberlain (4):
workflows/Kconfig: be consistent when enabling fstests or blktests
kdevops: move generic kdevops variables to its own file
playbooks: add a common playbook a git reset task for kdevops
kdevops: make linux-kdevops the default tree
kconfigs/workflows/Kconfig | 6 ++++ kconfigs/workflows/Kconfig.shared | 2 +- playbooks/common.yml | 4 +++ playbooks/roles/common/README.md | 38 ++++++++++++++++++++ playbooks/roles/common/defaults/main.yml | 7 ++++ playbooks/roles/common/tasks/main.yml | 23 ++++++++++++ workflows/Makefile | 33 +---------------- workflows/common/Makefile | 46 ++++++++++++++++++++++++ 8 files changed, 126 insertions(+), 33 deletions(-) create mode 100644 playbooks/common.yml create mode 100644 playbooks/roles/common/README.md create mode 100644 playbooks/roles/common/defaults/main.yml create mode 100644 playbooks/roles/common/tasks/main.yml create mode 100644 workflows/common/Makefile
Signature · 1 line
2.35.1
PATCH 1/4 workflows/Kconfig: be consistent when enabling fstests or blktests
We have two kconfig variables which we use to be able to express when we are going to enable fstests or blktests, either as a dedicated set of tests or when we want to enable testing both fstests and blktests in one system. But right now we only select this kconfig variable when we are using a dedicated system. This is not an issue as the kconfig is a kconfig symbols are bools which are set default to y if either the test is dedicated or not.
But to be pedantic, and clear, let's make sure the tests select the respective kconfig for each case as we'd expect to see it. Otherwise this can confuse folks reading this.
--- kconfigs/workflows/Kconfig | 2 ++ 1 file changed, 2 insertions(+)
kconfigs/workflows/Kconfig +2 -0
@@ -133,6 +133,7 @@ if !WORKFLOWS_DEDICATED_WORKFLOW config KDEVOPS_WORKFLOW_NOT_DEDICATED_ENABLE_FSTESTS bool "fstests" + select KDEVOPS_WORKFLOW_ENABLE_FSTESTS help Select this option if you are doing filesystem development and want to target development for a filesystem and enable fstests so that
@@ -146,6 +147,7 @@ config KDEVOPS_WORKFLOW_NOT_DEDICATED_ENABLE_FSTESTS config KDEVOPS_WORKFLOW_NOT_DEDICATED_ENABLE_BLKTESTS bool "blktests" + select KDEVOPS_WORKFLOW_ENABLE_BLKTESTS help Select this option if you are doing block layer development and want to run blktests. The git tree for blktests will be git clone and
Signature · 1 line
2.35.1
Re: [PATCH 1/4] workflows/Kconfig: be consistent when enabling fstests or blktests
Quoted 11 lines · Luis Chamberlain
We have two kconfig variables which we use to be able to express when we are going to enable fstests or blktests, either as a dedicated set of tests or when we want to enable testing both fstests and blktests in one system. But right now we only select this kconfig variable when we are using a dedicated system. This is not an issue as the kconfig is a kconfig symbols are bools which are set default to y if either the test is dedicated or not.
But to be pedantic, and clear, let's make sure the tests select the respective kconfig for each case as we'd expect to see it. Otherwise this can confuse folks reading this.
Is this patch perhaps intended for the kdevops project? If so, please add a prefix to make this clear (git format-patch --subject-prefix) when sending kdevops patches to Linux kernel mailing lists.
Thanks,
Bart.
Re: [PATCH 1/4] workflows/Kconfig: be consistent when enabling fstests or blktests
Quoted 9 lines · Bart Van Assche
Quoted 8 lines · Luis Chamberlain
We have two kconfig variables which we use to be able to express when we are going to enable fstests or blktests, either as a dedicated set of tests or when we want to enable testing both fstests and blktests in one system. But right now we only select this kconfig variable when we are using a dedicated system. This is not an issue as the kconfig is a kconfig symbols are bools which are set default to y if either the test is dedicated or not.
Quoted 7 lines
Quoted 3 lines
But to be pedantic, and clear, let's make sure the tests select the respective kconfig for each case as we'd expect to see it. Otherwise this can confuse folks reading this.
Is this patch perhaps intended for the kdevops project? If so, please add a prefix to make this clear (git format-patch --subject-prefix) when sending kdevops patches to Linux kernel mailing lists.
Yeah good idea for it to go into the [PATCH kdevops]. Thanks!
Luis
PATCH 2/4 kdevops: move generic kdevops variables to its own file
In reply to Luis ChamberlainThere are shared kdevops variables between different workflows, which *can* be used by any workflow. Move these into a generic kdevops helper Makefile, as we can later expand on this.
This makes no functional changes.
--- workflows/Makefile | 33 +-------------------------------- workflows/common/Makefile | 34 ++++++++++++++++++++++++++++++++++ 2 files changed, 35 insertions(+), 32 deletions(-) create mode 100644 workflows/common/Makefile
workflows/Makefile +1 -32
@@ -1,37 +1,6 @@ # SPDX-License-Identifier: copyleft-next-0.3.1 -# How we create the partition for the workflow data partition -WORKFLOW_DATA_DEVICE:=$(subst ",,$(CONFIG_WORKFLOW_DATA_DEVICE)) -WORKFLOW_DATA_PATH:=$(subst ",,$(CONFIG_WORKFLOW_DATA_PATH)) -WORKFLOW_DATA_FSTYPE:=$(subst ",,$(CONFIG_WORKFLOW_DATA_FSTYPE)) -WORKFLOW_DATA_LABEL:=$(subst ",,$(CONFIG_WORKFLOW_DATA_LABEL)) - -WORKFLOW_KDEVOPS_GIT:=$(subst ",,$(CONFIG_WORKFLOW_KDEVOPS_GIT)) -WORKFLOW_KDEVOPS_GIT_DATA:=$(subst ",,$(CONFIG_WORKFLOW_KDEVOPS_GIT_DATA)) -WORKFLOW_KDEVOPS_DIR:=$(subst ",,$(CONFIG_WORKFLOW_KDEVOPS_DIR)) - -WORKFLOW_ARGS += data_device=$(WORKFLOW_DATA_DEVICE) -WORKFLOW_ARGS += data_path=$(WORKFLOW_DATA_PATH) -WORKFLOW_ARGS += data_fstype=$(WORKFLOW_DATA_FSTYPE) -WORKFLOW_ARGS += data_label=$(WORKFLOW_DATA_LABEL) -WORKFLOW_ARGS += kdevops_git=$(WORKFLOW_KDEVOPS_GIT) -WORKFLOW_ARGS += kdevops_data=\"$(WORKFLOW_KDEVOPS_GIT_DATA)\" -WORKFLOW_ARGS += kdevops_dir=\"$(WORKFLOW_KDEVOPS_DIR)\" - -ifeq (y,$(CONFIG_WORKFLOW_MAKE_CMD_OVERRIDE)) -WORKFLOW_MAKE_CMD:=$(subst ",,$(CONFIG_WORKFLOW_MAKE_CMD)) -endif - -ifeq (y,$(CONFIG_WORKFLOW_INFER_USER_AND_GROUP)) -WORKFLOW_ARGS += infer_uid_and_group=True -else -WORKFLOW_DATA_USER:=$(subst ",,$(CONFIG_WORKFLOW_DATA_USER)) -WORKFLOW_DATA_GROUP:=$(subst ",,$(CONFIG_WORKFLOW_DATA_GROUP)) - -WORKFLOW_ARGS += data_user=$(WORKFLOW_DATA_USER) -WORKFLOW_ARGS += data_group=$(WORKFLOW_DATA_GROUP) - -endif # CONFIG_WORKFLOW_MAKE_CMD_OVERRIDE == y +include workflows/common/Makefile BOOTLINUX_ARGS := ifeq (y,$(CONFIG_BOOTLINUX))
workflows/common/Makefile new file +34 -0
@@ -0,0 +1,34 @@ +# SPDX-License-Identifier: copyleft-next-0.3.1 + +# How we create the partition for the workflow data partition +WORKFLOW_DATA_DEVICE:=$(subst ",,$(CONFIG_WORKFLOW_DATA_DEVICE)) +WORKFLOW_DATA_PATH:=$(subst ",,$(CONFIG_WORKFLOW_DATA_PATH)) +WORKFLOW_DATA_FSTYPE:=$(subst ",,$(CONFIG_WORKFLOW_DATA_FSTYPE)) +WORKFLOW_DATA_LABEL:=$(subst ",,$(CONFIG_WORKFLOW_DATA_LABEL)) + +WORKFLOW_KDEVOPS_GIT:=$(subst ",,$(CONFIG_WORKFLOW_KDEVOPS_GIT)) +WORKFLOW_KDEVOPS_GIT_DATA:=$(subst ",,$(CONFIG_WORKFLOW_KDEVOPS_GIT_DATA)) +WORKFLOW_KDEVOPS_DIR:=$(subst ",,$(CONFIG_WORKFLOW_KDEVOPS_DIR)) + +WORKFLOW_ARGS += data_device=$(WORKFLOW_DATA_DEVICE) +WORKFLOW_ARGS += data_path=$(WORKFLOW_DATA_PATH) +WORKFLOW_ARGS += data_fstype=$(WORKFLOW_DATA_FSTYPE) +WORKFLOW_ARGS += data_label=$(WORKFLOW_DATA_LABEL) +WORKFLOW_ARGS += kdevops_git=$(WORKFLOW_KDEVOPS_GIT) +WORKFLOW_ARGS += kdevops_data=\"$(WORKFLOW_KDEVOPS_GIT_DATA)\" +WORKFLOW_ARGS += kdevops_dir=\"$(WORKFLOW_KDEVOPS_DIR)\" + +ifeq (y,$(CONFIG_WORKFLOW_MAKE_CMD_OVERRIDE)) +WORKFLOW_MAKE_CMD:=$(subst ",,$(CONFIG_WORKFLOW_MAKE_CMD)) +endif + +ifeq (y,$(CONFIG_WORKFLOW_INFER_USER_AND_GROUP)) +WORKFLOW_ARGS += infer_uid_and_group=True +else +WORKFLOW_DATA_USER:=$(subst ",,$(CONFIG_WORKFLOW_DATA_USER)) +WORKFLOW_DATA_GROUP:=$(subst ",,$(CONFIG_WORKFLOW_DATA_GROUP)) + +WORKFLOW_ARGS += data_user=$(WORKFLOW_DATA_USER) +WORKFLOW_ARGS += data_group=$(WORKFLOW_DATA_GROUP) + +endif # CONFIG_WORKFLOW_INFER_USER_AND_GROUP == y
Signature · 1 line
2.35.1
PATCH 3/4 playbooks: add a common playbook a git reset task for kdevops
In reply to Luis ChamberlainTwo playbooks share the concept of git cloning kdevops into the target nodes (guests, cloud hosts, baremetal hosts) so that expunge files can be used for avoiding tests. If you decide you want to change the URL for that git tree it may not be so obvious what to do.
Fortunately the solution is simple. You just tell ansible to use the new git tree URL. That's it. It won't remove the old directory and things work as expected.
But since we use the kdevops git tree on both fstests and blktests it is not so obvious to developers that the thing to do here is to just run 'make fstests' or 'make blktests' and even that is not as efficient as that will also re-clone the fstests or blktests tree respectively. When we just want to reset the kdevops git tree we currently have no semantics to specify that. But since this is a common post-deployment goal, just add a common playbook that let's us do common tasks.
All we need then is the kconfig logic to define when some commmon tasks might make sense. So to reset your kdevops git tree, all you have to do now is change the configuration for it, then run:
make
make kdevops-git-reset
--- kconfigs/workflows/Kconfig | 4 +++ playbooks/common.yml | 4 +++ playbooks/roles/common/README.md | 38 ++++++++++++++++++++++++ playbooks/roles/common/defaults/main.yml | 7 +++++ playbooks/roles/common/tasks/main.yml | 23 ++++++++++++++ workflows/common/Makefile | 12 ++++++++ 6 files changed, 88 insertions(+) create mode 100644 playbooks/common.yml create mode 100644 playbooks/roles/common/README.md create mode 100644 playbooks/roles/common/defaults/main.yml create mode 100644 playbooks/roles/common/tasks/main.yml
kconfigs/workflows/Kconfig +4 -0
@@ -175,6 +175,10 @@ source "workflows/blktests/Kconfig" endmenu endif # KDEVOPS_WORKFLOW_ENABLE_BLKTESTS +config KDEVOPS_WORKFLOW_GIT_CLONES_KDEVOPS_GIT + bool + default y if KDEVOPS_WORKFLOW_ENABLE_FSTESTS || KDEVOPS_WORKFLOW_ENABLE_BLKTESTS + endif # WORKFLOWS_LINUX_TESTS endif # WORKFLOWS_TESTS
playbooks/common.yml new file +4 -0
@@ -0,0 +1,4 @@ +--- +- hosts: all + roles: + - role: common
playbooks/roles/common/README.md new file +38 -0
@@ -0,0 +1,38 @@ +common +====== + +The common role lets you add tasks which is commmon to all workflows. +Without this we would be duplicating code. + +Requirements +------------ + +None. + +Role Variables +-------------- + + * kdevops_git_reset: perform a git reset. This is useful in case you want + to change the URL you use for kdevops. + +Dependencies +------------ + +None. + +Example Playbook +---------------- + +Below is an example playbook task: + +``` +--- +- hosts: all + roles: + - role: common +``` + +License +------- + +copyleft-next-0.3.1
playbooks/roles/common/defaults/main.yml new file +7 -0
@@ -0,0 +1,7 @@ +# SPDX-License-Identifier copyleft-next-0.3.1 +--- + +kdevops_data: "/data/kdevops" +kdevops_git: "https://github.com/linux-kdevops/kdevops.git" + +kdevops_git_reset: False
playbooks/roles/common/tasks/main.yml new file +23 -0
@@ -0,0 +1,23 @@ +--- +- name: Import optional extra_args file + include_vars: "{{ item }}" + ignore_errors: yes + with_first_found: + - files: + - "../extra_vars.yml" + - "../extra_vars.yaml" + - "../extra_vars.json" + skip: true + tags: vars + +# Distro agnostic stuff goes below + +- name: git reset kdevops + environment: + GIT_SSL_NO_VERIFY: true + git: + repo: "{{ kdevops_git }}" + dest: "{{ kdevops_data }}" + tags: [ 'kdevops_reset'] + when: + - kdevops_git_reset|bool
workflows/common/Makefile +12 -0
@@ -32,3 +32,15 @@ WORKFLOW_ARGS += data_user=$(WORKFLOW_DATA_USER) WORKFLOW_ARGS += data_group=$(WORKFLOW_DATA_GROUP) endif # CONFIG_WORKFLOW_INFER_USER_AND_GROUP == y + +ifeq (y,$(CONFIG_KDEVOPS_WORKFLOW_GIT_CLONES_KDEVOPS_GIT)) +kdevops-git-reset: + $(Q)ansible-playbook -f 30 -i hosts playbooks/common.yml --tags vars,kdevops_reset --extra-vars '{ kdevops_git_reset: True }' $(LIMIT_HOSTS) + +kdevops-help-menu: + @echo "Common workflow options:" + @echo "kdevops-git-reset: - Resets your kdevops git tree URL and contents on guests" + @echo + +HELP_TARGETS += kdevops-help-menu +endif # CONFIG_KDEVOPS_WORKFLOW_GIT_CLONES_KDEVOPS_GIT
Signature · 1 line
2.35.1
Re: [PATCH 3/4] playbooks: add a common playbook a git reset task for kdevops
Hi Luis,
Quoted 19 lines · Luis Chamberlain
Two playbooks share the concept of git cloning kdevops into the target nodes (guests, cloud hosts, baremetal hosts) so that expunge files can be used for avoiding tests. If you decide you want to change the URL for that git tree it may not be so obvious what to do.
Fortunately the solution is simple. You just tell ansible to use the new git tree URL. That's it. It won't remove the old directory and things work as expected.
But since we use the kdevops git tree on both fstests and blktests it is not so obvious to developers that the thing to do here is to just run 'make fstests' or 'make blktests' and even that is not as efficient as that will also re-clone the fstests or blktests tree respectively. When we just want to reset the kdevops git tree we currently have no semantics to specify that. But since this is a common post-deployment goal, just add a common playbook that let's us do common tasks.
Quoted 7 lines
All we need then is the kconfig logic to define when some commmon tasks might make sense. So to reset your kdevops git tree, all you have to do now is change the configuration for it, then run:
make
make kdevops-git-reset
While I do like the idea of having this option, I still do not understand the main use case to have it as a separate make target. Wouldn't the developer already put the custom kdevops tree with CONFIG_WORKFLOW_KDEVOPS_GIT during the initial make menuconfig phase?
I am just trying to understand the usecase when someone wants to change the kdevops tree after a test run. Maybe I am missing something here.
Signature · 1 line
Pankaj Raghav
Re: [PATCH 3/4] playbooks: add a common playbook a git reset task for kdevops
Quoted 26 lines · Pankaj Raghav
Hi Luis,
Quoted 23 lines · Luis Chamberlain
Two playbooks share the concept of git cloning kdevops into the target nodes (guests, cloud hosts, baremetal hosts) so that expunge files can be used for avoiding tests. If you decide you want to change the URL for that git tree it may not be so obvious what to do.
Fortunately the solution is simple. You just tell ansible to use the new git tree URL. That's it. It won't remove the old directory and things work as expected.
But since we use the kdevops git tree on both fstests and blktests it is not so obvious to developers that the thing to do here is to just run 'make fstests' or 'make blktests' and even that is not as efficient as that will also re-clone the fstests or blktests tree respectively. When we just want to reset the kdevops git tree we currently have no semantics to specify that. But since this is a common post-deployment goal, just add a common playbook that let's us do common tasks.
All we need then is the kconfig logic to define when some commmon tasks might make sense. So to reset your kdevops git tree, all you have to do now is change the configuration for it, then run:
Quoted 8 lines
Quoted 3 lines
make
make kdevops-git-resetWhile I do like the idea of having this option, I still do not understand the main use case to have it as a separate make target. Wouldn't the developer already put the custom kdevops tree with
CONFIG_WORKFLOW_KDEVOPS_GITduring the initial make menuconfig phase?
For initial setup yes. The value of the new make target is for when you already deployed kdevops, and now you want to change the git URL for the guests if they have workflows which clone kdevops for using expunges when testing such as with fstests and blktest.
Quoted 2 lines
I am just trying to understand the usecase when someone wants to change the kdevops tree after a test run. Maybe I am missing something here.
That is right, the use case here of the new make target is so that a user can change the target kdevops tree on the guests if they are working with fstests and blktests. Otherwise then the git tree will only change on the host. If you ran 'make fstests' for instance you git cloned fstests, compiled and installed it, but the kdevops tree was also cloned and used on each guest so to ensure only tests which are not expunged for the target test are run. Without this new make target if you wanted to reset the git tree on the guests you'd have to re-run 'make fstests'. Where as with the new target, it's just a one liner.
Luis
PATCH 4/4 kdevops: make linux-kdevops the default tree
In reply to Luis ChamberlainAt LSFMM 2022 at Palm Springs it was discussed that we should *strive* towards a shared expunge list for fstests / blktests. Although that effort requires splitting out the expunge list from kdevops to a new git subtree, sharing a git tree for commit work for kdevops seems also innevitable and desirable since some of us are already collaborating on at least one shared test runner, kdevops.
So we can start by using a shared organization for what we need to share, we call this organization linux-kdevops [0]. This encompasses a few usual suspects git trees which can be used by both fstests and blktests picking a "stable" sort of git sha1sum for each and always striving towards the latest:
* fstests
* blktests
* dbench
* ndd
We'll be using the linux-kdevops organization for what trees we use and trust that those in the organization will communicate what is needed before making controversial changes.
This is perhaps the first controversial change, but it only applies to kdevops, so if you are not using kdevops as a test runner you probably won't care.
Those using kdevops should become aware that they should change their default upstream to use linux-kdevops URL now:
So we switch from:
https://github.com/mcgrof/kdevops
To:
https://github.com/linux-kdevops/kdevops
Users which change this on a live environemtn would also then just have to change the CONFIG_WORKFLOW_KDEVOPS_GIT on their configuration so they'd just run:
make menuconfig # set CONFIG_WORKFLOW_KDEVOPS_GIT to https://github.com/linux-kdevops/kdevops
make
make kdevops-git-reset
Those who have commit access to the organization linux-kdevops can then just commit as needed to help move baselines forwards and if and when something comes up which really seems controversial we can use the mailing lists as with this change.
[0] https://github.com/orgs/linux-kdevops/
--- kconfigs/workflows/Kconfig.shared | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kconfigs/workflows/Kconfig.shared +1 -1
@@ -47,7 +47,7 @@ endif config WORKFLOW_KDEVOPS_GIT string "Git tree of kdevops to clone on targets" - default "https://github.com/mcgrof/kdevops.git" if !GIT_ALTERNATIVES && !HAVE_CUSTOM_KDEVOPS_GIT + default "https://github.com/linux-kdevops/kdevops.git" if !GIT_ALTERNATIVES && !HAVE_CUSTOM_KDEVOPS_GIT default "https://github.com/linux-kdevops/kdevops.git" if GIT_LINUX_KDEVOPS_GITHUB default "https://gitlab.com/linux-kdevops/kdevops.git" if GIT_LINUX_KDEVOPS_GITLAB default CUSTOM_KDEVOPS_GIT if HAVE_CUSTOM_KDEVOPS_GIT
Signature · 1 line
2.35.1