Skip to contentSkip to conversation

kdevops: use linux-kdevops for the main tree

PATCH In review Luis Chamberlain 3 participants 9 messages May 13, 2022 – May 23, 2022

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover kdevops: use linux-kdevops for the main tree

More just like a heads up as to the next steps from discussions at LSFMM with regards to kdevops. Those using kdevops in some environemnts will want to read the last commit.

This work starts off by making the linux-kdevops organization the default for the source of commits for the kdevops projects. We can now share efforts, and commits. The last commit will contain more details as to the motivation for this.

Luis Chamberlain (4):
workflows/Kconfig: be consistent when enabling fstests or blktests
kdevops: move generic kdevops variables to its own file
playbooks: add a common playbook a git reset task for kdevops
kdevops: make linux-kdevops the default tree

 kconfigs/workflows/Kconfig               |  6 ++++
 kconfigs/workflows/Kconfig.shared        |  2 +-
 playbooks/common.yml                     |  4 +++
 playbooks/roles/common/README.md         | 38 ++++++++++++++++++++
 playbooks/roles/common/defaults/main.yml |  7 ++++
 playbooks/roles/common/tasks/main.yml    | 23 ++++++++++++
 workflows/Makefile                       | 33 +----------------
 workflows/common/Makefile                | 46 ++++++++++++++++++++++++
 8 files changed, 126 insertions(+), 33 deletions(-)
 create mode 100644 playbooks/common.yml
 create mode 100644 playbooks/roles/common/README.md
 create mode 100644 playbooks/roles/common/defaults/main.yml
 create mode 100644 playbooks/roles/common/tasks/main.yml
 create mode 100644 workflows/common/Makefile
Signature · 1 line
2.35.1
#1 lore raw

PATCH 1/4 workflows/Kconfig: be consistent when enabling fstests or blktests

We have two kconfig variables which we use to be able to express when we are going to enable fstests or blktests, either as a dedicated set of tests or when we want to enable testing both fstests and blktests in one system. But right now we only select this kconfig variable when we are using a dedicated system. This is not an issue as the kconfig is a kconfig symbols are bools which are set default to y if either the test is dedicated or not.

But to be pedantic, and clear, let's make sure the tests select the respective kconfig for each case as we'd expect to see it. Otherwise this can confuse folks reading this.

---
 kconfigs/workflows/Kconfig | 2 ++
 1 file changed, 2 insertions(+)
kconfigs/workflows/Kconfig +2 -0
index 7e5c518..7f71470 100644
@@ -133,6 +133,7 @@ if !WORKFLOWS_DEDICATED_WORKFLOW
 
 config KDEVOPS_WORKFLOW_NOT_DEDICATED_ENABLE_FSTESTS
 	bool "fstests"
+	select KDEVOPS_WORKFLOW_ENABLE_FSTESTS
 	help
 	  Select this option if you are doing filesystem development and want
 	  to target development for a filesystem and enable fstests so that
@@ -146,6 +147,7 @@ config KDEVOPS_WORKFLOW_NOT_DEDICATED_ENABLE_FSTESTS
 
 config KDEVOPS_WORKFLOW_NOT_DEDICATED_ENABLE_BLKTESTS
 	bool "blktests"
+	select KDEVOPS_WORKFLOW_ENABLE_BLKTESTS
 	help
 	  Select this option if you are doing block layer development and want
 	  to run blktests. The git tree for blktests will be git clone and
Signature · 1 line
2.35.1
#4 lore raw

PATCH 2/4 kdevops: move generic kdevops variables to its own file

In reply to Luis Chamberlain

There are shared kdevops variables between different workflows, which *can* be used by any workflow. Move these into a generic kdevops helper Makefile, as we can later expand on this.

This makes no functional changes.

---
 workflows/Makefile        | 33 +--------------------------------
 workflows/common/Makefile | 34 ++++++++++++++++++++++++++++++++++
 2 files changed, 35 insertions(+), 32 deletions(-)
 create mode 100644 workflows/common/Makefile
workflows/Makefile +1 -32
index 928e42f..033ffc7 100644
@@ -1,37 +1,6 @@
 # SPDX-License-Identifier: copyleft-next-0.3.1
 
-# How we create the partition for the workflow data partition
-WORKFLOW_DATA_DEVICE:=$(subst ",,$(CONFIG_WORKFLOW_DATA_DEVICE))
-WORKFLOW_DATA_PATH:=$(subst ",,$(CONFIG_WORKFLOW_DATA_PATH))
-WORKFLOW_DATA_FSTYPE:=$(subst ",,$(CONFIG_WORKFLOW_DATA_FSTYPE))
-WORKFLOW_DATA_LABEL:=$(subst ",,$(CONFIG_WORKFLOW_DATA_LABEL))
-
-WORKFLOW_KDEVOPS_GIT:=$(subst ",,$(CONFIG_WORKFLOW_KDEVOPS_GIT))
-WORKFLOW_KDEVOPS_GIT_DATA:=$(subst ",,$(CONFIG_WORKFLOW_KDEVOPS_GIT_DATA))
-WORKFLOW_KDEVOPS_DIR:=$(subst ",,$(CONFIG_WORKFLOW_KDEVOPS_DIR))
-
-WORKFLOW_ARGS	+= data_device=$(WORKFLOW_DATA_DEVICE)
-WORKFLOW_ARGS	+= data_path=$(WORKFLOW_DATA_PATH)
-WORKFLOW_ARGS	+= data_fstype=$(WORKFLOW_DATA_FSTYPE)
-WORKFLOW_ARGS	+= data_label=$(WORKFLOW_DATA_LABEL)
-WORKFLOW_ARGS	+= kdevops_git=$(WORKFLOW_KDEVOPS_GIT)
-WORKFLOW_ARGS	+= kdevops_data=\"$(WORKFLOW_KDEVOPS_GIT_DATA)\"
-WORKFLOW_ARGS	+= kdevops_dir=\"$(WORKFLOW_KDEVOPS_DIR)\"
-
-ifeq (y,$(CONFIG_WORKFLOW_MAKE_CMD_OVERRIDE))
-WORKFLOW_MAKE_CMD:=$(subst ",,$(CONFIG_WORKFLOW_MAKE_CMD))
-endif
-
-ifeq (y,$(CONFIG_WORKFLOW_INFER_USER_AND_GROUP))
-WORKFLOW_ARGS	+= infer_uid_and_group=True
-else
-WORKFLOW_DATA_USER:=$(subst ",,$(CONFIG_WORKFLOW_DATA_USER))
-WORKFLOW_DATA_GROUP:=$(subst ",,$(CONFIG_WORKFLOW_DATA_GROUP))
-
-WORKFLOW_ARGS	+= data_user=$(WORKFLOW_DATA_USER)
-WORKFLOW_ARGS	+= data_group=$(WORKFLOW_DATA_GROUP)
-
-endif # CONFIG_WORKFLOW_MAKE_CMD_OVERRIDE == y
+include workflows/common/Makefile
 
 BOOTLINUX_ARGS	:=
 ifeq (y,$(CONFIG_BOOTLINUX))
workflows/common/Makefile new file +34 -0
new file mode 100644
index 0000000..da21d78
@@ -0,0 +1,34 @@
+# SPDX-License-Identifier: copyleft-next-0.3.1
+
+# How we create the partition for the workflow data partition
+WORKFLOW_DATA_DEVICE:=$(subst ",,$(CONFIG_WORKFLOW_DATA_DEVICE))
+WORKFLOW_DATA_PATH:=$(subst ",,$(CONFIG_WORKFLOW_DATA_PATH))
+WORKFLOW_DATA_FSTYPE:=$(subst ",,$(CONFIG_WORKFLOW_DATA_FSTYPE))
+WORKFLOW_DATA_LABEL:=$(subst ",,$(CONFIG_WORKFLOW_DATA_LABEL))
+
+WORKFLOW_KDEVOPS_GIT:=$(subst ",,$(CONFIG_WORKFLOW_KDEVOPS_GIT))
+WORKFLOW_KDEVOPS_GIT_DATA:=$(subst ",,$(CONFIG_WORKFLOW_KDEVOPS_GIT_DATA))
+WORKFLOW_KDEVOPS_DIR:=$(subst ",,$(CONFIG_WORKFLOW_KDEVOPS_DIR))
+
+WORKFLOW_ARGS	+= data_device=$(WORKFLOW_DATA_DEVICE)
+WORKFLOW_ARGS	+= data_path=$(WORKFLOW_DATA_PATH)
+WORKFLOW_ARGS	+= data_fstype=$(WORKFLOW_DATA_FSTYPE)
+WORKFLOW_ARGS	+= data_label=$(WORKFLOW_DATA_LABEL)
+WORKFLOW_ARGS	+= kdevops_git=$(WORKFLOW_KDEVOPS_GIT)
+WORKFLOW_ARGS	+= kdevops_data=\"$(WORKFLOW_KDEVOPS_GIT_DATA)\"
+WORKFLOW_ARGS	+= kdevops_dir=\"$(WORKFLOW_KDEVOPS_DIR)\"
+
+ifeq (y,$(CONFIG_WORKFLOW_MAKE_CMD_OVERRIDE))
+WORKFLOW_MAKE_CMD:=$(subst ",,$(CONFIG_WORKFLOW_MAKE_CMD))
+endif
+
+ifeq (y,$(CONFIG_WORKFLOW_INFER_USER_AND_GROUP))
+WORKFLOW_ARGS	+= infer_uid_and_group=True
+else
+WORKFLOW_DATA_USER:=$(subst ",,$(CONFIG_WORKFLOW_DATA_USER))
+WORKFLOW_DATA_GROUP:=$(subst ",,$(CONFIG_WORKFLOW_DATA_GROUP))
+
+WORKFLOW_ARGS	+= data_user=$(WORKFLOW_DATA_USER)
+WORKFLOW_ARGS	+= data_group=$(WORKFLOW_DATA_GROUP)
+
+endif # CONFIG_WORKFLOW_INFER_USER_AND_GROUP == y
Signature · 1 line
2.35.1
#5 lore raw

PATCH 3/4 playbooks: add a common playbook a git reset task for kdevops

In reply to Luis Chamberlain

Two playbooks share the concept of git cloning kdevops into the target nodes (guests, cloud hosts, baremetal hosts) so that expunge files can be used for avoiding tests. If you decide you want to change the URL for that git tree it may not be so obvious what to do.

Fortunately the solution is simple. You just tell ansible to use the new git tree URL. That's it. It won't remove the old directory and things work as expected.

But since we use the kdevops git tree on both fstests and blktests it is not so obvious to developers that the thing to do here is to just run 'make fstests' or 'make blktests' and even that is not as efficient as that will also re-clone the fstests or blktests tree respectively. When we just want to reset the kdevops git tree we currently have no semantics to specify that. But since this is a common post-deployment goal, just add a common playbook that let's us do common tasks.

All we need then is the kconfig logic to define when some commmon tasks might make sense. So to reset your kdevops git tree, all you have to do now is change the configuration for it, then run:

make
make kdevops-git-reset

---
 kconfigs/workflows/Kconfig               |  4 +++
 playbooks/common.yml                     |  4 +++
 playbooks/roles/common/README.md         | 38 ++++++++++++++++++++++++
 playbooks/roles/common/defaults/main.yml |  7 +++++
 playbooks/roles/common/tasks/main.yml    | 23 ++++++++++++++
 workflows/common/Makefile                | 12 ++++++++
 6 files changed, 88 insertions(+)
 create mode 100644 playbooks/common.yml
 create mode 100644 playbooks/roles/common/README.md
 create mode 100644 playbooks/roles/common/defaults/main.yml
 create mode 100644 playbooks/roles/common/tasks/main.yml
kconfigs/workflows/Kconfig +4 -0
index 7f71470..817335b 100644
@@ -175,6 +175,10 @@ source "workflows/blktests/Kconfig"
 endmenu
 endif # KDEVOPS_WORKFLOW_ENABLE_BLKTESTS
 
+config KDEVOPS_WORKFLOW_GIT_CLONES_KDEVOPS_GIT
+	bool
+	default y if KDEVOPS_WORKFLOW_ENABLE_FSTESTS || KDEVOPS_WORKFLOW_ENABLE_BLKTESTS
+
 endif # WORKFLOWS_LINUX_TESTS
 
 endif # WORKFLOWS_TESTS
playbooks/common.yml new file +4 -0
new file mode 100644
index 0000000..48485e3
@@ -0,0 +1,4 @@
+---
+- hosts: all
+  roles:
+    - role: common
playbooks/roles/common/README.md new file +38 -0
new file mode 100644
index 0000000..2b0084c
@@ -0,0 +1,38 @@
+common
+======
+
+The common role lets you add tasks which is commmon to all workflows.
+Without this we would be duplicating code.
+
+Requirements
+------------
+
+None.
+
+Role Variables
+--------------
+
+  * kdevops_git_reset: perform a git reset. This is useful in case you want
+	to change the URL you use for kdevops.
+
+Dependencies
+------------
+
+None.
+
+Example Playbook
+----------------
+
+Below is an example playbook task:
+
+```
+---
+- hosts: all
+  roles:
+    - role: common
+```
+
+License
+-------
+
+copyleft-next-0.3.1
playbooks/roles/common/defaults/main.yml new file +7 -0
new file mode 100644
index 0000000..69cd0af
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier copyleft-next-0.3.1
+---
+
+kdevops_data: "/data/kdevops"
+kdevops_git: "https://github.com/linux-kdevops/kdevops.git"
+
+kdevops_git_reset: False
playbooks/roles/common/tasks/main.yml new file +23 -0
new file mode 100644
index 0000000..4482349
@@ -0,0 +1,23 @@
+---
+- name: Import optional extra_args file
+  include_vars: "{{ item }}"
+  ignore_errors: yes
+  with_first_found:
+    - files:
+      - "../extra_vars.yml"
+      - "../extra_vars.yaml"
+      - "../extra_vars.json"
+      skip: true
+  tags: vars
+
+# Distro agnostic stuff goes below
+
+- name: git reset kdevops
+  environment:
+    GIT_SSL_NO_VERIFY:  true
+  git:
+    repo: "{{ kdevops_git }}"
+    dest: "{{ kdevops_data }}"
+  tags: [ 'kdevops_reset']
+  when:
+    - kdevops_git_reset|bool
workflows/common/Makefile +12 -0
index da21d78..6596ed1 100644
@@ -32,3 +32,15 @@ WORKFLOW_ARGS	+= data_user=$(WORKFLOW_DATA_USER)
 WORKFLOW_ARGS	+= data_group=$(WORKFLOW_DATA_GROUP)
 
 endif # CONFIG_WORKFLOW_INFER_USER_AND_GROUP == y
+
+ifeq (y,$(CONFIG_KDEVOPS_WORKFLOW_GIT_CLONES_KDEVOPS_GIT))
+kdevops-git-reset:
+	$(Q)ansible-playbook -f 30 -i hosts playbooks/common.yml --tags vars,kdevops_reset --extra-vars '{ kdevops_git_reset: True }' $(LIMIT_HOSTS)
+
+kdevops-help-menu:
+	@echo "Common workflow options:"
+	@echo "kdevops-git-reset:                      - Resets your kdevops git tree URL and contents on guests"
+	@echo
+
+HELP_TARGETS += kdevops-help-menu
+endif # CONFIG_KDEVOPS_WORKFLOW_GIT_CLONES_KDEVOPS_GIT
Signature · 1 line
2.35.1
#8 lore raw

PATCH 4/4 kdevops: make linux-kdevops the default tree

In reply to Luis Chamberlain

At LSFMM 2022 at Palm Springs it was discussed that we should *strive* towards a shared expunge list for fstests / blktests. Although that effort requires splitting out the expunge list from kdevops to a new git subtree, sharing a git tree for commit work for kdevops seems also innevitable and desirable since some of us are already collaborating on at least one shared test runner, kdevops.

So we can start by using a shared organization for what we need to share, we call this organization linux-kdevops [0]. This encompasses a few usual suspects git trees which can be used by both fstests and blktests picking a "stable" sort of git sha1sum for each and always striving towards the latest:

* fstests
* blktests
* dbench
* ndd

We'll be using the linux-kdevops organization for what trees we use and trust that those in the organization will communicate what is needed before making controversial changes.

This is perhaps the first controversial change, but it only applies to kdevops, so if you are not using kdevops as a test runner you probably won't care.

Those using kdevops should become aware that they should change their default upstream to use linux-kdevops URL now:

So we switch from:
https://github.com/mcgrof/kdevops
To:
https://github.com/linux-kdevops/kdevops

Users which change this on a live environemtn would also then just have to change the CONFIG_WORKFLOW_KDEVOPS_GIT on their configuration so they'd just run:

make menuconfig # set CONFIG_WORKFLOW_KDEVOPS_GIT to https://github.com/linux-kdevops/kdevops
make
make kdevops-git-reset

Those who have commit access to the organization linux-kdevops can then just commit as needed to help move baselines forwards and if and when something comes up which really seems controversial we can use the mailing lists as with this change.

[0] https://github.com/orgs/linux-kdevops/

---
 kconfigs/workflows/Kconfig.shared | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kconfigs/workflows/Kconfig.shared +1 -1
index 4cccfbe..4a2ecf6 100644
@@ -47,7 +47,7 @@ endif
 
 config WORKFLOW_KDEVOPS_GIT
 	string "Git tree of kdevops to clone on targets"
-	default "https://github.com/mcgrof/kdevops.git" if !GIT_ALTERNATIVES && !HAVE_CUSTOM_KDEVOPS_GIT
+	default "https://github.com/linux-kdevops/kdevops.git" if !GIT_ALTERNATIVES && !HAVE_CUSTOM_KDEVOPS_GIT
 	default "https://github.com/linux-kdevops/kdevops.git" if GIT_LINUX_KDEVOPS_GITHUB
 	default "https://gitlab.com/linux-kdevops/kdevops.git" if GIT_LINUX_KDEVOPS_GITLAB
 	default CUSTOM_KDEVOPS_GIT if HAVE_CUSTOM_KDEVOPS_GIT
Signature · 1 line
2.35.1
#2 lore raw

Re: [PATCH 1/4] workflows/Kconfig: be consistent when enabling fstests or blktests

In reply to Luis Chamberlain
Quoted 11 lines · Luis Chamberlain

We have two kconfig variables which we use to be able to express when we are going to enable fstests or blktests, either as a dedicated set of tests or when we want to enable testing both fstests and blktests in one system. But right now we only select this kconfig variable when we are using a dedicated system. This is not an issue as the kconfig is a kconfig symbols are bools which are set default to y if either the test is dedicated or not.

But to be pedantic, and clear, let's make sure the tests select the respective kconfig for each case as we'd expect to see it. Otherwise this can confuse folks reading this.

Is this patch perhaps intended for the kdevops project? If so, please add a prefix to make this clear (git format-patch --subject-prefix) when sending kdevops patches to Linux kernel mailing lists.

Thanks,

Bart.

#3 lore raw

Re: [PATCH 1/4] workflows/Kconfig: be consistent when enabling fstests or blktests

Quoted 9 lines · Bart Van Assche
Quoted 8 lines · Luis Chamberlain

We have two kconfig variables which we use to be able to express when we are going to enable fstests or blktests, either as a dedicated set of tests or when we want to enable testing both fstests and blktests in one system. But right now we only select this kconfig variable when we are using a dedicated system. This is not an issue as the kconfig is a kconfig symbols are bools which are set default to y if either the test is dedicated or not.

Quoted 7 lines
Quoted 3 lines

But to be pedantic, and clear, let's make sure the tests select the respective kconfig for each case as we'd expect to see it. Otherwise this can confuse folks reading this.

Is this patch perhaps intended for the kdevops project? If so, please add a prefix to make this clear (git format-patch --subject-prefix) when sending kdevops patches to Linux kernel mailing lists.

Yeah good idea for it to go into the [PATCH kdevops]. Thanks!

Luis

#6 lore raw

Re: [PATCH 3/4] playbooks: add a common playbook a git reset task for kdevops

In reply to Luis Chamberlain

Hi Luis,

Quoted 19 lines · Luis Chamberlain

Two playbooks share the concept of git cloning kdevops into the target nodes (guests, cloud hosts, baremetal hosts) so that expunge files can be used for avoiding tests. If you decide you want to change the URL for that git tree it may not be so obvious what to do.

Fortunately the solution is simple. You just tell ansible to use the new git tree URL. That's it. It won't remove the old directory and things work as expected.

But since we use the kdevops git tree on both fstests and blktests it is not so obvious to developers that the thing to do here is to just run 'make fstests' or 'make blktests' and even that is not as efficient as that will also re-clone the fstests or blktests tree respectively. When we just want to reset the kdevops git tree we currently have no semantics to specify that. But since this is a common post-deployment goal, just add a common playbook that let's us do common tasks.

Quoted 7 lines

All we need then is the kconfig logic to define when some commmon tasks might make sense. So to reset your kdevops git tree, all you have to do now is change the configuration for it, then run:

make
make kdevops-git-reset

While I do like the idea of having this option, I still do not understand the main use case to have it as a separate make target. Wouldn't the developer already put the custom kdevops tree with CONFIG_WORKFLOW_KDEVOPS_GIT during the initial make menuconfig phase?

I am just trying to understand the usecase when someone wants to change the kdevops tree after a test run. Maybe I am missing something here.

Signature · 1 line
Pankaj Raghav
#7 lore raw

Re: [PATCH 3/4] playbooks: add a common playbook a git reset task for kdevops

Quoted 26 lines · Pankaj Raghav

Hi Luis,

Quoted 23 lines · Luis Chamberlain

Two playbooks share the concept of git cloning kdevops into the target nodes (guests, cloud hosts, baremetal hosts) so that expunge files can be used for avoiding tests. If you decide you want to change the URL for that git tree it may not be so obvious what to do.

Fortunately the solution is simple. You just tell ansible to use the new git tree URL. That's it. It won't remove the old directory and things work as expected.

But since we use the kdevops git tree on both fstests and blktests it is not so obvious to developers that the thing to do here is to just run 'make fstests' or 'make blktests' and even that is not as efficient as that will also re-clone the fstests or blktests tree respectively. When we just want to reset the kdevops git tree we currently have no semantics to specify that. But since this is a common post-deployment goal, just add a common playbook that let's us do common tasks.

All we need then is the kconfig logic to define when some commmon tasks might make sense. So to reset your kdevops git tree, all you have to do now is change the configuration for it, then run:

Quoted 8 lines
Quoted 3 lines

make
make kdevops-git-reset

While I do like the idea of having this option, I still do not understand the main use case to have it as a separate make target. Wouldn't the developer already put the custom kdevops tree with CONFIG_WORKFLOW_KDEVOPS_GIT during the initial make menuconfig phase?

For initial setup yes. The value of the new make target is for when you already deployed kdevops, and now you want to change the git URL for the guests if they have workflows which clone kdevops for using expunges when testing such as with fstests and blktest.

Quoted 2 lines

I am just trying to understand the usecase when someone wants to change the kdevops tree after a test run. Maybe I am missing something here.

That is right, the use case here of the new make target is so that a user can change the target kdevops tree on the guests if they are working with fstests and blktests. Otherwise then the git tree will only change on the host. If you ran 'make fstests' for instance you git cloned fstests, compiled and installed it, but the kdevops tree was also cloned and used on each guest so to ensure only tests which are not expunged for the target test are run. Without this new make target if you wanted to reset the git tree on the guests you'd have to re-run 'make fstests'. Where as with the new target, it's just a one liner.

Luis