Skip to contentSkip to conversation

linux: generate git refs automatically

PATCH Superseded Daniel Gomez 2 participants 4 messages Jun 3, 2024 – Jun 5, 2024 1 message not in the archive Kbuild

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): CGME20240603203059eucas1p24a51c04a7c2bb86fe325fba0ad4b28ba@eucas1p2.samsung.com

#1 lore raw

v3 cover linux: generate git refs automatically

This adds support to generate git references automatically using git-ls-remote and kernel.org/releases.json.

Questions:

* Default behaviour (make) will generate Kconfigs for linux, next and mainline repos. Using make dynconfig will force and regenerate these files with the latest 15 references. Do we need this 15 refs or should we target latest-and-greatest only for these 3 repos?

* This patch includes autogenerated files for all trees (mcgrof-linus, mcgrof-next...) including only the static configuration. Running make dynconfig will force an update of these files with latest 15 references. Since autogenerated files are part of the git tree, kdevops users will see that some files have changed with git status. Any suggestion on how to improve this path? Not including the files would make 'make' fail as it will try to include the missing files. An alternative could be to create empty files on the way (to avoid the file creation delay) and force the user to run 'make dynconfig' in case these refs need to be generated.

* Default behaviour will update linux, next and mainline after 24h hours since last file modification. If the user has overwritten these files with make dynconfig, the values will be lost. Do we want a different behaviour here?

Daniel

Changes since v2:
* Rebase changes to latest kdevops.
* Add parser support for kernel.org/releases.json
* Add generate_refs.py subcommand support: gitref and kreleases.
* gitref: Retrieves refs from upstream repository.
* kreleases: Retrieves latest-and-greatest refs from kernel.org/releases.
* Integrate yaml with static configuration for mcgrof, jlayton and cel trees.
* Generate a default configuration file for all trees except for linus, next and stable.
* Kconfig.refs-{linus,next,stable} refs will automatically retrieve the information from kernel.org/releases.json every time make is run.
* Kconfig.refs-* files will be generated/forced when make dynconfig is run.
* Minor script changes: e.g. special case handling for CUSTOM entries.
* Minor Kconfig/Makefile renaming/cleanup.

Changes since v1:
* Script generation moved and called from the main Makefile.
* scripts/gen-refs.Makefile to generate refs.
* Conversion to python.
* Added support for:
* Command-line arguments to control options.
* YAML parser to configure extra and static references. Example added for mcgrof and mcgrof-next.
* Filter for tags and heads. Example to filter semantic versioning would be --filter-tags "*.*.*".
* File generation based on time. Update it every 24h (hardcoded). Possibility to use --force to always generate the file.
* git-ls-remote sorting inverted using --sort option and git versionsort.suffix option (to avoid sort --reverse).

Daniel Gomez (1):
linux: generate refs automatically

 .gitignore                                 |   2 +
 Makefile                                   |   3 +
 defconfigs/nfsd                            |   2 +-
 scripts/dynamic-kconfig.Makefile           |   4 +
 scripts/gen-gitrefs.Makefile               | 111 ++++++++
 scripts/gen-kreleases.Makefile             |  25 ++
 scripts/generate_refs.py                   | 304 +++++++++++++++++++++
 workflows/linux/Kconfig                    |  32 +--
 workflows/linux/Kconfig.btrfs              |   4 +-
 workflows/linux/Kconfig.cel                |  31 +--
 workflows/linux/Kconfig.jlayton            |  31 +--
 workflows/linux/Kconfig.kdevops            |  17 +-
 workflows/linux/Kconfig.linus              |  69 +----
 workflows/linux/Kconfig.mcgrof             |  48 +---
 workflows/linux/Kconfig.next               |   4 +-
 workflows/linux/Kconfig.refs-btrfs-devel   |  15 +
 workflows/linux/Kconfig.refs-cel-linux     |  27 ++
 workflows/linux/Kconfig.refs-jlayton-linux |  27 ++
 workflows/linux/Kconfig.refs-kdevops-linus |  15 +
 workflows/linux/Kconfig.refs-mcgrof-linus  |  21 ++
 workflows/linux/Kconfig.refs-mcgrof-next   |  21 ++
 workflows/linux/Kconfig.stable             |  63 +----
 workflows/linux/btrfs-devel-refs.yaml      |   6 +
 workflows/linux/cel-linux-refs.yaml        |  14 +
 workflows/linux/jlayton-linux-refs.yaml    |  14 +
 workflows/linux/kdevops-linus-refs.yaml    |   6 +
 workflows/linux/mcgrof-linus-refs.yaml     |  10 +
 workflows/linux/mcgrof-next-refs.yaml      |  10 +
 28 files changed, 669 insertions(+), 267 deletions(-)
 create mode 100644 scripts/gen-gitrefs.Makefile
 create mode 100644 scripts/gen-kreleases.Makefile
 create mode 100755 scripts/generate_refs.py
 create mode 100644 workflows/linux/Kconfig.refs-btrfs-devel
 create mode 100644 workflows/linux/Kconfig.refs-cel-linux
 create mode 100644 workflows/linux/Kconfig.refs-jlayton-linux
 create mode 100644 workflows/linux/Kconfig.refs-kdevops-linus
 create mode 100644 workflows/linux/Kconfig.refs-mcgrof-linus
 create mode 100644 workflows/linux/Kconfig.refs-mcgrof-next
 create mode 100644 workflows/linux/btrfs-devel-refs.yaml
 create mode 100644 workflows/linux/cel-linux-refs.yaml
 create mode 100644 workflows/linux/jlayton-linux-refs.yaml
 create mode 100644 workflows/linux/kdevops-linus-refs.yaml
 create mode 100644 workflows/linux/mcgrof-linus-refs.yaml
 create mode 100644 workflows/linux/mcgrof-next-refs.yaml
Signature · 1 line
2.43.0
#2 lore raw

PATCH v3 1/1 linux: generate refs automatically

Generate a Kconfig choice/endchoice menu with a list of the latest git references for upstream repositories.

* Default behaviour: latest-and-greatest.
The script included (generate_refs.py) will create (if it does not exist) the Kconfig choice options with the latest and greatest kernel release references provided by kernel.org/releases.json. Every time make is run, the script will check for the date of the last modification, and will force an update if more than 24h has passed since last modification. The generated files are: Kconfig.refs-{next,stable,linus}.

* make dynconfig:
In addition, the script will generate and overwrite all the existing Kconfig.refs-* files (including next, stable and linus) when dynconfig is run. This mode will retrieve the information from the upstream git repoitories for all supported kernel trees in kdevops and will create a list with the latest 15 references found.

---
 .gitignore                                 |   2 +
 Makefile                                   |   3 +
 defconfigs/nfsd                            |   2 +-
 scripts/dynamic-kconfig.Makefile           |   4 +
 scripts/gen-gitrefs.Makefile               | 111 ++++++++
 scripts/gen-kreleases.Makefile             |  25 ++
 scripts/generate_refs.py                   | 304 +++++++++++++++++++++
 workflows/linux/Kconfig                    |  32 +--
 workflows/linux/Kconfig.btrfs              |   4 +-
 workflows/linux/Kconfig.cel                |  31 +--
 workflows/linux/Kconfig.jlayton            |  31 +--
 workflows/linux/Kconfig.kdevops            |  17 +-
 workflows/linux/Kconfig.linus              |  69 +----
 workflows/linux/Kconfig.mcgrof             |  48 +---
 workflows/linux/Kconfig.next               |   4 +-
 workflows/linux/Kconfig.refs-btrfs-devel   |  15 +
 workflows/linux/Kconfig.refs-cel-linux     |  27 ++
 workflows/linux/Kconfig.refs-jlayton-linux |  27 ++
 workflows/linux/Kconfig.refs-kdevops-linus |  15 +
 workflows/linux/Kconfig.refs-mcgrof-linus  |  21 ++
 workflows/linux/Kconfig.refs-mcgrof-next   |  21 ++
 workflows/linux/Kconfig.stable             |  63 +----
 workflows/linux/btrfs-devel-refs.yaml      |   6 +
 workflows/linux/cel-linux-refs.yaml        |  14 +
 workflows/linux/jlayton-linux-refs.yaml    |  14 +
 workflows/linux/kdevops-linus-refs.yaml    |   6 +
 workflows/linux/mcgrof-linus-refs.yaml     |  10 +
 workflows/linux/mcgrof-next-refs.yaml      |  10 +
 28 files changed, 669 insertions(+), 267 deletions(-)
 create mode 100644 scripts/gen-gitrefs.Makefile
 create mode 100644 scripts/gen-kreleases.Makefile
 create mode 100755 scripts/generate_refs.py
 create mode 100644 workflows/linux/Kconfig.refs-btrfs-devel
 create mode 100644 workflows/linux/Kconfig.refs-cel-linux
 create mode 100644 workflows/linux/Kconfig.refs-jlayton-linux
 create mode 100644 workflows/linux/Kconfig.refs-kdevops-linus
 create mode 100644 workflows/linux/Kconfig.refs-mcgrof-linus
 create mode 100644 workflows/linux/Kconfig.refs-mcgrof-next
 create mode 100644 workflows/linux/btrfs-devel-refs.yaml
 create mode 100644 workflows/linux/cel-linux-refs.yaml
 create mode 100644 workflows/linux/jlayton-linux-refs.yaml
 create mode 100644 workflows/linux/kdevops-linus-refs.yaml
 create mode 100644 workflows/linux/mcgrof-linus-refs.yaml
 create mode 100644 workflows/linux/mcgrof-next-refs.yaml
.gitignore +2 -0
index 3a1ecd4..163570a 100644
@@ -80,6 +80,8 @@ playbooks/roles/linux-mirror/linux-mirror-systemd/xfsprogs-mirror.timer
 #   yet.
 workflows/selftests/results/
 
+workflows/linux/Kconfig.refs-*
+
 .kernel-ci.*
 .kotd.*
 
Makefile +3 -0
index 266eccd..1678103 100644
@@ -173,6 +173,9 @@ endif
 
 include scripts/gen-hosts.Makefile
 include scripts/gen-nodes.Makefile
+ifneq ($(MAKECMDGOALS),dynconfig)
+	include scripts/gen-kreleases.Makefile
+endif
 
 # disable built-in rules for this
 .SUFFIXES:
defconfigs/nfsd +1 -1
index 796c1e4..2bac0f6 100644
@@ -279,7 +279,7 @@ CONFIG_BOOTLINUX_TREE_NEXT=y
 # CONFIG_BOOTLINUX_TREE_BTRFS_DEVEL is not set
 # CONFIG_BOOTLINUX_TREE_JLAYTON_LINUX is not set
 # CONFIG_BOOTLINUX_TREE_MCGROF_NEXT is not set
-CONFIG_BOOTLINUX_TREE_NEXT_TAG="next-20240108"
+CONFIG_BOOTLINUX_TREE_NEXT_REF="next-20240108"
 CONFIG_BOOTLINUX_TREE_NEXT_NAME="linux-next"
 CONFIG_BOOTLINUX_TREE_NEXT_URL="/mirror/linux-next.git"
 CONFIG_BOOTLINUX_TREE_NAME="linux-next"
scripts/dynamic-kconfig.Makefile +4 -0
index bdeb499..6fb5749 100644
@@ -17,6 +17,10 @@ dynamic-kconfig-help:
 
 PHONY += dynamic-kconfig-help
 
+ifeq ($(MAKECMDGOALS),dynconfig)
+	include scripts/gen-gitrefs.Makefile
+endif
+
 dynconfig:
 	$(Q)$(MAKE) dynconfig-pci
 
scripts/gen-gitrefs.Makefile new file +111 -0

111 lines not on this page · raw message

scripts/gen-kreleases.Makefile new file +25 -0
new file mode 100644
index 0000000..103915c
@@ -0,0 +1,25 @@
+# SPDX-License-Identifier: copyleft-next-0.3.1
+
+GENERATED_REFS_NEXT := $(shell \
+	./scripts/generate_refs.py \
+		--prefix BOOTLINUX_TREE_NEXT \
+		--output workflows/linux/Kconfig.refs-next \
+		kreleases \
+		--moniker linux-next \
+	)
+
+GENERATED_REFS_STABLE := $(shell \
+	./scripts/generate_refs.py \
+		--prefix BOOTLINUX_TREE_STABLE \
+		--output workflows/linux/Kconfig.refs-stable \
+		kreleases \
+		--moniker stable \
+	)
+
+GENERATED_REFS_LINUS := $(shell \
+	./scripts/generate_refs.py \
+		--prefix BOOTLINUX_TREE_LINUS \
+		--output workflows/linux/Kconfig.refs-linus \
+		kreleases \
+		--moniker mainline \
+	)
scripts/generate_refs.py new file +304 -0

304 lines not on this page · raw message

workflows/linux/Kconfig +16 -16
index 650fc05..7de1242 100644
@@ -171,10 +171,10 @@ config BOOTLINUX_TREE_NAME
 	default BOOTLINUX_TREE_STABLE_NAME if BOOTLINUX_TREE_STABLE
 	default BOOTLINUX_TREE_NEXT_NAME if BOOTLINUX_TREE_NEXT
 	default BOOTLINUX_TREE_BTRFS_DEVEL_NAME if BOOTLINUX_TREE_BTRFS_DEVEL
-	default BOOTLINUX_KDEVOPS_TREE_NAME if BOOTLINUX_TREE_KDEVOPS_LINUS
-	default BOOTLINUX_MCGROF_TREE_NAME if BOOTLINUX_MCGROF_TREE
-	default BOOTLINUX_JLAYTON_TREE_NAME if BOOTLINUX_TREE_JLAYTON_LINUX
-	default BOOTLINUX_CEL_TREE_NAME if BOOTLINUX_TREE_CEL_LINUX
+	default BOOTLINUX_TREE_KDEVOPS_NAME if BOOTLINUX_TREE_KDEVOPS_LINUS
+	default BOOTLINUX_TREE_MCGROF_NAME if BOOTLINUX_TREE_MCGROF
+	default BOOTLINUX_TREE_JLAYTON_NAME if BOOTLINUX_TREE_JLAYTON_LINUX
+	default BOOTLINUX_TREE_CEL_NAME if BOOTLINUX_TREE_CEL_LINUX
 	default BOOTLINUX_CUSTOM_NAME if BOOTLINUX_CUSTOM
 
 config BOOTLINUX_TREE
@@ -182,11 +182,11 @@ config BOOTLINUX_TREE
 	default BOOTLINUX_TREE_LINUS_URL if BOOTLINUX_TREE_LINUS
 	default BOOTLINUX_TREE_STABLE_URL if BOOTLINUX_TREE_STABLE
 	default BOOTLINUX_TREE_NEXT_URL if BOOTLINUX_TREE_NEXT
-	default BOOTLINUX_MCGROF_TREE_URL if BOOTLINUX_MCGROF_TREE
+	default BOOTLINUX_TREE_MCGROF_URL if BOOTLINUX_TREE_MCGROF
 	default BOOTLINUX_TREE_KDEVOPS_URL if BOOTLINUX_TREE_KDEVOPS_LINUS
-	default BOOTLINUX_TREE_BTRFS_TREE_URL if BOOTLINUX_TREE_BTRFS_DEVEL
-	default BOOTLINUX_JLAYTON_TREE_URL if BOOTLINUX_TREE_JLAYTON_LINUX
-	default BOOTLINUX_CEL_TREE_URL if BOOTLINUX_TREE_CEL_LINUX
+	default BOOTLINUX_TREE_BTRFS_URL if BOOTLINUX_TREE_BTRFS_DEVEL
+	default BOOTLINUX_TREE_JLAYTON_URL if BOOTLINUX_TREE_JLAYTON_LINUX
+	default BOOTLINUX_TREE_CEL_URL if BOOTLINUX_TREE_CEL_LINUX
 	default BOOTLINUX_CUSTOM_URL if BOOTLINUX_CUSTOM
 
 config BOOTLINUX_TEST_MESSAGE_ID_SET_BY_CLI
@@ -224,14 +224,14 @@ config BOOTLINUX_TEST_MESSAGE_ID_INSTALL_B4
 
 config BOOTLINUX_TREE_TAG
 	string
-	default BOOTLINUX_TREE_LINUS_TAG if BOOTLINUX_TREE_LINUS
-	default BOOTLINUX_TREE_STABLE_TAG if BOOTLINUX_STABLE
-	default BOOTLINUX_TREE_NEXT_TAG if BOOTLINUX_TREE_NEXT
-	default BOOTLINUX_TREE_BTRFS_DEVEL_TAG if BOOTLINUX_TREE_BTRFS_DEVEL
-	default BOOTLINUX_KDEVOPS_TREE_TAG if BOOTLINUX_TREE_KDEVOPS_LINUS
-	default BOOTLINUX_MCGROF_TREE_TAG if BOOTLINUX_MCGROF_TREE
-	default BOOTLINUX_TREE_JLAYTON_LINUX_TAG if BOOTLINUX_TREE_JLAYTON_LINUX
-	default BOOTLINUX_CEL_TREE_TAG if BOOTLINUX_TREE_CEL_LINUX
+	default BOOTLINUX_TREE_LINUS_REF if BOOTLINUX_TREE_LINUS
+	default BOOTLINUX_TREE_STABLE_REF if BOOTLINUX_STABLE
+	default BOOTLINUX_TREE_NEXT_REF if BOOTLINUX_TREE_NEXT
+	default BOOTLINUX_TREE_BTRFS_DEVEL_REF if BOOTLINUX_TREE_BTRFS_DEVEL
+	default BOOTLINUX_KDEVOPS_TREE_REF if BOOTLINUX_TREE_KDEVOPS_LINUS
+	default BOOTLINUX_MCGROF_TREE_REF if BOOTLINUX_MCGROF_TREE
+	default BOOTLINUX_TREE_JLAYTON_LINUX_REF if BOOTLINUX_TREE_JLAYTON_LINUX
+	default BOOTLINUX_TREE_CEL_REF if BOOTLINUX_TREE_CEL_LINUX
 	default BOOTLINUX_CUSTOM_TAG if BOOTLINUX_CUSTOM
 
 config BOOTLINUX_TREE_KERNELRELEASE
workflows/linux/Kconfig.btrfs +1 -3
index be1e1fa..9fd4160 100644
@@ -1,8 +1,6 @@
 if BOOTLINUX_TREE_BTRFS_DEVEL
 
-config BOOTLINUX_TREE_BTRFS_DEVEL_TAG
-	string
-	default "btrfs-devel"
+source "workflows/linux/Kconfig.refs-btrfs-devel"
 
 config BOOTLINUX_TREE_BTRFS_DEVEL_NAME
 	string
workflows/linux/Kconfig.cel +3 -28
index 341ed69..782ae55 100644
@@ -1,25 +1,6 @@
 if BOOTLINUX_TREE_CEL_LINUX
 
-choice
-	prompt "cel-linux branch to use"
-	default BOOTLINUX_CEL_LINUX_NEXT
-
-config BOOTLINUX_CEL_LINUX_NEXT
-	bool "nfsd-next"
-	help
-	  Pull the nfsd-next branch.
-
-config BOOTLINUX_CEL_LINUX_FIXES
-	bool "nfsd-fixes"
-	help
-	  Pull the nfsd-fixes branch.
-
-config BOOTLINUX_CEL_LINUX_CUSTOM
-	bool "Custom branch name"
-	help
-	  Pull a custom branch name.
-
-endchoice
+source "workflows/linux/Kconfig.refs-cel-linux"
 
 config BOOTLINUX_CEL_LINUX_CUSTOM_NAME
 	string "Custom branch name"
@@ -28,17 +9,11 @@ config BOOTLINUX_CEL_LINUX_CUSTOM_NAME
 	help
 	  Custom branch name to use for Chuck Lever's linux tree.
 
-config BOOTLINUX_CEL_TREE_TAG
-	string
-	default "nfsd-next" if BOOTLINUX_CEL_LINUX_NEXT
-	default "nfsd-fixes" if BOOTLINUX_CEL_LINUX_FIXES
-	default BOOTLINUX_CEL_LINUX_CUSTOM_NAME if BOOTLINUX_CEL_LINUX_CUSTOM
-
-config BOOTLINUX_CEL_TREE_NAME
+config BOOTLINUX_TREE_CEL_NAME
 	string
 	default "cel-linux"
 
-config BOOTLINUX_CEL_TREE_URL
+config BOOTLINUX_TREE_CEL_URL
 	string
 	default "https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git" if BOOTLINUX_TREE_CEL_LINUX
 
workflows/linux/Kconfig.jlayton +3 -28
index d80d2d3..d058532 100644
@@ -1,25 +1,6 @@
 if BOOTLINUX_TREE_JLAYTON_LINUX
 
-choice
-	prompt "jlayton-linux branch to use"
-	default BOOTLINUX_JLAYTON_LINUX_KDEVOPS
-
-config BOOTLINUX_JLAYTON_LINUX_KDEVOPS
-	bool "kdevops"
-	help
-	  Use the kdevops branch.
-
-config BOOTLINUX_JLAYTON_LINUX_IVERSION
-	bool "iversion-next"
-	help
-	  Use the iversion-next branch.
-
-config BOOTLINUX_JLAYTON_LINUX_CUSTOM
-	bool "Custom branch name"
-	help
-	  Use a custom branch name.
-
-endchoice
+source "workflows/linux/Kconfig.refs-jlayton-linux"
 
 config BOOTLINUX_JLAYTON_LINUX_CUSTOM_NAME
 	string "Custom branch name"
@@ -28,17 +9,11 @@ config BOOTLINUX_JLAYTON_LINUX_CUSTOM_NAME
 	help
 	  Custom branch name to use for Jeff Layton's linux tree.
 
-config BOOTLINUX_JLAYTON_LINUX_TAG
-	string
-	default BOOTLINUX_JLAYTON_LINUX_CUSTOM_NAME if BOOTLINUX_JLAYTON_LINUX_CUSTOM
-	default "kdevops" if BOOTLINUX_JLAYTON_LINUX_KDEVOPS
-	default "iversion-next" if BOOTLINUX_JLAYTON_LINUX_IVERSION
-
-config BOOTLINUX_JLAYTON_TREE_NAME
+config BOOTLINUX_TREE_JLAYTON_NAME
 	string
 	default "jlayton-linux"
 
-config BOOTLINUX_JLAYTON_TREE_URL
+config BOOTLINUX_TREE_JLAYTON_URL
 	string
 	default "git://git.kernel.org/pub/scm/linux/kernel/git/jlayton/linux.git" if BOOTLINUX_TREE_JLAYTON_LINUX
 
workflows/linux/Kconfig.kdevops +2 -15
index 4cb8388..be2e020 100644
@@ -1,26 +1,13 @@
 if BOOTLINUX_TREE_KDEVOPS_LINUS
 
-choice
-	prompt "kdevops-linus branch to use"
-	default BOOTLINUX_KDEVOPS_LINUS_LBS_MINORDER
-
-config BOOTLINUX_KDEVOPS_LINUS_LBS_MINORDER
-	bool "large-block-minorder"
-	help
-	  Use the latest LBS branch based on Linus' tree.
-
-endchoice
+source "workflows/linux/Kconfig.refs-kdevops-linus"
 
 config BOOTLINUX_TREE_KDEVOPS_URL
 	string
 	default "https://github.com/linux-kdevops/linux.git" if !USE_LOCAL_LINUX_MIRROR
 	default "/mirror/kdevops-linus.git" if USE_LIBVIRT_MIRROR && BOOTLINUX_9P
 
-config BOOTLINUX_KDEVOPS_TREE_TAG
-	string
-	default "large-block-minorder"
-
-config BOOTLINUX_KDEVOPS_TREE_NAME
+config BOOTLINUX_TREE_KDEVOPS_NAME
 	string
 	default "kdevops-linus"
 
workflows/linux/Kconfig.linus +1 -68

75 lines not on this page · raw message

workflows/linux/Kconfig.mcgrof +5 -43

70 lines not on this page · raw message

workflows/linux/Kconfig.next +1 -3
index 6be0de4..e1c6c87 100644
@@ -1,8 +1,6 @@
 if BOOTLINUX_TREE_NEXT
 
-config BOOTLINUX_TREE_NEXT_TAG
-	string "linux-next tag to pull"
-	default "next-20240424"
+source "workflows/linux/Kconfig.refs-next"
 
 config BOOTLINUX_TREE_NEXT_NAME
 	string
workflows/linux/Kconfig.refs-btrfs-devel new file +15 -0
new file mode 100644
index 0000000..f97ab16
@@ -0,0 +1,15 @@
+# SPDX-License-Identifier: copyleft-next-0.3.1
+# Automatically generated file
+choice
+	prompt "Tag or branch to use"
+
+config BOOTLINUX_BTRFS_DEVEL
+	bool "btrfs-devel"
+	help
+		Use development branch.
+
+endchoice
+
+config BOOTLINUX_TREE_BTRFS_DEVEL_REF
+	string
+	default "btrfs-devel" if BOOTLINUX_BTRFS_DEVEL
workflows/linux/Kconfig.refs-cel-linux new file +27 -0
new file mode 100644
index 0000000..f990235
@@ -0,0 +1,27 @@
+# SPDX-License-Identifier: copyleft-next-0.3.1
+# Automatically generated file
+choice
+	prompt "Tag or branch to use"
+
+config BOOTLINUX_CEL_LINUX_NEXT
+	bool "nfsd-next"
+	help
+		Pull the nfsd-next branch.
+
+config BOOTLINUX_CEL_LINUX_FIXES
+	bool "nfsd-fixed"
+	help
+		Pull the nfsd-fixed branch.
+
+config BOOTLINUX_CEL_LINUX_CUSTOM
+	bool "custom"
+	help
+		Use a custom branch name.
+
+endchoice
+
+config BOOTLINUX_TREE_CEL_LINUX_REF
+	string
+	default "nfsd-next" if BOOTLINUX_CEL_LINUX_NEXT
+	default "nfsd-fixed" if BOOTLINUX_CEL_LINUX_FIXES
+	default BOOTLINUX_CEL_LINUX_CUSTOM_NAME if BOOTLINUX_CEL_LINUX_CUSTOM
workflows/linux/Kconfig.refs-jlayton-linux new file +27 -0
new file mode 100644
index 0000000..52b0fbb
@@ -0,0 +1,27 @@
+# SPDX-License-Identifier: copyleft-next-0.3.1
+# Automatically generated file
+choice
+	prompt "Tag or branch to use"
+
+config BOOTLINUX_JLAYTON_LINUX_KDEVOPS
+	bool "kdevops"
+	help
+		Use the latest kdevops branch.
+
+config BOOTLINUX_JLAYTON_LINUX_IVERSION
+	bool "iversion-next"
+	help
+		Use the iversion-next branch.
+
+config BOOTLINUX_JLAYTON_LINUX_CUSTOM
+	bool "custom"
+	help
+		Use a custom branch name.
+
+endchoice
+
+config BOOTLINUX_TREE_JLAYTON_LINUX_REF
+	string
+	default "kdevops" if BOOTLINUX_JLAYTON_LINUX_KDEVOPS
+	default "iversion-next" if BOOTLINUX_JLAYTON_LINUX_IVERSION
+	default BOOTLINUX_JLAYTON_LINUX_CUSTOM_NAME if BOOTLINUX_JLAYTON_LINUX_CUSTOM
workflows/linux/Kconfig.refs-kdevops-linus new file +15 -0
new file mode 100644
index 0000000..120842d
@@ -0,0 +1,15 @@
+# SPDX-License-Identifier: copyleft-next-0.3.1
+# Automatically generated file
+choice
+	prompt "Tag or branch to use"
+
+config BOOTLINUX_KDEVOPS_LINUS_LBS_MINORDER
+	bool "large-block-minorder"
+	help
+		Use the latest LBS branch based on Linus's tree
+
+endchoice
+
+config BOOTLINUX_TREE_KDEVOPS_LINUS_REF
+	string
+	default "large-block-minorder" if BOOTLINUX_KDEVOPS_LINUS_LBS_MINORDER
workflows/linux/Kconfig.refs-mcgrof-linus new file +21 -0
new file mode 100644
index 0000000..d2ca889
@@ -0,0 +1,21 @@
+# SPDX-License-Identifier: copyleft-next-0.3.1
+# Automatically generated file
+choice
+	prompt "Tag or branch to use"
+
+config BOOTLINUX_MCGROF_LINUS_LBS
+	bool "large-block-linus"
+	help
+		Use the latest LBS branch based on Linus' tree.
+
+config BOOTLINUX_MCGROF_LINUS_LBS_NOBDEV
+	bool "large-block-linus-nobdev"
+	help
+		Use the latest minimal branch for LBS which let's you use LBS devices with a sectorsize compatibility based on Linus' tree.
+
+endchoice
+
+config BOOTLINUX_TREE_MCGROF_LINUS_REF
+	string
+	default "large-block-linus" if BOOTLINUX_MCGROF_LINUS_LBS
+	default "large-block-linus-nobdev" if BOOTLINUX_MCGROF_LINUS_LBS_NOBDEV
workflows/linux/Kconfig.refs-mcgrof-next new file +21 -0
new file mode 100644
index 0000000..36d93c2
@@ -0,0 +1,21 @@
+# SPDX-License-Identifier: copyleft-next-0.3.1
+# Automatically generated file
+choice
+	prompt "Tag or branch to use"
+
+config BOOTLINUX_MCGROF_NEXT_LBS
+	bool "large-block-next"
+	help
+		Use the latest stable LBS branch which enables all the bells and whistles for all types of LBS devices.
+
+config BOOTLINUX_MCGROF_NEXT_LBS_NOBDEV
+	bool "large-block-nobdev"
+	help
+		Use the latest minimal branch for LBS which let's you use LBS devices with a sectorsize compatibility
+
+endchoice
+
+config BOOTLINUX_TREE_MCGROF_NEXT_REF
+	string
+	default "large-block-next" if BOOTLINUX_MCGROF_NEXT_LBS
+	default "large-block-nobdev" if BOOTLINUX_MCGROF_NEXT_LBS_NOBDEV
workflows/linux/Kconfig.stable +1 -62

72 lines not on this page · raw message

workflows/linux/btrfs-devel-refs.yaml new file +6 -0
new file mode 100644
index 0000000..226a166
@@ -0,0 +1,6 @@
+---
+configs:
+  - btrfs:
+    config: BOOTLINUX_BTRFS_DEVEL
+    ref: "btrfs-devel"
+    help: "Use development branch."
workflows/linux/cel-linux-refs.yaml new file +14 -0
new file mode 100644
index 0000000..a2676c8
@@ -0,0 +1,14 @@
+---
+configs:
+  - next:
+    config: BOOTLINUX_CEL_LINUX_NEXT
+    ref: "nfsd-next"
+    help: "Pull the nfsd-next branch."
+  - fixes:
+    config: BOOTLINUX_CEL_LINUX_FIXES
+    ref: "nfsd-fixed"
+    help: "Pull the nfsd-fixed branch."
+  - custom:
+    config: BOOTLINUX_CEL_LINUX_CUSTOM
+    ref: BOOTLINUX_CEL_LINUX_CUSTOM_NAME
+    help: "Use a custom branch name."
workflows/linux/jlayton-linux-refs.yaml new file +14 -0
new file mode 100644
index 0000000..a490b2b
@@ -0,0 +1,14 @@
+---
+configs:
+  - kdevops:
+    config: BOOTLINUX_JLAYTON_LINUX_KDEVOPS
+    ref: "kdevops"
+    help: "Use the latest kdevops branch."
+  - iversion:
+    config: BOOTLINUX_JLAYTON_LINUX_IVERSION
+    ref: "iversion-next"
+    help: "Use the iversion-next branch."
+  - custom:
+    config: BOOTLINUX_JLAYTON_LINUX_CUSTOM
+    ref: BOOTLINUX_JLAYTON_LINUX_CUSTOM_NAME
+    help: "Use a custom branch name."
workflows/linux/kdevops-linus-refs.yaml new file +6 -0
new file mode 100644
index 0000000..b37e075
@@ -0,0 +1,6 @@
+---
+configs:
+  - minorder:
+    config: BOOTLINUX_KDEVOPS_LINUS_LBS_MINORDER
+    ref: "large-block-minorder"
+    help: "Use the latest LBS branch based on Linus's tree"
workflows/linux/mcgrof-linus-refs.yaml new file +10 -0
new file mode 100644
index 0000000..d343b17
@@ -0,0 +1,10 @@
+---
+configs:
+  - master:
+    config: BOOTLINUX_MCGROF_LINUS_LBS
+    ref: "large-block-linus"
+    help: "Use the latest LBS branch based on Linus' tree."
+  - main:
+    config: BOOTLINUX_MCGROF_LINUS_LBS_NOBDEV
+    ref: "large-block-linus-nobdev"
+    help: "Use the latest minimal branch for LBS which let's you use LBS devices with a sectorsize compatibility based on Linus' tree."
workflows/linux/mcgrof-next-refs.yaml new file +10 -0
new file mode 100644
index 0000000..d430a0a
@@ -0,0 +1,10 @@
+---
+configs:
+  - master:
+    config: BOOTLINUX_MCGROF_NEXT_LBS
+    ref: "large-block-next"
+    help: "Use the latest stable LBS branch which enables all the bells and whistles for all types of LBS devices."
+  - main:
+    config: BOOTLINUX_MCGROF_NEXT_LBS_NOBDEV
+    ref: "large-block-nobdev"
+    help: "Use the latest minimal branch for LBS which let's you use LBS devices with a sectorsize compatibility"
Signature · 1 line
2.43.0
#3 lore raw

Re: [PATCH v3 1/1] linux: generate refs automatically

Quoted 11 lines · Daniel Gomez

Generate a Kconfig choice/endchoice menu with a list of the latest git references for upstream repositories.

* Default behaviour: latest-and-greatest.
The script included (generate_refs.py) will create (if it does not exist) the Kconfig choice options with the latest and greatest kernel release references provided by kernel.org/releases.json. Every time make is run, the script will check for the date of the last modification, and will force an update if more than 24h has passed since last modification. The generated files are: Kconfig.refs-{next,stable,linus}.

Quoted 6 lines

* make dynconfig:
In addition, the script will generate and overwrite all the existing Kconfig.refs-* files (including next, stable and linus) when dynconfig is run. This mode will retrieve the information from the upstream git repoitories for all supported kernel trees in kdevops and will create a list with the latest 15 references found.

This is pretty fucking cool.

Quoted 5 lines
---
 .gitignore                                 |   2 +
 Makefile                                   |   3 +
 defconfigs/nfsd                            |   2 +-

Oh wow, that's the only one that needed changing?

scripts/gen-gitrefs.Makefile line 7

Quoted 13 lines
scripts/gen-gitrefs.Makefile new file +7 -0
new file mode 100644
index 0000000..6cea238
@@ -0,0 +1,111 @@
+# SPDX-License-Identifier: copyleft-next-0.3.1
+
+$(info Generating git references...)
+
+export SRC_URI_HTTPS_LINUS = https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
+export SRC_URI_HTTPS_NEXT = https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
+export SRC_URI_HTTPS_STABLE = https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git

If Linux had a religion, the above would be our canon for development.

scripts/gen-gitrefs.Makefile line 14

Quoted 6 lines
scripts/gen-gitrefs.Makefile new file +6 -0
+export SRC_URI_HTTPS_MCGROF_LINUS = https://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/linux.git
+export SRC_URI_HTTPS_MCGROF_NEXT = https://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/linux-next.git
+export SRC_URI_HTTPS_BTRFS_DEVEL = https://github.com/kdave/btrfs-devel.git
+export SRC_URI_HTTPS_CEL_LINUX = https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git
+export SRC_URI_HTTPS_JLAYTON_LINUX = https://git.kernel.org/pub/scm/linux/kernel/git/jlayton/linux.git
+export SRC_URI_HTTPS_KDEVOPS_LINUS = https://github.com/linux-kdevops/linux.git

While this is the apocrypha. And in thinking about scaling this, I wonder if we should just have this section later, the apocrypha, automatically generated. Doesn't have to happen now, I think this a *huge* improvement over what we had before, so I welcome this as-is now with just a few comments below:

scripts/gen-gitrefs.Makefile line 23

Quoted 8 lines
scripts/gen-gitrefs.Makefile new file +8 -0
+GENERATED_REFS_LINUS := $(shell \
+	./scripts/generate_refs.py \
+		--prefix BOOTLINUX_TREE_LINUS \
+		--output workflows/linux/Kconfig.refs-linus \
+		--force \
+		gitref \
+		--repo $(SRC_URI_HTTPS_LINUS) \
+		--refs 15 \

These seem to only differ from the non dynconfig approach with a few differences, and I wonder if those differences could be set with Makefile variables, and then having the dynconfig include a shared file. Doing a cursory review of the arguments passed, I don't think it's a as easy as a few variable changes, so I'll leave it up to you to decide. I'd prefer to get this merged sooner rather than later and we deal with other enhancements later.

As for the 15 latest refs per tree, hell I'd say just last 5 is good but after checking the results, and thining that one is already querying git so asking for 5 Vs 15 doesn't cause less latencies on make dynconfig, so I think 15 seems pretty sensible to me right now. If anything a user environment variable could be set to allow this to go beyond 15 too.

What concerns me slightly is just the default canon only gets the latest and greatest, while I think it is right that our Kconfig should default to the the latest and greatest, it would not allow us to allow scripts with older .configs to exist which rely on older rc releases for example.

If using default refs of 15 even for canon fixes this up to the latest 15 releases, then I'd be in favor of us just adopting that. I'd like other's feedback on this.

Luis

#4 lore raw

Re: [PATCH v3 1/1] linux: generate refs automatically

Quoted 21 lines · Luis Chamberlain
Quoted 11 lines · Daniel Gomez

Generate a Kconfig choice/endchoice menu with a list of the latest git references for upstream repositories.

* Default behaviour: latest-and-greatest.
The script included (generate_refs.py) will create (if it does not exist) the Kconfig choice options with the latest and greatest kernel release references provided by kernel.org/releases.json. Every time make is run, the script will check for the date of the last modification, and will force an update if more than 24h has passed since last modification. The generated files are: Kconfig.refs-{next,stable,linus}.

Quoted 6 lines

* make dynconfig:
In addition, the script will generate and overwrite all the existing Kconfig.refs-* files (including next, stable and linus) when dynconfig is run. This mode will retrieve the information from the upstream git repoitories for all supported kernel trees in kdevops and will create a list with the latest 15 references found.

This is pretty fucking cool.

Quoted 7 lines
Quoted 5 lines
---
 .gitignore                                 |   2 +
 Makefile                                   |   3 +
 defconfigs/nfsd                            |   2 +-

Oh wow, that's the only one that needed changing?

You are right. I did some Kconfig renames and I'm missing few places that need update for example, to replace BOOTLINUX_TREE_TAG with he new BOOTLINUX_TREE_REF introduced in this series. I'll double check it. Thanks!

Quoted 30 lines
Quoted 7 lines
scripts/gen-gitrefs.Makefile new file +0 -0
new file mode 100644
index 0000000..6cea238

scripts/gen-gitrefs.Makefile line 7

Quoted 7 lines
scripts/gen-gitrefs.Makefile new file +7 -0
@@ -0,0 +1,111 @@
+# SPDX-License-Identifier: copyleft-next-0.3.1
+
+$(info Generating git references...)
+
+export SRC_URI_HTTPS_LINUS = https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
+export SRC_URI_HTTPS_NEXT = https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
+export SRC_URI_HTTPS_STABLE = https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git

If Linux had a religion, the above would be our canon for development.

scripts/gen-gitrefs.Makefile

Quoted 6 lines
scripts/gen-gitrefs.Makefile new file +6 -0
+export SRC_URI_HTTPS_MCGROF_LINUS = https://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/linux.git
+export SRC_URI_HTTPS_MCGROF_NEXT = https://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/linux-next.git
+export SRC_URI_HTTPS_BTRFS_DEVEL = https://protect2.fireeye.com/v1/url?k=26391c8b-47b209b2-263897c4-000babffae10-69325ae4f77a3118&q=1&e=e0306a31-3fd8-4b61-a6db-8aad13329128&u=https%3A%2F%2Fgithub.com%2Fkdave%2Fbtrfs-devel.git
+export SRC_URI_HTTPS_CEL_LINUX = https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git
+export SRC_URI_HTTPS_JLAYTON_LINUX = https://git.kernel.org/pub/scm/linux/kernel/git/jlayton/linux.git
+export SRC_URI_HTTPS_KDEVOPS_LINUS = https://protect2.fireeye.com/v1/url?k=1fabbbfc-7e20aec5-1faa30b3-000babffae10-7fccea486acfbba5&q=1&e=e0306a31-3fd8-4b61-a6db-8aad13329128&u=https%3A%2F%2Fgithub.com%2Flinux-kdevops%2Flinux.git

While this is the apocrypha. And in thinking about scaling this, I wonder if we should just have this section later, the apocrypha, automatically generated. Doesn't have to happen now, I think this a *huge* improvement over what we had before, so I welcome this as-is now with just a few comments below:

Quoted 14 lines

scripts/gen-gitrefs.Makefile line 23

Quoted 8 lines
scripts/gen-gitrefs.Makefile new file +8 -0
+GENERATED_REFS_LINUS := $(shell \
+	./scripts/generate_refs.py \
+		--prefix BOOTLINUX_TREE_LINUS \
+		--output workflows/linux/Kconfig.refs-linus \
+		--force \
+		gitref \
+		--repo $(SRC_URI_HTTPS_LINUS) \
+		--refs 15 \

These seem to only differ from the non dynconfig approach with a few differences, and I wonder if those differences could be set with Makefile variables, and then having the dynconfig include a shared file. Doing a cursory review of the arguments passed, I don't think it's a as easy as a few variable changes, so I'll leave it up to you to decide.

The problem is to include the git refs differences in the already existing Kconfig file as you need to add not only the new choices but also to increase the _REF list. Perhaps the easiest is to regenerate Kconfig files with the kernel.org/releases.json refs found first.

Quoted 2 lines

I'd prefer to get this merged sooner rather than later and we deal with other enhancements later.

Ok.

Quoted 6 lines

As for the 15 latest refs per tree, hell I'd say just last 5 is good but after checking the results, and thining that one is already querying git so asking for 5 Vs 15 doesn't cause less latencies on make dynconfig, so I think 15 seems pretty sensible to me right now. If anything a user environment variable could be set to allow this to go beyond 15 too.

I think user config is better option and 15 by default looks sensible considering we have typically 7/8 -rc versions so I was thinking to be able to go at least 1 release back (for mainline).

Quoted 6 lines

What concerns me slightly is just the default canon only gets the latest and greatest, while I think it is right that our Kconfig should default to the the latest and greatest, it would not allow us to allow scripts with older .configs to exist which rely on older rc releases for example.

Old .configs pushed to kdevops can also include the static references needed in the yaml file. Then, we can include them with kreleases command (with --extra).

Quoted 4 lines

If using default refs of 15 even for canon fixes this up to the latest 15 releases, then I'd be in favor of us just adopting that. I'd like other's feedback on this.

That would require quering kernel.org which I think we were trying to avoid in the canon path? Local git mirror can be used too but would require the user to have set them up.

I cannot find better options other than quering via http/git kernel.org or manually including them in the static yaml file. Any alternatives?

Maybe we can just go back from the latest version? So from 6.10-rc2 go back and generate all v6.x mainline stable releases. And we would have just to include the latest for each major version (e.g. v5.19, v4.20, v3.19...) and generate all the refs from there. Since we do not have a pattern with -rcX we would not be able to automatically generate them with this approach.

Quoted 2 lines

Luis