terraform/OCI: Grab secrets from ~/.oci/config
RFC PATCH 1/1 terraform/OCI: Grab secrets from ~/.oci/config
From: Chuck Lever <[email protected]>
Instead of storing authentication secrets in the kdevops .config file, pull them from the authentication profiles already set up in ~/.oci/config. This arrangement is more secure.
terraform's API authentication is now managed outside of Kconfig, as is done with AWS. An update to docs/kdevops-terraform.md to follow.
--- .../templates/oci/terraform.tfvars.j2 | 5 +--- scripts/terraform.Makefile | 4 --- terraform/oci/kconfigs/Kconfig.identity | 27 +++++++------------ terraform/oci/provider.tf | 7 ++--- terraform/oci/vars.tf | 25 ++++------------- 5 files changed, 17 insertions(+), 51 deletions(-)
The tenancy OCID, user OCID, fingerprint, and private key path Kconfig settings would no longer be needed. This patch fits somewhere in the middle of the 00/31 series, replacing several of those patches.
playbooks/roles/gen_tfvars/templates/oci/terraform.tfvars.j2 +1 -4
@@ -1,8 +1,5 @@ oci_region = "{{ terraform_oci_region }}" -oci_tenancy_ocid = "{{ terraform_oci_tenancy_ocid }}" -oci_user_ocid = "{{ terraform_oci_user_ocid }}" -oci_user_private_key_path = "{{ terraform_oci_user_private_key_path }}" -oci_user_fingerprint = "{{ terraform_oci_user_fingerprint }}" +oci_config_file_profile = "{{ terraform_oci_config_file_profile }}" oci_availablity_domain = "{{ terraform_oci_availablity_domain }}" oci_compartment_ocid = "{{ terraform_oci_compartment_ocid }}" oci_shape = "{{ terraform_oci_shape }}"
scripts/terraform.Makefile +0 -4
@@ -96,10 +96,6 @@ endif ifeq (y,$(CONFIG_TERRAFORM_OCI)) TERRAFORM_EXTRA_VARS += terraform_oci_region=$(subst ",,$(CONFIG_TERRAFORM_OCI_REGION)) -TERRAFORM_EXTRA_VARS += terraform_oci_tenancy_ocid=$(subst ",,$(CONFIG_TERRAFORM_OCI_TENANCY_OCID)) -TERRAFORM_EXTRA_VARS += terraform_oci_user_ocid=$(subst ",,$(CONFIG_TERRAFORM_OCI_USER_OCID)) -TERRAFORM_EXTRA_VARS += terraform_oci_user_private_key_path=$(subst ",,$(CONFIG_TERRAFORM_OCI_USER_PRIVATE_KEY_PATH)) -TERRAFORM_EXTRA_VARS += terraform_oci_user_fingerprint=$(subst ",,$(CONFIG_TERRAFORM_OCI_USER_FINGERPRINT)) TERRAFORM_EXTRA_VARS += terraform_oci_availablity_domain=$(subst ",,$(CONFIG_TERRAFORM_OCI_AVAILABLITY_DOMAIN)) TERRAFORM_EXTRA_VARS += terraform_oci_compartment_ocid=$(subst ",,$(CONFIG_TERRAFORM_OCI_COMPARTMENT_OCID)) TERRAFORM_EXTRA_VARS += terraform_oci_shape=$(subst ",,$(CONFIG_TERRAFORM_OCI_SHAPE))
terraform/oci/kconfigs/Kconfig.identity +9 -18
@@ -5,23 +5,14 @@ config TERRAFORM_OCI_TENANCY_OCID Read this: https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/terraformproviderconfiguration.htm -config TERRAFORM_OCI_USER_OCID - string "OCI user OCID" +config TERRAFORM_OCI_CONFIG_FILE_PROFILE + string "Authentication profile entry" + output yaml + default "DEFAULT" help - The OCI ID of the user to use. - Read this: - https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/terraformproviderconfiguration.htm + Authentication secrets are stored in ~/.oci/config. That + file may contain more than one entry/identity. Select the + entry in ~/.oci/config that kdevops uses to + authenticate to OCI. Examples: -config TERRAFORM_OCI_USER_PRIVATE_KEY_PATH - string "OCI user private key path" - help - Path to the file containing the private key of the user. - Read this: - https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/terraformproviderconfiguration.htm - -config TERRAFORM_OCI_USER_FINGERPRINT - string "OCI user fingerprint" - help - Fingerprint for the key pair being used. - Read this: - https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/terraformproviderconfiguration.htm + https://docs.oracle.com/en-us/iaas/Content/API/Concepts/sdkconfig.htm
terraform/oci/provider.tf +2 -5
@@ -2,9 +2,6 @@ terraform { } provider "oci" { - tenancy_ocid = var.oci_tenancy_ocid - user_ocid = var.oci_user_ocid - private_key_path = var.oci_user_private_key_path - fingerprint = var.oci_user_fingerprint - region = var.oci_region + config_file_profile = var.oci_config_file_profile + region = var.oci_region }
terraform/oci/vars.tf +5 -20
@@ -1,28 +1,13 @@ +variable "oci_config_file_profile" { + description = "Entry in ~/.oci/config to use for API authentication" + type = string +} + variable "oci_region" { description = "An OCI region" type = string } -variable "oci_tenancy_ocid" { - description = "OCID of your tenancy" - type = string -} - -variable "oci_user_ocid" { - description = "OCID of the user calling the API" - type = string -} - -variable "oci_user_private_key_path" { - description = "The path of the private key stored on your computer" - type = string -} - -variable "oci_user_fingerprint" { - description = "Fingerprint for the key pair being used" - type = string -} - variable "oci_availablity_domain" { description = "Name of availability domain" type = string
Signature · 1 line
2.48.1
Quoted 20 lines · On 4/3/25 10:49 AM, [email protected] wrote:
From: Chuck Lever <[email protected]>
Instead of storing authentication secrets in the kdevops .config file, pull them from the authentication profiles already set up in ~/.oci/config. This arrangement is more secure.
terraform's API authentication is now managed outside of
Kconfig, as is done with AWS. An update todocs/kdevops-terraform.mdto follow.Suggested-by Luis Chamberlain <[email protected]>Signed-off-by Chuck Lever <[email protected]>--- .../templates/oci/terraform.tfvars.j2 | 5 +--- scripts/terraform.Makefile | 4 --- terraform/oci/kconfigs/Kconfig.identity | 27 +++++++------------ terraform/oci/provider.tf | 7 ++--- terraform/oci/vars.tf | 25 ++++------------- 5 files changed, 17 insertions(+), 51 deletions(-)
Quoted 4 lines
The tenancy OCID, user OCID, fingerprint, and private key path
Kconfigsettings would no longer be needed. This patch fits somewhere in the middle of the 00/31 series, replacing several of those patches.
It appears that, though undocumented, terraform's azurerm provider can also pull its authentication material from a home directory dot file (~/.azure/azureProfile.json). Proof-of-concept tested and working.
Google already works this way. So we can use the "secrets are stored outside of kdevops' .config file" for all four major cloud providers.
Quoted 120 lines
playbooks/roles/gen_tfvars/templates/oci/terraform.tfvars.j2 +1
-4@@ -1,8 +1,5 @@ oci_region = "{{ terraform_oci_region }}" -oci_tenancy_ocid = "{{ terraform_oci_tenancy_ocid }}" -oci_user_ocid = "{{ terraform_oci_user_ocid }}" -oci_user_private_key_path = "{{ terraform_oci_user_private_key_path }}" -oci_user_fingerprint = "{{ terraform_oci_user_fingerprint }}" +oci_config_file_profile = "{{ terraform_oci_config_file_profile }}" oci_availablity_domain = "{{ terraform_oci_availablity_domain }}" oci_compartment_ocid = "{{ terraform_oci_compartment_ocid }}" oci_shape = "{{ terraform_oci_shape }}"scripts/terraform.Makefile +0
-4@@ -96,10 +96,6 @@ endif ifeq (y,$(CONFIG_TERRAFORM_OCI)) TERRAFORM_EXTRA_VARS += terraform_oci_region=$(subst ",,$(CONFIG_TERRAFORM_OCI_REGION)) -TERRAFORM_EXTRA_VARS += terraform_oci_tenancy_ocid=$(subst ",,$(CONFIG_TERRAFORM_OCI_TENANCY_OCID)) -TERRAFORM_EXTRA_VARS += terraform_oci_user_ocid=$(subst ",,$(CONFIG_TERRAFORM_OCI_USER_OCID)) -TERRAFORM_EXTRA_VARS += terraform_oci_user_private_key_path=$(subst ",,$(CONFIG_TERRAFORM_OCI_USER_PRIVATE_KEY_PATH)) -TERRAFORM_EXTRA_VARS += terraform_oci_user_fingerprint=$(subst ",,$(CONFIG_TERRAFORM_OCI_USER_FINGERPRINT)) TERRAFORM_EXTRA_VARS += terraform_oci_availablity_domain=$(subst ",,$(CONFIG_TERRAFORM_OCI_AVAILABLITY_DOMAIN)) TERRAFORM_EXTRA_VARS += terraform_oci_compartment_ocid=$(subst ",,$(CONFIG_TERRAFORM_OCI_COMPARTMENT_OCID)) TERRAFORM_EXTRA_VARS += terraform_oci_shape=$(subst ",,$(CONFIG_TERRAFORM_OCI_SHAPE))terraform/oci/kconfigs/Kconfig.identity +9
-18@@ -5,23 +5,14 @@ config TERRAFORM_OCI_TENANCY_OCID Read this: https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/terraformproviderconfiguration.htm -config TERRAFORM_OCI_USER_OCID - string "OCI user OCID" +config TERRAFORM_OCI_CONFIG_FILE_PROFILE + string "Authentication profile entry" + output yaml + default "DEFAULT" help - The OCI ID of the user to use. - Read this: - https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/terraformproviderconfiguration.htm + Authentication secrets are stored in ~/.oci/config. That + file may contain more than one entry/identity. Select the + entry in ~/.oci/config that kdevops uses to + authenticate to OCI. Examples: -config TERRAFORM_OCI_USER_PRIVATE_KEY_PATH - string "OCI user private key path" - help - Path to the file containing the private key of the user. - Read this: - https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/terraformproviderconfiguration.htm - -config TERRAFORM_OCI_USER_FINGERPRINT - string "OCI user fingerprint" - help - Fingerprint for the key pair being used. - Read this: - https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/terraformproviderconfiguration.htm + https://docs.oracle.com/en-us/iaas/Content/API/Concepts/sdkconfig.htmterraform/oci/provider.tf +2
-5@@ -2,9 +2,6 @@ terraform { } provider "oci" { - tenancy_ocid = var.oci_tenancy_ocid - user_ocid = var.oci_user_ocid - private_key_path = var.oci_user_private_key_path - fingerprint = var.oci_user_fingerprint - region = var.oci_region + config_file_profile = var.oci_config_file_profile + region = var.oci_region }terraform/oci/vars.tf +5
-20@@ -1,28 +1,13 @@ +variable "oci_config_file_profile" { + description = "Entry in ~/.oci/config to use for API authentication" + type = string +} + variable "oci_region" { description = "An OCI region" type = string } -variable "oci_tenancy_ocid" { - description = "OCID of your tenancy" - type = string -} - -variable "oci_user_ocid" { - description = "OCID of the user calling the API" - type = string -} - -variable "oci_user_private_key_path" { - description = "The path of the private key stored on your computer" - type = string -} - -variable "oci_user_fingerprint" { - description = "Fingerprint for the key pair being used" - type = string -} - variable "oci_availablity_domain" { description = "Name of availability domain" type = string
Signature · 1 line
Chuck Lever
Quoted 21 lines · Chuck Lever
Quoted 20 lines · On 4/3/25 10:49 AM, [email protected] wrote:
From: Chuck Lever <[email protected]>
Instead of storing authentication secrets in the kdevops .config file, pull them from the authentication profiles already set up in ~/.oci/config. This arrangement is more secure.
terraform's API authentication is now managed outside of
Kconfig, as is done with AWS. An update todocs/kdevops-terraform.mdto follow.Suggested-by Luis Chamberlain <[email protected]>Signed-off-by Chuck Lever <[email protected]>--- .../templates/oci/terraform.tfvars.j2 | 5 +--- scripts/terraform.Makefile | 4 --- terraform/oci/kconfigs/Kconfig.identity | 27 +++++++------------ terraform/oci/provider.tf | 7 ++--- terraform/oci/vars.tf | 25 ++++------------- 5 files changed, 17 insertions(+), 51 deletions(-)
Quoted 8 lines
Quoted 4 lines
The tenancy OCID, user OCID, fingerprint, and private key path
Kconfigsettings would no longer be needed. This patch fits somewhere in the middle of the 00/31 series, replacing several of those patches.It appears that, though undocumented, terraform's azurerm provider can also pull its authentication material from a home directory dot file (
~/.azure/azureProfile.json). Proof-of-concept tested and working.
Nice!
Quoted 2 lines
Google already works this way. So we can use the "secrets are stored outside of kdevops' .config file" for all four major cloud providers.
Do we want a Kconfig SUPPORTS_SEPARATE_CLOUD_SECRETS or do we want to just implicate this is a requirement?
Luis
Quoted 31 lines · Luis Chamberlain
Quoted 21 lines · Chuck Lever
Quoted 20 lines · On 4/3/25 10:49 AM, [email protected] wrote:
From: Chuck Lever <[email protected]>
Instead of storing authentication secrets in the kdevops .config file, pull them from the authentication profiles already set up in ~/.oci/config. This arrangement is more secure.
terraform's API authentication is now managed outside of
Kconfig, as is done with AWS. An update todocs/kdevops-terraform.mdto follow.Suggested-by Luis Chamberlain <[email protected]>Signed-off-by Chuck Lever <[email protected]>--- .../templates/oci/terraform.tfvars.j2 | 5 +--- scripts/terraform.Makefile | 4 --- terraform/oci/kconfigs/Kconfig.identity | 27 +++++++------------ terraform/oci/provider.tf | 7 ++--- terraform/oci/vars.tf | 25 ++++------------- 5 files changed, 17 insertions(+), 51 deletions(-)Quoted 8 lines
Quoted 4 lines
The tenancy OCID, user OCID, fingerprint, and private key path
Kconfigsettings would no longer be needed. This patch fits somewhere in the middle of the 00/31 series, replacing several of those patches.It appears that, though undocumented, terraform's azurerm provider can also pull its authentication material from a home directory dot file (
~/.azure/azureProfile.json). Proof-of-concept tested and working.
Quoted 7 lines
Nice!
Quoted 2 lines
Google already works this way. So we can use the "secrets are stored outside of kdevops' .config file" for all four major cloud providers.
Do we want a
KconfigSUPPORTS_SEPARATE_CLOUD_SECRETSor do we want to just implicate this is a requirement?
IMHO "separate secrets" is just more secure overall, and I generally lean towards not enabling users to select a lower level of security unless we absolutely have to.
- kdevops in Google and AWS already support only separate secrets
- kdevops in Azure was not working (for other reasons) so there's no backwards compatibility issue if we change it
Only possible issue is with OCI, and this series makes enough changes that anyone who upgrades kdevops will need to rework their .config anyway.
The series has a patch at the end (maybe I didn't post that) that updates docs/kdevops-terraform.md.
On a related topic, if we ever want to fully support running the kdevops /control host/ in the cloud, terraform supports an authentication mechanism that just uses the local instance's service principal, so no extra authentication material is needed for provisioning the test runners as separate cloud instances. Interesting to consider.
Signature · 1 line
Chuck Lever
Quoted 37 lines · Chuck Lever
Quoted 36 lines · Luis Chamberlain
Quoted 21 lines · Chuck Lever
Quoted 20 lines · On 4/3/25 10:49 AM, [email protected] wrote:
From: Chuck Lever <[email protected]>
Instead of storing authentication secrets in the kdevops .config file, pull them from the authentication profiles already set up in ~/.oci/config. This arrangement is more secure.
terraform's API authentication is now managed outside of
Kconfig, as is done with AWS. An update todocs/kdevops-terraform.mdto follow.Suggested-by Luis Chamberlain <[email protected]>Signed-off-by Chuck Lever <[email protected]>--- .../templates/oci/terraform.tfvars.j2 | 5 +--- scripts/terraform.Makefile | 4 --- terraform/oci/kconfigs/Kconfig.identity | 27 +++++++------------ terraform/oci/provider.tf | 7 ++--- terraform/oci/vars.tf | 25 ++++------------- 5 files changed, 17 insertions(+), 51 deletions(-)Quoted 8 lines
Quoted 4 lines
The tenancy OCID, user OCID, fingerprint, and private key path
Kconfigsettings would no longer be needed. This patch fits somewhere in the middle of the 00/31 series, replacing several of those patches.It appears that, though undocumented, terraform's azurerm provider can also pull its authentication material from a home directory dot file (
~/.azure/azureProfile.json). Proof-of-concept tested and working.Nice!
Quoted 2 lines
Google already works this way. So we can use the "secrets are stored outside of kdevops' .config file" for all four major cloud providers.
Quoted 6 lines
Quoted 2 lines
Do we want a
KconfigSUPPORTS_SEPARATE_CLOUD_SECRETSor do we want to just implicate this is a requirement?IMHO "separate secrets" is just more secure overall, and I generally lean towards not enabling users to select a lower level of security unless we absolutely have to.
I'm all in favor of simply making a rule to never allow secrets on .config. That lets us scale for CIs and enable reprodducible ci-workloads safely and collection of results on kdevops-results-archive.
Quoted 3 lines
- kdevops in Google and AWS already support only separate secrets
- kdevops in Azure was not working (for other reasons) so there's no backwards compatibility issue if we change it
Perfect.
Quoted 3 lines
Only possible issue is with OCI, and this series makes enough changes that anyone who upgrades kdevops will need to rework their .config anyway.
Makes sense.
Quoted 9 lines
The series has a patch at the end (maybe I didn't post that) that updates
docs/kdevops-terraform.md.On a related topic, if we ever want to fully support running the kdevops /control host/ in the cloud, terraform supports an authentication mechanism that just uses the local instance's service principal, so no extra authentication material is needed for provisioning the test runners as separate cloud instances. Interesting to consider.
Sorry I failed to understand this, what is mean by separate cloud instances?
Luis
Quoted 9 lines · Luis Chamberlain
Quoted 5 lines · Chuck Lever
On a related topic, if we ever want to fully support running the kdevops /control host/ in the cloud, terraform supports an authentication mechanism that just uses the local instance's service principal, so no extra authentication material is needed for provisioning the test runners as separate cloud instances. Interesting to consider.
Sorry I failed to understand this, what is mean by separate cloud instances?
The usual situation is the control host (where terraform runs) is outside the cloud. Like, on your workstation. I'm talking about a scenario where kdevops and terraform are running on an instance in the same cloud as where your target nodes are going to run.
In that case, terraform can scoop up the service principal for that instance, and use it in place of dot file authentication parameters. So you don't have to maintain the dot file on the instance that is running terraform, if it is already in the cloud.
Signature · 1 line
Chuck Lever
Quoted 16 lines · Chuck Lever
Quoted 9 lines · Luis Chamberlain
Quoted 5 lines · Chuck Lever
On a related topic, if we ever want to fully support running the kdevops /control host/ in the cloud, terraform supports an authentication mechanism that just uses the local instance's service principal, so no extra authentication material is needed for provisioning the test runners as separate cloud instances. Interesting to consider.
Sorry I failed to understand this, what is mean by separate cloud instances?
The usual situation is the control host (where terraform runs) is outside the cloud. Like, on your workstation. I'm talking about a scenario where kdevops and terraform are running on an instance in the same cloud as where your target nodes are going to run.
Quoted 4 lines
In that case, terraform can scoop up the service principal for that instance, and use it in place of dot file authentication parameters. So you don't have to maintain the dot file on the instance that is running terraform, if it is already in the cloud.
I see, so you first bring that cloud instance, and then use that as your command and control for test nodes. Does that first cloud instance need to be instantiated through another kdevops setup?
Luis
Quoted 17 lines · Luis Chamberlain
Quoted 16 lines · Chuck Lever
Quoted 9 lines · Luis Chamberlain
Quoted 5 lines · Chuck Lever
On a related topic, if we ever want to fully support running the kdevops /control host/ in the cloud, terraform supports an authentication mechanism that just uses the local instance's service principal, so no extra authentication material is needed for provisioning the test runners as separate cloud instances. Interesting to consider.
Sorry I failed to understand this, what is mean by separate cloud instances?
The usual situation is the control host (where terraform runs) is outside the cloud. Like, on your workstation. I'm talking about a scenario where kdevops and terraform are running on an instance in the same cloud as where your target nodes are going to run.
Quoted 8 lines
Quoted 4 lines
In that case, terraform can scoop up the service principal for that instance, and use it in place of dot file authentication parameters. So you don't have to maintain the dot file on the instance that is running terraform, if it is already in the cloud.
I see, so you first bring that cloud instance, and then use that as your command and control for test nodes. Does that first cloud instance need to be instantiated through another kdevops setup?
No, you can create it with terraform or from the cloud console. Start it up when you want, or leave it running.
I'm wondering if we can get a cloud devops pipeline to trigger it when it sees a PR against a watched repo. Haven't really thought that through.
Signature · 1 line
Chuck Lever
Quoted 27 lines · Chuck Lever
Quoted 17 lines · Luis Chamberlain
Quoted 16 lines · Chuck Lever
Quoted 9 lines · Luis Chamberlain
Quoted 5 lines · Chuck Lever
On a related topic, if we ever want to fully support running the kdevops /control host/ in the cloud, terraform supports an authentication mechanism that just uses the local instance's service principal, so no extra authentication material is needed for provisioning the test runners as separate cloud instances. Interesting to consider.
Sorry I failed to understand this, what is mean by separate cloud instances?
The usual situation is the control host (where terraform runs) is outside the cloud. Like, on your workstation. I'm talking about a scenario where kdevops and terraform are running on an instance in the same cloud as where your target nodes are going to run.
Quoted 8 lines
Quoted 4 lines
In that case, terraform can scoop up the service principal for that instance, and use it in place of dot file authentication parameters. So you don't have to maintain the dot file on the instance that is running terraform, if it is already in the cloud.
I see, so you first bring that cloud instance, and then use that as your command and control for test nodes. Does that first cloud instance need to be instantiated through another kdevops setup?
Quoted 6 lines
No, you can create it with terraform or from the cloud console. Start it up when you want, or leave it running.
I'm wondering if we can get a cloud devops pipeline to trigger it when it sees a PR against a watched repo. Haven't really thought that through.
OK well so when we add a new commit to kdevops we should ideally try to run a test if the commit is related to terraform or all commits to just verify we don't break cloud instance support. Given we already rely on dedicated github hosted servers, if such hosted servers have the credential files in place then the below would work to automatically trigger a cloud instance test.
For example below could be a .github/workflows/aws.yml which would run on every single commit we submit to kdevops. This would just do a bringup and destroy.
If the console creates a cloud instance, and github can get access to it as a dedicated self hosted runner, and it will persist then that guest can simply just be a bare bones guest to run ansible as the command and control center. And the example below could be used to push github actions onto the runner.
Could this step be removed and have github directly do the bringup? I suspect we can rely on github docker containers which just run kdevops configs for the cloud for it, but they won't be hosted on the respective cloud of choice. Other than that, it would be nice if github integrated support for instantiating a simple guest just to be command control. Otherwise a dedicated self hosted runner outside of the cloud could just have the credentials file.
The kdeops-ci repo [0] has sufficient workflow branch examples, some of which we are already using for different subsystems to trigger runs with kevops on dedicated servers. If a subsystem wanted to leverage the cloud their defconfig would just have the setup for the respective cloud of choice. For example defconfigs/linux-xfs-kpd is what se use for testing pushes to branches to the linux-kdevops/linux-xfs-kpd and only Carlos has access to push to that repo. Which dedicated self hosted runner is opaque to kdevops, but it can easily have cloud credentials to easily do bringup.
I hadn't enabled the below on kdevops just because of lack of time of testing, and trying to then optimize the cost so that a super bare bones cloud istanace would be used for every kdevops commit pushed. However, that's a bit extreme of course, and we can tone it down to only commits perhaps that touch kdevops. But if that works, then by all means so would a push to a git tree as we do with linux-kdevops/linux-xfs-kpd using the examples in the branches for /linux-kdevops/kdevops-ci.
[0] https://github.com/linux-kdevops/kdevops-ci
name: Run kdevops on self-hosted runner for aws cloud instances
on:
workflow_dispatch: # Add this for manual triggering of the workflow
jobs:
run-kdevops:
name: Run kdevops CI
runs-on: [self-hosted, Linux, X64]
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Set CI metadata for kdevops-results-archive
run: |
echo "$(basename ${{ github.repository }})" > ci.trigger LOG=$(git log -1 --pretty=format:"%s") > ci.subject echo "cloud aws:" $LOG" > ci.subject
# Start out pessimistic
echo "not ok" > ci.result
echo "Nothing to write home about." > ci.commit_extra
- name: Set kdevops path
run: echo "KDEVOPS_PATH=$GITHUB_WORKSPACE" >> $GITHUB_ENV
- name: Configure git
run: |
git config --global --add safe.directory '*' git config --global user.name "kdevops"
git config --global user.email "[email protected]"
- name: Run kdevops make defconfig-repo
run: |
KDEVOPS_TREE_REF="${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}" SHORT_PREFIX="$(echo ${KDEVOPS_TREE_REF:0:12})"
make KDEVOPS_HOSTS_PREFIX="$SHORT_PREFIX" defconfig-aws
- name: Run kdevops make
run: |
make -j$(nproc)
- name: Run kdevops make bringup
run: |
make bringup
- name: Just check the kernel
run: |
make uname
echo "ok" > ci.result
- name: Start SSH Agent
if: always() # Ensure this step runs even if previous steps failed
uses: webfactory/[email protected]
with:
ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }}
- name: Build our kdevops archive results
if: always() # This ensures the step runs even if previous steps failed
run: |
make ci-archive
- name: Upload our kdevops results archive
if: always() # This ensures the step runs even if previous steps failed
uses: actions/upload-artifact@v4
with:
name: kdevops-ci-results
path: ${{ env.KDEVOPS_PATH }}/archive/*.zip
# Ensure make destroy always runs, even on failure
- name: Run kdevops make destroy
if: always() # This ensures the step runs even if previous steps failed
run: |
make destroy
Quoted 46 lines · Luis Chamberlain
Quoted 27 lines · Chuck Lever
Quoted 17 lines · Luis Chamberlain
Quoted 16 lines · Chuck Lever
Quoted 9 lines · Luis Chamberlain
Quoted 5 lines · Chuck Lever
On a related topic, if we ever want to fully support running the kdevops /control host/ in the cloud, terraform supports an authentication mechanism that just uses the local instance's service principal, so no extra authentication material is needed for provisioning the test runners as separate cloud instances. Interesting to consider.
Sorry I failed to understand this, what is mean by separate cloud instances?
The usual situation is the control host (where terraform runs) is outside the cloud. Like, on your workstation. I'm talking about a scenario where kdevops and terraform are running on an instance in the same cloud as where your target nodes are going to run.
Quoted 8 lines
Quoted 4 lines
In that case, terraform can scoop up the service principal for that instance, and use it in place of dot file authentication parameters. So you don't have to maintain the dot file on the instance that is running terraform, if it is already in the cloud.
I see, so you first bring that cloud instance, and then use that as your command and control for test nodes. Does that first cloud instance need to be instantiated through another kdevops setup?
Quoted 6 lines
No, you can create it with terraform or from the cloud console. Start it up when you want, or leave it running.
I'm wondering if we can get a cloud devops pipeline to trigger it when it sees a PR against a watched repo. Haven't really thought that through.
OK well so when we add a new commit to kdevops we should ideally try to run a test if the commit is related to terraform or all commits to just verify we don't break cloud instance support. Given we already rely on dedicated github hosted servers, if such hosted servers have the credential files in place then the below would work to automatically trigger a cloud instance test.
For example below could be a
.github/workflows/aws.ymlwhich would run on every single commit we submit to kdevops. This would just do a bringup and destroy.
Quoted 7 lines
If the console creates a cloud instance, and github can get access to it as a dedicated self hosted runner, and it will persist then that guest can simply just be a bare bones guest to run ansible as the command and control center. And the example below could be used to push github actions onto the runner.
Could this step be removed and have github directly do the bringup?
I think you can hide the authentication material in one or more GitHub secrets, if we were to explicitly enable support for passing terraform authentication parameters via environment variables.
Then the GH Action can just run the terraform steps ("make bringup" and "make destroy") without effort. But more below.
Quoted 6 lines
I suspect we can rely on github docker containers which just run kdevops configs for the cloud for it, but they won't be hosted on the respective cloud of choice. Other than that, it would be nice if github integrated support for instantiating a simple guest just to be command control. Otherwise a dedicated self hosted runner outside of the cloud could just have the credentials file.
As far as I can tell, the major four all implement devops pipelines that can hook into GitHub pull requests and bring up instances and what not. Seems like that might obviate the need for managing authentication material on GitHub itself...?
Check out AWS CloudBuild:
https://docs.aws.amazon.com/codebuild/latest/userguide/concepts.html
You give it a little yaml file that tells it to watch a GitHub repo and then execute steps on AWS when it notices a PR. Very much like a GitHub Action, but the runner is on AWS and not GitHub, and because the activity is triggered by an AWS service (CloudBuild) the API authentication step is already done.
OCI calls this "Instance Principal Authentication":
Basically the service principal in the instance where terraform is running is used to provision new terraform resources.
So we set up one of these CloudBuild projects for each cloud provider, and for any PR in linux-kdevops/kdevops, it will try a bringup/destroy sequence on that cloud. The build logs can be placed in a public spot on the AWS web site.
</handwave>
So I would need to add a Kconfig widget to each cloud provider's submenu to select "Instance Principal Authentication" (or whatever terminology that cloud provider users) and then kdevops will set up the terraform provider block properly for that scenario.
Quoted 10 lines
The kdeops-ci repo [0] has sufficient workflow branch examples, some of which we are already using for different subsystems to trigger runs with kevops on dedicated servers. If a subsystem wanted to leverage the cloud their defconfig would just have the setup for the respective cloud of choice. For example defconfigs/linux-xfs-kpd is what se use for testing pushes to branches to the linux-kdevops/linux-xfs-kpd and only Carlos has access to push to that repo. Which dedicated self hosted runner is opaque to kdevops, but it can easily have cloud credentials to easily do bringup.
Quoted 3 lines
I hadn't enabled the below on kdevops just because of lack of time of testing, and trying to then optimize the cost so that a super bare bones cloud istanace would be used for every kdevops commit pushed.
As long as the instance configurations are fairly minimal, they can be kept in the free tier on most if not all cloud providers. You do need an active credit card to open a cloud account, though.
Quoted 81 lines
However,
that's a bit extreme of course, and we can tone it down to only commits perhaps that touch kdevops. But if that works, then by all means so would a push to a git tree as we do with linux-kdevops/linux-xfs-kpd using the examples in the branches for /linux-kdevops/kdevops-ci.[0] https://github.com/linux-kdevops/kdevops-ci
name: Run kdevops on self-hosted runner for aws cloud instances
on:
workflow_dispatch: # Add this for manual triggering of the workflowjobs:
run-kdevops:
name: Run kdevops CI
runs-on: [self-hosted, Linux, X64]
steps:
- name: Checkout repository
uses: actions/checkout@v4- name: Set CI metadata for kdevops-results-archive
run: |
echo "$(basename ${{ github.repository }})" > ci.trigger LOG=$(git log -1 --pretty=format:"%s") > ci.subject echo "cloud aws:" $LOG" > ci.subject
# Start out pessimistic
echo "not ok" > ci.result
echo "Nothing to write home about." > ci.commit_extra- name: Set kdevops path
run: echo "KDEVOPS_PATH=$GITHUB_WORKSPACE" >> $GITHUB_ENV- name: Configure git
run: |
git config --global --add safe.directory '*' git config --global user.name "kdevops"
git config --global user.email "[email protected]"- name: Run kdevops make defconfig-repo
run: |
KDEVOPS_TREE_REF="${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}" SHORT_PREFIX="$(echo ${KDEVOPS_TREE_REF:0:12})"
make KDEVOPS_HOSTS_PREFIX="$SHORT_PREFIX" defconfig-aws- name: Run kdevops make
run: |
make -j$(nproc)- name: Run kdevops make bringup
run: |
make bringup- name: Just check the kernel
run: |
make uname
echo "ok" > ci.result- name: Start SSH Agent
if:always()# Ensure this step runs even if previous steps failed
uses: webfactory/[email protected]
with:
ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }}- name: Build our kdevops archive results
if:always()# This ensures the step runs even if previous steps failed
run: |
make ci-archive- name: Upload our kdevops results archive
if:always()# This ensures the step runs even if previous steps failed
uses: actions/upload-artifact@v4
with:
name: kdevops-ci-results
path: ${{ env.KDEVOPS_PATH }}/archive/*.zip# Ensure make destroy always runs, even on failure
- name: Run kdevops make destroy
if:always()# This ensures the step runs even if previous steps failed
run: |
make destroy
Signature · 1 line
Chuck Lever
Quoted 6 lines · On Thu, Apr 03, 2025 at 10:49:06AM -0400, [email protected] wrote:
From: Chuck Lever <[email protected]>
Instead of storing authentication secrets in the kdevops .config file, pull them from the authentication profiles already set up in ~/.oci/config. This arrangement is more secure.
Quoted 6 lines
terraform's API authentication is now managed outside of
Kconfig, as is done with AWS. An update todocs/kdevops-terraform.mdto follow.Suggested-by Luis Chamberlain <[email protected]>Signed-off-by Chuck Lever <[email protected]>
Awesome! Might be useful to suggest this can be also be an enabler to use cloud for CIs and later share .config through kdevops-results-archive so to enable to reproduce tests on the cloud easily without revealing cloud secrets.
Luis