Skip to contentSkip to conversation

gitlab pipeline check/watch script improvements

PATCH In review Cleber Rosa 2 participants 11 messages Sep 4, 2020

lore
Share & tools
lore
#0 lore raw

cover gitlab pipeline check/watch script improvements

This is a collection of refactors and improvements based on feedback received.

The biggest difference is that now the script can be used right after a git push, despite the pipeline having been created yet or not. The proper status of the pipeline (not created, pending, etc) will be given as output.

Cleber Rosa (7):
scripts/ci/gitlab-pipeline-status: make branch name configurable scripts/ci/gitlab-pipeline-status: improve message regarding timeout scripts/ci/gitlab-pipeline-status: give early feedback on running
pipelines
scripts/ci/gitlab-pipeline-status: refactor parser creation scripts/ci/gitlab-pipeline-status: handle keyboard interrupts scripts/ci/gitlab-pipeline-status: use more descriptive exceptions scripts/ci/gitlab-pipeline-status: wait for pipeline creation

 scripts/ci/gitlab-pipeline-status | 63 +++++++++++++++++++++----------
 1 file changed, 44 insertions(+), 19 deletions(-)
Signature · 1 line
2.25.4
#1 lore raw

PATCH 1/7 scripts/ci/gitlab-pipeline-status: make branch name configurable

With the utility function get_local_staging_branch_commit(), the name of the branch is hard coded (including in the function name).

For extensibility reasons, let's make that configurable.

---
 scripts/ci/gitlab-pipeline-status | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
scripts/ci/gitlab-pipeline-status +6 -6
index 348a49b6a4..194dd4d0bb 100755
@@ -23,20 +23,20 @@ import time
 import sys
 
 
-def get_local_staging_branch_commit():
+def get_local_branch_commit(branch='staging'):
     """
     Returns the commit sha1 for the *local* branch named "staging"
     """
-    result = subprocess.run(['git', 'rev-parse', 'staging'],
+    result = subprocess.run(['git', 'rev-parse', branch],
                             stdin=subprocess.DEVNULL,
                             stdout=subprocess.PIPE,
                             stderr=subprocess.DEVNULL,
                             cwd=os.path.dirname(__file__),
                             universal_newlines=True).stdout.strip()
-    if result == 'staging':
-        raise ValueError("There's no local branch named 'staging'")
+    if result == branch:
+        raise ValueError("There's no local branch named '%s'" % branch)
     if len(result) != 40:
-        raise ValueError("Branch staging HEAD doesn't look like a sha1")
+        raise ValueError("Branch '%s' HEAD doesn't look like a sha1" % branch)
     return result
 
 
@@ -110,7 +110,7 @@ def main():
                               'for https://gitlab.com/qemu-project/qemu, that '
                               'is, "%(default)s"'))
     try:
-        default_commit = get_local_staging_branch_commit()
+        default_commit = get_local_branch_commit()
         commit_required = False
     except ValueError:
         default_commit = ''
Signature · 1 line
2.25.4
Reviewed-by on v1 1/7
#2 lore raw

Re: [PATCH 1/7] scripts/ci/gitlab-pipeline-status: make branch name configurable

Quoted 6 lines · Cleber Rosa

With the utility function get_local_staging_branch_commit(), the name of the branch is hard coded (including in the function name).

For extensibility reasons, let's make that configurable.

Quoted 44 lines
---
 scripts/ci/gitlab-pipeline-status | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
scripts/ci/gitlab-pipeline-status +6 -6
index 348a49b6a4..194dd4d0bb 100755
@@ -23,20 +23,20 @@ import time
 import sys
 
 
-def get_local_staging_branch_commit():
+def get_local_branch_commit(branch='staging'):
     """
     Returns the commit sha1 for the *local* branch named "staging"
     """
-    result = subprocess.run(['git', 'rev-parse', 'staging'],
+    result = subprocess.run(['git', 'rev-parse', branch],
                             stdin=subprocess.DEVNULL,
                             stdout=subprocess.PIPE,
                             stderr=subprocess.DEVNULL,
                             cwd=os.path.dirname(__file__),
                             universal_newlines=True).stdout.strip()
-    if result == 'staging':
-        raise ValueError("There's no local branch named 'staging'")
+    if result == branch:
+        raise ValueError("There's no local branch named '%s'" % branch)
     if len(result) != 40:
-        raise ValueError("Branch staging HEAD doesn't look like a sha1")
+        raise ValueError("Branch '%s' HEAD doesn't look like a sha1" % branch)
     return result
 
 
@@ -110,7 +110,7 @@ def main():
                               'for https://gitlab.com/qemu-project/qemu, that '
                               'is, "%(default)s"'))
     try:
-        default_commit = get_local_staging_branch_commit()
+        default_commit = get_local_branch_commit()
         commit_required = False
     except ValueError:
         default_commit = ''
#3 lore raw

PATCH 2/7 scripts/ci/gitlab-pipeline-status: improve message regarding timeout

In reply to Cleber Rosa

The script has its own timeout, which is about how long the script will wait (when called with --wait) for the pipeline to complete, and not necessarily for the pipeline to complete.

Hopefully this will new wording will be clearer.

---
 scripts/ci/gitlab-pipeline-status | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
scripts/ci/gitlab-pipeline-status +4 -1
index 194dd4d0bb..2a36f74696 100755
@@ -69,7 +69,10 @@ def wait_on_pipeline_success(timeout, interval,
     start = time.time()
     while True:
         if time.time() >= (start + timeout):
-            print("Waiting on the pipeline timed out")
+            msg = ("Timeout (-t/--timeout) of %i seconds reached, "
+                   "won't wait any longer for the pipeline to complete")
+            msg %= timeout
+            print(msg)
             return False
 
         status = get_pipeline_status(project_id, commit_sha)
Signature · 1 line
2.25.4
#4 lore raw

Re: [PATCH 2/7] scripts/ci/gitlab-pipeline-status: improve message regarding timeout

Quoted 5 lines · Cleber Rosa

The script has its own timeout, which is about how long the script will wait (when called with --wait) for the pipeline to complete, and not necessarily for the pipeline to complete.

Hopefully this will new wording will be clearer.

too many wills?

Quoted 11 lines
---
 scripts/ci/gitlab-pipeline-status | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
scripts/ci/gitlab-pipeline-status +0 -0
index 194dd4d0bb..2a36f74696 100755

scripts/ci/gitlab-pipeline-status def wait_on_pipeline_success(timeout, interval, · line 75

Quoted 8 lines
scripts/ci/gitlab-pipeline-status +4 -1
@@ -69,7 +69,10 @@ def wait_on_pipeline_success(timeout, interval,
     start = time.time()
     while True:
         if time.time() >= (start + timeout):
-            print("Waiting on the pipeline timed out")
+            msg = ("Timeout (-t/--timeout) of %i seconds reached, "
+                   "won't wait any longer for the pipeline to complete")
+            msg %= timeout
+            print(msg)

Isn't it more pythonic / cheaper to declare the msg format out of the loop and call directly:

print(msg % timeout)

?

Quoted 4 lines
scripts/ci/gitlab-pipeline-status +0 -0
             return False
 
         status = get_pipeline_status(project_id, commit_sha)
#5 lore raw

PATCH 3/7 scripts/ci/gitlab-pipeline-status: give early feedback on running pipelines

In reply to Cleber Rosa

When waiting for a pipeline to run and finish, it's better to give early feedback, and then sleep and wait, than the other wait around.

Specially for the first iteration, it's frustrating to see nothing while the script is sleeping.

---
 scripts/ci/gitlab-pipeline-status | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/ci/gitlab-pipeline-status +1 -1
index 2a36f74696..18609553be 100755
@@ -77,8 +77,8 @@ def wait_on_pipeline_success(timeout, interval,
 
         status = get_pipeline_status(project_id, commit_sha)
         if status['status'] == 'running':
-            time.sleep(interval)
             print('running...')
+            time.sleep(interval)
             continue
 
         if status['status'] == 'success':
Signature · 1 line
2.25.4
Reviewed-by on v1 3/7
#6 lore raw

Re: [PATCH 3/7] scripts/ci/gitlab-pipeline-status: give early feedback on running pipelines

Quoted 7 lines · Cleber Rosa

When waiting for a pipeline to run and finish, it's better to give early feedback, and then sleep and wait, than the other wait around.

Specially for the first iteration, it's frustrating to see nothing while the script is sleeping.

Quoted 19 lines
---
 scripts/ci/gitlab-pipeline-status | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/ci/gitlab-pipeline-status +1 -1
index 2a36f74696..18609553be 100755
@@ -77,8 +77,8 @@ def wait_on_pipeline_success(timeout, interval,
 
         status = get_pipeline_status(project_id, commit_sha)
         if status['status'] == 'running':
-            time.sleep(interval)
             print('running...')
+            time.sleep(interval)
             continue
 
         if status['status'] == 'success':
#7 lore raw

PATCH 4/7 scripts/ci/gitlab-pipeline-status: refactor parser creation

In reply to Cleber Rosa

Out of the main function.

---
 scripts/ci/gitlab-pipeline-status | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
scripts/ci/gitlab-pipeline-status +7 -4
index 18609553be..8355b6a427 100755
@@ -89,10 +89,7 @@ def wait_on_pipeline_success(timeout, interval,
         return False
 
 
-def main():
-    """
-    Script entry point
-    """
+def create_parser():
     parser = argparse.ArgumentParser(
         prog='pipeline-status',
         description='check or wait on a pipeline status')
@@ -127,7 +124,13 @@ def main():
     parser.add_argument('--verbose', action='store_true', default=False,
                         help=('A minimal verbosity level that prints the '
                               'overall result of the check/wait'))
+    return parser
 
+def main():
+    """
+    Script entry point
+    """
+    parser = create_parser()
     args = parser.parse_args()
 
     try:
Signature · 1 line
2.25.4
#8 lore raw

PATCH 5/7 scripts/ci/gitlab-pipeline-status: handle keyboard interrupts

In reply to Cleber Rosa

So that exits based on user requests are handled more gracefully.

---
 scripts/ci/gitlab-pipeline-status | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
scripts/ci/gitlab-pipeline-status +4 -2
index 8355b6a427..ced488f27c 100755
@@ -132,7 +132,7 @@ def main():
     """
     parser = create_parser()
     args = parser.parse_args()
-
+    success = False
     try:
         if args.wait:
             success = wait_on_pipeline_success(
@@ -145,9 +145,11 @@ def main():
                                          args.commit)
             success = status['status'] == 'success'
     except Exception as error:      # pylint: disable=W0703
-        success = False
         if args.verbose:
             print("ERROR: %s" % error.args[0])
+    except KeyboardInterrupt:
+        if args.verbose:
+            print("Exiting on user's request")
 
     if success:
         if args.verbose:
Signature · 1 line
2.25.4
#9 lore raw

PATCH 6/7 scripts/ci/gitlab-pipeline-status: use more descriptive exceptions

In reply to Cleber Rosa

For two very different error conditions.

---
 scripts/ci/gitlab-pipeline-status | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
scripts/ci/gitlab-pipeline-status +10 -2
index ced488f27c..628150ce0b 100755
@@ -23,6 +23,14 @@ import time
 import sys
 
 
+class CommunicationFailure(Exception):
+    """Failed to communicate to gitlab.com APIs."""
+
+
+class NoPipelineFound(Exception):
+    """Communication is successfull but pipeline is not found."""
+
+
 def get_local_branch_commit(branch='staging'):
     """
     Returns the commit sha1 for the *local* branch named "staging"
@@ -50,14 +58,14 @@ def get_pipeline_status(project_id, commit_sha1):
     connection.request('GET', url=url)
     response = connection.getresponse()
     if response.code != http.HTTPStatus.OK:
-        raise ValueError("Failed to receive a successful response")
+        raise CommunicationFailure("Failed to receive a successful response")
     json_response = json.loads(response.read())
 
     # As far as I can tell, there should be only one pipeline for the same
     # project + commit. If this assumption is false, we can add further
     # filters to the url, such as username, and order_by.
     if not json_response:
-        raise ValueError("No pipeline found")
+        raise NoPipelineFound("No pipeline found")
     return json_response[0]
 
 
Signature · 1 line
2.25.4
#10 lore raw

PATCH 7/7 scripts/ci/gitlab-pipeline-status: wait for pipeline creation

In reply to Cleber Rosa

When called in wait mode, this script will also wait for the pipeline to be get to a "running" state. Because many more status may be seen until a pipeline gets to "running", and those need to be handle too.

Reference: https://docs.gitlab.com/ee/api/pipelines.html#list-project-pipelines

---
 scripts/ci/gitlab-pipeline-status | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)
scripts/ci/gitlab-pipeline-status +13 -4
index 628150ce0b..bac8233079 100755
@@ -83,13 +83,22 @@ def wait_on_pipeline_success(timeout, interval,
             print(msg)
             return False
 
-        status = get_pipeline_status(project_id, commit_sha)
-        if status['status'] == 'running':
-            print('running...')
+        try:
+            status = get_pipeline_status(project_id, commit_sha)
+        except NoPipelineFound:
+            print('Pipeline has not been found, it may not have been created yet.')
+            time.sleep(1)
+            continue
+
+        pipeline_status = status['status']
+        status_to_wait = ('created', 'waiting_for_resource', 'preparing',
+                          'pending', 'running')
+        if pipeline_status in status_to_wait:
+            print('%s...' % pipeline_status)
             time.sleep(interval)
             continue
 
-        if status['status'] == 'success':
+        if pipeline_status == 'success':
             return True
 
         msg = "Pipeline failed, check: %s" % status['web_url']
Signature · 1 line
2.25.4