Skip to contentSkip to conversation

backports: add kernel uploading support

PATCH Superseded Luis R. Rodriguez 2 participants 13 messages May 23, 2013 – Jul 27, 2013 Library code

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v3 cover backports: add kernel uploading support

From: "Luis R. Rodriguez" <[email protected]>

On this series I address Johannes' feedback, clean things up a bit more, and add the --kup-test argument on the last patch.

Luis R. Rodriguez (8):
lib/bpgit.py: enable extra arguments on git describe lib/bpgit.py: add git status support lib/bpgit.py: add git clean support lib/bpgit.py: add git tree verification support lib/bpgit.py: add support for git paranoia lib/bpgpg.py: add simple gpg helper for gpg signing lib/bpkup.py: add or own kernel uploader helper lib gentree.py: add kernel upload support

 gentree.py   |  237 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++--
 lib/bpgit.py |   93 ++++++++++++++++++++++-
 lib/bpgpg.py |   22 ++++++
 lib/bpkup.py |   44 +++++++++++
 4 files changed, 388 insertions(+), 8 deletions(-)
 create mode 100644 lib/bpgpg.py
 create mode 100644 lib/bpkup.py
Signature · 1 line
1.7.10.4
#1 lore raw

PATCH v3 1/8 lib/bpgit.py: enable extra arguments on git describe

From: "Luis R. Rodriguez" <[email protected]>

This lets us throw at it whatever extra stuff we want.

---
 gentree.py   |    5 +++--
 lib/bpgit.py |   10 ++++++++--
 2 files changed, 11 insertions(+), 4 deletions(-)
gentree.py +3 -2
index 86b3f3c..2810415 100755
@@ -447,9 +447,10 @@ def process(kerneldir, outdir, copy_list_file, git_revision=None,
         backports_version = "(see git)"
         kernel_version = "(see git)"
     else:
-        backports_version = git.describe(tree=source_dir)
+        backports_version = git.describe(tree=source_dir, extra_args=['--long'])
         kernel_version = git.describe(rev=args.git_revision or 'HEAD',
-                                      tree=args.kerneldir)
+                                      tree=args.kerneldir,
+                                      extra_args=['--long'])
     f = open(os.path.join(args.outdir, 'versions'), 'w')
     f.write('BACKPORTS_VERSION="%s"\n' % backports_version)
     f.write('BACKPORTED_KERNEL_VERSION="%s"\n' % kernel_version)
lib/bpgit.py +8 -2
index 31b7de3..9a3f430 100644
@@ -27,8 +27,14 @@ def rev_parse(rev='HEAD', tree=None):
         raise SHAError()
     return sha
 
-def describe(rev='HEAD', tree=None):
-    process = subprocess.Popen(['git', 'describe', '--always', '--long', rev],
+def describe(rev='HEAD', tree=None, extra_args=[]):
+    cmd = ['git', 'describe', '--always']
+
+    cmd.extend(extra_args)
+    if rev is not None:
+        cmd.append(rev)
+
+    process = subprocess.Popen(cmd,
                                stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
                                close_fds=True, universal_newlines=True, cwd=tree)
     stdout = process.communicate()[0]
Signature · 1 line
1.7.10.4
#2 lore raw

PATCH v3 2/8 lib/bpgit.py: add git status support

In reply to Luis R. Rodriguez

From: "Luis R. Rodriguez" <[email protected]>

We use --porcelain given that this spits out the results in an easy-to-parse format for scripts and will remain stable across git versions and regardless of user configuration.

We will use this later.

---
 lib/bpgit.py |   54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 54 insertions(+)
lib/bpgit.py +54 -0
index 9a3f430..3f7eb1b 100644
@@ -27,6 +27,60 @@ def rev_parse(rev='HEAD', tree=None):
         raise SHAError()
     return sha
 
+def status(tree=None):
+    '''
+    For interpretation of the porcelain output refer to
+    the git status(1) man page. In summary the first column is the
+    index state, the second represents the working tree state and
+    the third column are files in cases of renames. This gives back
+    None in case no changes are present, otherwise it returns a list
+    of dict entries with key values: index, work_tree, and files. The
+    files are a list of all files found on that entry, in case of a rename
+    this would consist of a list of 2 files.
+
+    As an example if you had this on your git status:
+
+    R  udev/foo.sh -> poo/foo.sh
+    D  scripts/bar.sh
+    ?? .poo.py.swp
+
+    This would be transposed into the following dict:
+
+        results = status(tree=your_git_tree_path)
+        if not results:
+            return 0
+        for entry in results:
+            print entry
+
+    {'files': [' udev/foo.sh', 'poo/foo.sh'], 'index': 'R', 'work_tree': ' '}
+    {'files': [' scripts/bar.sh'], 'index': 'D', 'work_tree': ' '}
+    {'files': [' .poo.py.swp'], 'index': '?', 'work_tree': '?'}
+    '''
+    def split_status(entry):
+        if len(entry) == 0:
+            return None
+        if len(entry) == 1:
+            return dict(index=entry[0], work_tree=None, files=None)
+        if len(entry) == 2:
+            return dict(index=entry[0], work_tree=entry[1], files=None)
+        else:
+            return dict(index=entry[0], work_tree=entry[1],
+                        files=entry[2:].split(' -> '))
+
+    cmd = ['git', 'status', '--porcelain']
+                                                                                
+    process = subprocess.Popen(cmd,
+            stdout=subprocess.PIPE, stderr=subprocess.STDOUT, close_fds=True,
+            universal_newlines=True, cwd=tree)
+    stdout = process.communicate()[0]
+    process.wait()
+    _check(process)
+
+    list_status = stdout.split('\n')
+    if (len(list_status) == 1 and list_status[0] == ''):
+        return None
+    return [split_status(entry) for entry in list_status]
+
 def describe(rev='HEAD', tree=None, extra_args=[]):
     cmd = ['git', 'describe', '--always']
 
Signature · 1 line
1.7.10.4
#3 lore raw

PATCH v3 3/8 lib/bpgit.py: add git clean support

In reply to Luis R. Rodriguez

From: "Luis R. Rodriguez" <[email protected]>

We'll use this later.

---
 lib/bpgit.py |   10 ++++++++++
 1 file changed, 10 insertions(+)
lib/bpgit.py +10 -0
index 3f7eb1b..8ce8568 100644
@@ -27,6 +27,16 @@ def rev_parse(rev='HEAD', tree=None):
         raise SHAError()
     return sha
 
+def clean(tree=None):
+    cmd = ['git', 'clean', '-f', '-x', '-d', '-q']
+
+    process = subprocess.Popen(cmd,
+                               stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
+                               close_fds=True, universal_newlines=True, cwd=tree)
+    stdout = process.communicate()[0]
+    process.wait()
+    _check(process)
+
 def status(tree=None):
     '''
     For interpretation of the porcelain output refer to
Signature · 1 line
1.7.10.4
#4 lore raw

PATCH v3 4/8 lib/bpgit.py: add git tree verification support

In reply to Luis R. Rodriguez

From: "Luis R. Rodriguez" <[email protected]>

This verifies that the tree provided has its last commit both tagged and also digitally signed. In short it does:

  git tag -v $(git describe --dirty)

This will be used later.

---
 lib/bpgit.py |   12 ++++++++++++
 1 file changed, 12 insertions(+)
lib/bpgit.py +12 -0
index 8ce8568..da5453c 100644
@@ -107,6 +107,18 @@ def describe(rev='HEAD', tree=None, extra_args=[]):
 
     return stdout.strip()
 
+def verify(git_tree):
+    tag = describe(rev=None, tree=git_tree, extra_args=['--dirty'])
+    cmd = ['git', 'tag', '-v', tag]
+
+    process = subprocess.Popen(cmd,
+                               stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
+                               close_fds=True, universal_newlines=True, cwd=git_tree)
+    stdout = process.communicate()[0]
+    process.wait()
+
+    return dict(r=process.returncode, output=stdout)
+
 def init(tree=None):
     process = subprocess.Popen(['git', 'init'],
                                stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
Signature · 1 line
1.7.10.4
#5 lore raw

PATCH v3 5/8 lib/bpgit.py: add support for git paranoia

In reply to Luis R. Rodriguez

From: "Luis R. Rodriguez" <[email protected]>

This cleans sanitizes a source tree and ensures
no content is present from what was intended.

---
 lib/bpgit.py |    7 +++++++
 1 file changed, 7 insertions(+)
lib/bpgit.py +7 -0
index da5453c..44d22ad 100644
@@ -119,6 +119,13 @@ def verify(git_tree):
 
     return dict(r=process.returncode, output=stdout)
 
+def paranoia(tree):
+    clean(tree)
+    poo = status(tree)
+    if (poo):
+        return dict(r=-1, output=poo)
+    return verify(tree)
+
 def init(tree=None):
     process = subprocess.Popen(['git', 'init'],
                                stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
Signature · 1 line
1.7.10.4
#6 lore raw

PATCH v3 6/8 lib/bpgpg.py: add simple gpg helper for gpg signing

In reply to Luis R. Rodriguez

From: "Luis R. Rodriguez" <[email protected]>

Note that kup expects --armor --detach-sign, we'll pass that as part of our extra_args. The python gpg lib doesn't provide support for random arguments as this one, so just add our own little helper and carry it over.

---
 lib/bpgpg.py |   22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
 create mode 100644 lib/bpgpg.py
lib/bpgpg.py new file +22 -0
new file mode 100644
index 0000000..3b5ec78
@@ -0,0 +1,22 @@
+import subprocess, os
+
+class GpgError(Exception):
+    pass
+class ExecutionError(GpgError):
+    def __init__(self, errcode):
+        self.error_code = errcode
+
+def sign(input_file, extra_args=[]):
+    cmd = ['gpg', '--sign']
+
+    cmd.extend(extra_args)
+    cmd.append(input_file)
+
+    process = subprocess.Popen(cmd,
+                               stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
+                               close_fds=True, universal_newlines=True)
+    stdout = process.communicate()[0]
+    process.wait()
+    if process.returncode != 0:
+        raise ExecutionError(process.returncode)
+    return stdout
Signature · 1 line
1.7.10.4
#7 lore raw

PATCH v3 7/8 lib/bpkup.py: add or own kernel uploader helper lib

In reply to Luis R. Rodriguez

From: "Luis R. Rodriguez" <[email protected]>

kup is written in perl, add some basic python interfaces for it to allow us to upload to kernel.org backport releases.

---
 lib/bpkup.py |   44 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)
 create mode 100644 lib/bpkup.py
lib/bpkup.py new file +44 -0
new file mode 100644
index 0000000..03504bb
@@ -0,0 +1,44 @@
+import subprocess, os
+
+class KupError(Exception):
+    pass
+class ExecutionError(KupError):
+    def __init__(self, errcode):
+        self.error_code = errcode
+
+def _check(process):
+    if process.returncode != 0:
+        raise ExecutionError(process.returncode)
+
+def mkdir(path):
+    cmd = ['kup', 'mkdir', path]
+    process = subprocess.Popen(cmd,
+                               stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
+                               close_fds=True, universal_newlines=True)
+    stdout = process.communicate()[0]
+    process.wait()
+    _check(process)
+
+    return stdout
+
+def ls(path=None):
+    cmd = ['kup', 'ls', path]
+    process = subprocess.Popen(cmd,
+                               stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
+                               close_fds=True, universal_newlines=True)
+    stdout = process.communicate()[0]
+    process.wait()
+    _check(process)
+
+    return stdout
+
+def put(tar_bz2, signed_tar, path):
+    cmd = ['kup', 'put', tar_bz2, signed_tar, path]
+    process = subprocess.Popen(cmd,
+                               stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
+                               close_fds=True, universal_newlines=True)
+    stdout = process.communicate()[0]
+    process.wait()
+    _check(process)
+
+    return stdout
Signature · 1 line
1.7.10.4
#8 lore raw

PATCH v3 8/8 gentree.py: add kernel upload support

In reply to Luis R. Rodriguez

From: "Luis R. Rodriguez" <[email protected]>

This streamlines our release process making it easier for transitioning releases onto other backport developers.

---
 gentree.py |  232 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 228 insertions(+), 4 deletions(-)
gentree.py +228 -4
index 2810415..c700309 100755
@@ -4,6 +4,7 @@
 #
 
 import argparse, sys, os, errno, shutil, re, subprocess
+import tarfile, bz2
 
 # find self
 source_dir = os.path.abspath(os.path.dirname(__file__))
@@ -11,6 +12,8 @@ sys.path.append(source_dir)
 # and import libraries we have
 from lib import kconfig, patch, make
 from lib import bpgit as git
+from lib import bpgpg as gpg
+from lib import bpkup as kup
 
 def read_copy_list(copyfile):
     """
@@ -250,6 +253,169 @@ def git_debug_snapshot(args, name):
         return
     git.commit_all(name, tree=args.outdir)
 
+def get_rel_spec_stable(rel):
+    """
+    Returns release specs for a linux-stable backports based release.
+    """
+    if ("rc" in rel):
+        m = re.match(r"(?P<VERSION>\d+)\.+" \
+                     "(?P<PATCHLEVEL>\d+)[.]*" \
+                     "(?P<SUBLEVEL>\d*)" \
+                     "[-rc]+(?P<RC_VERSION>\d+)\-+" \
+                     "(?P<RELMOD_UPDATE>\d+)[-]*" \
+                     "(?P<RELMOD_TYPE>[usnpc]*)", \
+                     rel)
+    else:
+        m = re.match(r"(?P<VERSION>\d+)\.+" \
+                     "(?P<PATCHLEVEL>\d+)[.]*" \
+                     "(?P<SUBLEVEL>\d*)\-+" \
+                     "(?P<RELMOD_UPDATE>\d+)[-]*" \
+                     "(?P<RELMOD_TYPE>[usnpc]*)", \
+                     rel)
+    if (not m):
+        return m
+    return m.groupdict()
+
+def get_rel_spec_next(rel):
+    """
+    Returns release specs for a linux-next backports based release.
+    """
+    m = re.match(r"(?P<DATE_VERSION>\d+)[-]*" \
+                 "(?P<RELMOD_UPDATE>\d*)[-]*" \
+                 "(?P<RELMOD_TYPE>[usnpc]*)", \
+                 rel)
+    if (not m):
+        return m
+    return m.groupdict()
+
+def get_rel_prep(rel):
+    """
+    Returns a dict with prep work details we need prior to
+    uploading a backports release to kernel.org
+    """
+    rel_specs = get_rel_spec_stable(rel)
+    is_stable = True
+    rel_tag = ""
+    paths = list()
+    if (not rel_specs):
+        rel_specs = get_rel_spec_next(rel)
+        if (not rel_specs):
+            sys.stdout.write("rel: %s\n" % rel)
+            return None
+        if (rel_specs['RELMOD_UPDATE'] == '0' or
+            rel_specs['RELMOD_UPDATE'] == '1'):
+            return None
+        is_stable = False
+        date = rel_specs['DATE_VERSION']
+        year = date[0:4]
+        if (len(year) != 4):
+            return None
+        month = date[4:6]
+        if (len(month) != 2):
+            return None
+        day = date[6:8]
+        if (len(day) != 2):
+            return None
+        paths.append(year)
+        paths.append(month)
+        paths.append(day)
+        rel_tag = "backports-" + rel.replace(rel_specs['RELMOD_TYPE'], "")
+    else:
+        ignore = "-"
+        if (not rel_specs['RELMOD_UPDATE']):
+            return None
+        if (rel_specs['RELMOD_UPDATE'] == '0'):
+            return None
+        ignore += rel_specs['RELMOD_UPDATE']
+        if (rel_specs['RELMOD_TYPE'] != ''):
+            ignore += rel_specs['RELMOD_TYPE']
+        base_rel = rel.replace(ignore, "")
+        paths.append(base_rel)
+        rel_tag = "v" + rel.replace(rel_specs['RELMOD_TYPE'], "")
+
+    rel_prep = dict(stable = is_stable,
+                    expected_tag = rel_tag,
+                    paths_to_create = paths)
+    return rel_prep
+
+def create_tar_and_bz2(tar_name, dir_to_tar):
+    """
+    We need both a tar file and bzip2 for kernel.org, the tar file
+    gets signed, then we upload the compressed version, kup-server
+    in the backend decompresses and verifies the tarball against
+    our signature.
+    """
+    parent = os.path.dirname(tar_name)
+    tar = tarfile.open(tar_name, "w")
+    tar.add(dir_to_tar)
+    tar.close()
+
+    tar_file = open(tar_name, "r")
+
+    bz2_file = bz2.BZ2File(tar_name + ".bz2", 'wb', compresslevel=9)
+    bz2_file.write(tar_file.read())
+    bz2_file.close()
+
+    tar.close()
+
+def upload_release(args, rel_prep, logwrite=lambda x:None):
+    """
+    Given a path of a relase make tarball out of it, PGP sign it, and
+    then upload it to kernel.org using kup.
+
+    The linux-next based release do not require a RELMOD_UPDATE
+    given that typically only one release is made per day. Using
+    RELMOD_UPDATE for these releases is allowed though and if
+    present it must be > 1.
+
+    The linux-stable based releases require a RELMOD_UPDATE.
+
+    RELMOD_UPDATE must be numeric and > 0 just as the RC releases
+    of the Linux kernel.
+
+    The tree must also be tagged with the respective release, without
+    the RELMOD_TYPE. For linux-next based releases this consists of
+    backports- followed by DATE_VERSION and if RELMOD_TYPE is present.
+    For linux-stable releases this consists of v followed by the
+    full release version except the RELMOD_TYPE.
+
+    Uploads will not be allowed if these rules are not followed.
+    """
+    korg_path = "/pub/linux/kernel/projects/backports"
+
+    if (rel_prep['stable']):
+        korg_path += "stable"
+
+    parent = os.path.dirname(args.outdir)
+    release = os.path.basename(args.outdir)
+    tar_name = parent + '/' + release + ".tar"
+    bzip2_name = tar_name + ".bz2"
+
+    create_tar_and_bz2(tar_name, args.outdir)
+
+    logwrite(gpg.sign(tar_name, extra_args=['--armor', '--detach-sign']))
+
+    logwrite("------------------------------------------------------")
+
+    if (not args.kup_test):
+        logwrite("About to upload, current target path contents:")
+    else:
+        logwrite("kup-test: current target path contents:")
+
+    logwrite(kup.ls(path=korg_path))
+
+    for path in rel_prep['paths_to_create']:
+        korg_path += '/' + path
+        if (not args.kup_test):
+            logwrite(kup.mkdir(path))
+    if (not args.kup_test):
+        logwrite(kup.put(bzip2_name, tar_name + '.asc', korg_path))
+        logwrite("\nFinished upload!\n")
+        logwrite("Target path contents:")
+        logwrite(kup.ls(path=korg_path))
+    else:
+        kup_cmd = "kup put /\n\t\t%s /\n\t\t%s /\n\t\t%s" % (bzip2_name, tar_name + '.asc', korg_path)
+        logwrite("kup-test: skipping cmd: %s" % kup_cmd)
 
 def _main():
     # set up and parse arguments
@@ -278,6 +444,15 @@ def _main():
                         help='Print more verbose information')
     parser.add_argument('--extra-driver', nargs=2, metavar=('<source dir>', '<copy-list>'), type=str,
                         action='append', default=[], help='Extra driver directory/copy-list.')
+    parser.add_argument('--kup', const=True, default=False, action="store_const",
+                        help='For maintainers: upload a release to kernel.org')
+    parser.add_argument('--kup-test', const=True, default=False, action="store_const",
+                        help='For maintainers: do all the work as if you were about to ' +
+                             'upload to kernel.org but do not do the final `kup put` ' +
+                             'and also do not run any `kup mkdir` commands. This will ' +
+                             'however run `kup ls` on the target paths so ' +
+                             'at the very least we test your kup configuration. ' +
+                             'If this is your first time uploading use this first!')
     args = parser.parse_args()
 
     def logwrite(msg):
@@ -289,16 +464,22 @@ def _main():
                    git_revision=args.git_revision, clean=args.clean,
                    refresh=args.refresh, base_name=args.base_name,
                    gitdebug=args.gitdebug, verbose=args.verbose,
-                   extra_driver=args.extra_driver, logwrite=logwrite)
+                   extra_driver=args.extra_driver,
+                   kup=args.kup,
+                   kup_test=args.kup_test,
+                   logwrite=logwrite)
 
 def process(kerneldir, outdir, copy_list_file, git_revision=None,
             clean=False, refresh=False, base_name="Linux", gitdebug=False,
-            verbose=False, extra_driver=[], logwrite=lambda x:None,
+            verbose=False, extra_driver=[], kup=False,
+            kup_test=False,
+            logwrite=lambda x:None,
             git_tracked_version=False):
     class Args(object):
         def __init__(self, kerneldir, outdir, copy_list_file,
                      git_revision, clean, refresh, base_name,
-                     gitdebug, verbose, extra_driver):
+                     gitdebug, verbose, extra_driver, kup,
+                     kup_test):
             self.kerneldir = kerneldir
             self.outdir = outdir
             self.copy_list = copy_list_file
@@ -309,10 +490,48 @@ def process(kerneldir, outdir, copy_list_file, git_revision=None,
             self.gitdebug = gitdebug
             self.verbose = verbose
             self.extra_driver = extra_driver
+            self.kup = kup
+            self.kup_test = kup_test
+    def git_paranoia(tree=None, logwrite=lambda x:None):
+        data = git.paranoia(tree)
+        if (data['r'] != 0):
+            logwrite('Cannot use %s' % tree)
+            logwrite('%s' % data['output'])
+            sys.exit(data['r'])
+        else:
+            logwrite('Validated tree: %s' % tree)
+
     args = Args(kerneldir, outdir, copy_list_file,
                 git_revision, clean, refresh, base_name,
-                gitdebug, verbose, extra_driver)
+                gitdebug, verbose, extra_driver, kup, kup_test)
+    rel_prep = None
+
     # start processing ...
+    if (args.kup or args.kup_test):
+        git_paranoia(source_dir, logwrite)
+        git_paranoia(kerneldir, logwrite)
+
+        rel_describe = git.describe(rev=None, tree=source_dir, extra_args=['--dirty'])
+        release = os.path.basename(args.outdir)
+        version = release.replace("backports-", "")
+
+        rel_prep = get_rel_prep(version)
+        if (not rel_prep):
+            logwrite('Invalid backports release name: %s' % release)
+            logwrite('For rules on the release name see upload_release()')
+            sys.exit(1)
+        rel_type = "linux-stable"
+        if (not rel_prep['stable']):
+            rel_type = "linux-next"
+        if (rel_prep['expected_tag'] != rel_describe):
+            logwrite('Unexpected %s based backports release tag on' % rel_type)
+            logwrite('the backports tree tree: %s\n' % rel_describe)
+            logwrite('You asked to make a release with this ')
+            logwrite('directory name: %s' % release)
+            logwrite('The actual expected tag we should find on')
+            logwrite('the backports tree then is: %s\n' % rel_prep['expected_tag'])
+            logwrite('For rules on the release name see upload_release()')
+            sys.exit(1)
 
     copy_list = read_copy_list(args.copy_list)
     deplist = read_dependencies(os.path.join(source_dir, 'dependencies'))
@@ -342,6 +561,8 @@ def process(kerneldir, outdir, copy_list_file, git_revision=None,
 
     # FIXME: should we add a git version of this (e.g. --git-extra-driver)?
     for src, copy_list in args.extra_driver:
+        if (args.kup or args.kup_test):
+            git_paranoia(src)
         copy_files(src, read_copy_list(open(copy_list, 'r')), args.outdir)
 
     git_debug_snapshot(args, 'Add driver sources')
@@ -537,6 +758,9 @@ def process(kerneldir, outdir, copy_list_file, git_revision=None,
         fo.close()
     git_debug_snapshot(args, "disable unsatisfied Makefile parts")
 
+    if (args.kup or args.kup_test):
+        upload_release(args, rel_prep, logwrite=logwrite)
+
     logwrite('Done!')
     return 0
 
Signature · 1 line
1.7.10.4
#9 lore raw

Re: [PATCH v3 8/8] gentree.py: add kernel upload support

gentree.py def git_debug_snapshot(args, name): · line 260

Quoted 6 lines
gentree.py +5 -0
+def get_rel_spec_stable(rel):
+    """
+    Returns release specs for a linux-stable backports based release.
+    """
+    if ("rc" in rel):

I really think you should drop those parentheses on ifs, it looks weirdly C-ish and this is python :-)

gentree.py def git_debug_snapshot(args, name): · line 264

Quoted 4 lines
gentree.py +4 -0
+        m = re.match(r"(?P<VERSION>\d+)\.+" \
+                     "(?P<PATCHLEVEL>\d+)[.]*" \
+                     "(?P<SUBLEVEL>\d*)" \
+                     "[-rc]+(?P<RC_VERSION>\d+)\-+" \

"[-rc]+" seems weird, don't you just want "(-rc)?". And then in fact I think you don't need the "if rc in" above?

Quoted 1 line

+ if (not m):

ditto

Quoted 1 line

+ m = re.match(r"(?P<DATE_VERSION>\d+)[-]*" \

"[-]*" is odd, why not just write "-*" if that's what you want? Maybe you actually want "-+", otherwise there's no boundary between DATE_VERSION and RELMOD_UPDATE.

gentree.py def git_debug_snapshot(args, name): · line 303

Quoted 4 lines
gentree.py +4 -0
+    if (not rel_specs):
+        rel_specs = get_rel_spec_next(rel)
+        if (not rel_specs):
+            sys.stdout.write("rel: %s\n" % rel)

print ?

Quoted 11 lines
gentree.py +11 -0
+        paths.append(year)
+        paths.append(month)
+        paths.append(day)
+        rel_tag = "backports-" + rel.replace(rel_specs['RELMOD_TYPE'], "")
+    else:
+        ignore = "-"
+        if (not rel_specs['RELMOD_UPDATE']):
+            return None
+        if (rel_specs['RELMOD_UPDATE'] == '0'):
+            return None
+        ignore += rel_specs['RELMOD_UPDATE']

gentree.py def git_debug_snapshot(args, name): · line 339

Quoted 10 lines
gentree.py +10 -0
+        if (rel_specs['RELMOD_TYPE'] != ''):
+            ignore += rel_specs['RELMOD_TYPE']
+        base_rel = rel.replace(ignore, "")
+        paths.append(base_rel)
+        rel_tag = "v" + rel.replace(rel_specs['RELMOD_TYPE'], "")
+
+    rel_prep = dict(stable = is_stable,
+                    expected_tag = rel_tag,
+                    paths_to_create = paths)
+    return rel_prep

I truly don't understand this function. Also, what's with the dict? Why not just return a tuple:
return (is_stable, rel_tag, paths)

gentree.py def git_debug_snapshot(args, name): · line 394

Quoted 6 lines
gentree.py +6 -0
+    parent = os.path.dirname(args.outdir)
+    release = os.path.basename(args.outdir)
+    tar_name = parent + '/' + release + ".tar"
+    bzip2_name = tar_name + ".bz2"
+
+    create_tar_and_bz2(tar_name, args.outdir)

I'd recommend creating it in a temporary directory. In fact, I'd recommend making args.outdir optional in this case (which needs some refactoring) and making _it_ a subdir in a temp directory too?

Quoted 7 lines
gentree.py +2 -1
@@ -289,16 +464,22 @@ def _main():
                    git_revision=args.git_revision, clean=args.clean,
                    refresh=args.refresh, base_name=args.base_name,
                    gitdebug=args.gitdebug, verbose=args.verbose,
-                   extra_driver=args.extra_driver, logwrite=logwrite)
+                   extra_driver=args.extra_driver,
+                   kup=args.kup,

gentree.py def _main(): · line 477

Quoted 10 lines
gentree.py +5 -1
+                   kup_test=args.kup_test,
+                   logwrite=logwrite)
 
 def process(kerneldir, outdir, copy_list_file, git_revision=None,
             clean=False, refresh=False, base_name="Linux", gitdebug=False,
-            verbose=False, extra_driver=[], logwrite=lambda x:None,
+            verbose=False, extra_driver=[], kup=False,
+            kup_test=False,
+            logwrite=lambda x:None,
             git_tracked_version=False):

That may just be me, but maybe you should be consistent and put more things on one line (or reformat the other lines?)

gentree.py def process(kerneldir, outdir, copy_list_file, git_revision=None, · line 495

Quoted 7 lines
gentree.py +3 -0
@@ -309,10 +490,48 @@ def process(kerneldir, outdir, copy_list_file, git_revision=None,
             self.gitdebug = gitdebug
             self.verbose = verbose
             self.extra_driver = extra_driver
+            self.kup = kup
+            self.kup_test = kup_test
+    def git_paranoia(tree=None, logwrite=lambda x:None):

This is a really strange place to define a function?

gentree.py def process(kerneldir, outdir, copy_list_file, git_revision=None, · line 511

Quoted 3 lines
gentree.py +2 -0
     # start processing ...
+    if (args.kup or args.kup_test):
+        git_paranoia(source_dir, logwrite)

This part might not be handled by my git-revision suggestion, but then maybe it'd be better to actually point the scripting to a

gentree.py def process(kerneldir, outdir, copy_list_file, git_revision=None, · line 515

Quoted 4 lines
gentree.py +4 -0
+        git_paranoia(kerneldir, logwrite)
+
+        rel_describe = git.describe(rev=None, tree=source_dir, extra_args=['--dirty'])
+        release = os.path.basename(args.outdir)

Taking the file path for a release description seems like a strange thing to do? Maybe you should pass it on the command line?

johannes

#10 lore raw

Re: [PATCH v3 8/8] gentree.py: add kernel upload support

Quoted 9 lines · Johannes Berg

gentree.py def git_debug_snapshot(args, name): · line 260

Quoted 6 lines
gentree.py +5 -0
+def get_rel_spec_stable(rel):
+    """
+    Returns release specs for a linux-stable backports based release.
+    """
+    if ("rc" in rel):

I really think you should drop those parentheses on ifs, it looks weirdly C-ish and this is python :-)

Ah I missed that on my new iteration, will address this next.

Quoted 6 lines

gentree.py def git_debug_snapshot(args, name): · line 264

Quoted 4 lines
gentree.py +4 -0
+        m = re.match(r"(?P<VERSION>\d+)\.+" \
+                     "(?P<PATCHLEVEL>\d+)[.]*" \
+                     "(?P<SUBLEVEL>\d*)" \
+                     "[-rc]+(?P<RC_VERSION>\d+)\-+" \

"[-rc]+" seems weird, don't you just want "(-rc)?"

Indeed, thanks.

Quoted 1 line

And then in fact I think you don't need the "if rc in" above?

Will revise, thanks.

Quoted 2 lines
Quoted 1 line

+ if (not m):

ditto

OK!

Quoted 5 lines
Quoted 1 line

+ m = re.match(r"(?P<DATE_VERSION>\d+)[-]*" \

"[-]*" is odd, why not just write "-*" if that's what you want? Maybe you actually want "-+", otherwise there's no boundary between DATE_VERSION and RELMOD_UPDATE.

-* it is, given that RELMOD_UPDATE can be empty as per the defined rules on naming convention on releases for backports described in upload_release()

Quoted 6 lines

gentree.py def git_debug_snapshot(args, name): · line 303

Quoted 4 lines
gentree.py +4 -0
+    if (not rel_specs):
+        rel_specs = get_rel_spec_next(rel)
+        if (not rel_specs):
+            sys.stdout.write("rel: %s\n" % rel)

print ?

I'll nuke.

Quoted 22 lines
Quoted 11 lines
gentree.py +11 -0
+        paths.append(year)
+        paths.append(month)
+        paths.append(day)
+        rel_tag = "backports-" + rel.replace(rel_specs['RELMOD_TYPE'], "")
+    else:
+        ignore = "-"
+        if (not rel_specs['RELMOD_UPDATE']):
+            return None
+        if (rel_specs['RELMOD_UPDATE'] == '0'):
+            return None
+        ignore += rel_specs['RELMOD_UPDATE']

gentree.py def git_debug_snapshot(args, name): · line 339

Quoted 10 lines
gentree.py +10 -0
+        if (rel_specs['RELMOD_TYPE'] != ''):
+            ignore += rel_specs['RELMOD_TYPE']
+        base_rel = rel.replace(ignore, "")
+        paths.append(base_rel)
+        rel_tag = "v" + rel.replace(rel_specs['RELMOD_TYPE'], "")
+
+    rel_prep = dict(stable = is_stable,
+                    expected_tag = rel_tag,
+                    paths_to_create = paths)
+    return rel_prep
Quoted 1 line

I truly don't understand this function.

OK so I figured it'd be good to lay out a defined pattern for releases. Based on previous releases and practices this is defined on the documentation of upload_release(). Given that in this iteration --git-revision is not used *and* that older releases had -snpc possible tags and that these additional postfix arguments were never used as part of the signed digital tag on the tree for releases this routine:

1) verifies that rules on upload_release() are followed
2) throws back the expected git tag on the backports tree
3) throws back the list of directories that need to be created, given kup does not support mkdir -p, and given that the directory path varies if its a stable release also determine if its a stable release or not

Quoted 3 lines

Also, what's with the dict? Why
not just return a tuple:
return (is_stable, rel_tag, paths)

Just what I have been used to, is tuple preferred in the Python world in these cases where you have to return multiple data?

Quoted 8 lines

gentree.py def git_debug_snapshot(args, name): · line 394

Quoted 6 lines
gentree.py +6 -0
+    parent = os.path.dirname(args.outdir)
+    release = os.path.basename(args.outdir)
+    tar_name = parent + '/' + release + ".tar"
+    bzip2_name = tar_name + ".bz2"
+
+    create_tar_and_bz2(tar_name, args.outdir)

I'd recommend creating it in a temporary directory.

Under what parent directory?

Quoted 3 lines

In fact, I'd
recommend making args.outdir optional in this case (which needs some refactoring) and making _it_ a subdir in a temp directory too?

Sure, but what parent directory to use then?

Quoted 10 lines
Quoted 10 lines
gentree.py +4 -1
@@ -289,16 +464,22 @@ def _main():
                    git_revision=args.git_revision, clean=args.clean,
                    refresh=args.refresh, base_name=args.base_name,
                    gitdebug=args.gitdebug, verbose=args.verbose,
-                   extra_driver=args.extra_driver, logwrite=logwrite)
+                   extra_driver=args.extra_driver,
+                   kup=args.kup,
+                   kup_test=args.kup_test,
+                   logwrite=logwrite)
 
Quoted 10 lines

gentree.py def _main(): · line 477

Quoted 7 lines
gentree.py +3 -1
 def process(kerneldir, outdir, copy_list_file, git_revision=None,
             clean=False, refresh=False, base_name="Linux", gitdebug=False,
-            verbose=False, extra_driver=[], logwrite=lambda x:None,
+            verbose=False, extra_driver=[], kup=False,
+            kup_test=False,
+            logwrite=lambda x:None,
             git_tracked_version=False):

That may just be me, but maybe you should be consistent and put more things on one line (or reformat the other lines?)

I'm more of a fan of the later given that then patches address argument per argument. Will send a patch for that, or address this after.

Quoted 9 lines

gentree.py def process(kerneldir, outdir, copy_list_file, git_revision=None, · line 495

Quoted 7 lines
gentree.py +3 -0
@@ -309,10 +490,48 @@ def process(kerneldir, outdir, copy_list_file, git_revision=None,
             self.gitdebug = gitdebug
             self.verbose = verbose
             self.extra_driver = extra_driver
+            self.kup = kup
+            self.kup_test = kup_test
+    def git_paranoia(tree=None, logwrite=lambda x:None):

This is a really strange place to define a function?

It didn't seem to make sense to share this globally on the file. Is this not good practice?

Quoted 6 lines

gentree.py def process(kerneldir, outdir, copy_list_file, git_revision=None, · line 511

Quoted 3 lines
gentree.py +2 -0
     # start processing ...
+    if (args.kup or args.kup_test):
+        git_paranoia(source_dir, logwrite)

This part might not be handled by my git-revision suggestion, but then maybe it'd be better to actually point the scripting to a

It seems you left off writing what you wanted to on this part of the thread. I would assume you were alluding to the idea that perhaps its good to point the script to a backports git tag to use to check out as well ?

Quoted 7 lines

gentree.py def process(kerneldir, outdir, copy_list_file, git_revision=None, · line 515

Quoted 4 lines
gentree.py +4 -0
+        git_paranoia(kerneldir, logwrite)
+
+        rel_describe = git.describe(rev=None, tree=source_dir, extra_args=['--dirty'])
+        release = os.path.basename(args.outdir)

Taking the file path for a release description seems like a strange thing to do? Maybe you should pass it on the command line?

Indeed, this is just trying to reuse the args.outdir but technically we should be able to infer the target release name from the backports git tag in place with the caveat that we still have to address the additional flags for releases (-s -n -p -c -u). If that's done though then maybe its better to not require an args.outdir and instead have a args.tmp_rel_path or else assume a /tmp/ random dir would be used and use the git tag from backports to infer the release (-s -n -p -c -u still pending as additional tags to peg later).

Thanks for the feedback, this really helps.

Luis

#11 lore raw

Re: [PATCH v3 8/8] gentree.py: add kernel upload support

Quoted 8 lines · Luis R. Rodriguez

OK so I figured it'd be good to lay out a defined pattern for releases. Based on previous releases and practices this is defined on the documentation of upload_release(). Given that in this iteration --git-revision is not used *and* that older releases had -snpc possible tags and that these additional postfix arguments were never used as part of the signed digital tag on the tree for releases this routine:

Quoted 6 lines

1) verifies that rules on upload_release() are followed
2) throws back the expected git tag on the backports tree
3) throws back the list of directories that need to be created, given kup does not support mkdir -p, and given that the directory path varies if its a stable release also determine if its a stable release or not

Huh, ok ... whatever ;-)

Quoted 6 lines
Quoted 3 lines

Also, what's with the dict? Why
not just return a tuple:
return (is_stable, rel_tag, paths)

Just what I have been used to, is tuple preferred in the Python world in these cases where you have to return multiple data?

It seems to mostly be, unless the keys are in some way dynamic. I'll bet you can find somebody with another opinion though ;-)

Quoted 9 lines
Quoted 8 lines

gentree.py def git_debug_snapshot(args, name): · line 394

Quoted 6 lines
gentree.py +6 -0
+    parent = os.path.dirname(args.outdir)
+    release = os.path.basename(args.outdir)
+    tar_name = parent + '/' + release + ".tar"
+    bzip2_name = tar_name + ".bz2"
+
+    create_tar_and_bz2(tar_name, args.outdir)

I'd recommend creating it in a temporary directory.

Quoted 7 lines

Under what parent directory?

Quoted 3 lines

In fact, I'd
recommend making args.outdir optional in this case (which needs some refactoring) and making _it_ a subdir in a temp directory too?

Sure, but what parent directory to use then?

Well I have this pattern a few times in git-tracker, just create and nuke a temp dir, that's totally easy here:

from lib import tempdir # shipped in the backport sources

with tempdir.tempdir() as my_tempdir:
os.chdir(my_tempdir)

or whatever. It'll be nuked after the with block finishes.

Quoted 6 lines
Quoted 2 lines

That may just be me, but maybe you should be consistent and put more things on one line (or reformat the other lines?)

I'm more of a fan of the later given that then patches address argument per argument. Will send a patch for that, or address this after.

makes sense.

Quoted 10 lines
Quoted 9 lines

gentree.py def process(kerneldir, outdir, copy_list_file, git_revision=None, · line 495

Quoted 7 lines
gentree.py +3 -0
@@ -309,10 +490,48 @@ def process(kerneldir, outdir, copy_list_file, git_revision=None,
             self.gitdebug = gitdebug
             self.verbose = verbose
             self.extra_driver = extra_driver
+            self.kup = kup
+            self.kup_test = kup_test
+    def git_paranoia(tree=None, logwrite=lambda x:None):

This is a really strange place to define a function?

Quoted 2 lines

It didn't seem to make sense to share this globally on the file. Is this not good practice?

*shrug*
It seems weird to me, but then I mostly live in a C world :)

Quoted 11 lines
Quoted 6 lines

gentree.py def process(kerneldir, outdir, copy_list_file, git_revision=None, · line 511

Quoted 3 lines
gentree.py +2 -0
     # start processing ...
+    if (args.kup or args.kup_test):
+        git_paranoia(source_dir, logwrite)

This part might not be handled by my git-revision suggestion, but then maybe it'd be better to actually point the scripting to a

It seems you left off writing what you wanted to on this part of the thread. I would assume you were alluding to the idea that perhaps its good to point the script to a backports git tag to use to check out as well ?

Heh. Yeah, I guess I'm slowly losing it ...

Yes, I think I meant that.

Quoted 7 lines
Quoted 5 lines

gentree.py def process(kerneldir, outdir, copy_list_file, git_revision=None, · line 515

Quoted 2 lines
gentree.py +2 -0
+        rel_describe = git.describe(rev=None, tree=source_dir, extra_args=['--dirty'])
+        release = os.path.basename(args.outdir)

Taking the file path for a release description seems like a strange thing to do? Maybe you should pass it on the command line?

Indeed, this is just trying to reuse the args.outdir

Well I'm not sure I'd use the outdir, that also seems a bit odd. Actually the thing is, you don't actually _want_ an outdir. In this case, where you're directly uploading, you don't care about the interim result at all, you only care about the tarball & it being uploaded. You could completely work in a temp dir.

That makes me think that maybe this all should be a wrapper script kinda like git-tracker? A separate tool that just calls into this after doing all the paranoia stuff?

Although then I'd still recommend using (the equivalent of) --git-revision to avoid all the paranoia stuff and potential races with it.

johannes

#12 lore raw

Re: [PATCH v3 8/8] gentree.py: add kernel upload support

Quoted 7 lines · Johannes Berg

That makes me think that maybe this all should be a wrapper script kinda like git-tracker? A separate tool that just calls into this after doing all the paranoia stuff?

Although then I'd still recommend using (the equivalent of) --git-revision to avoid all the paranoia stuff and potential races with it.

So far --git-revision proves to be much slower than a git reset --hard variant, but both are reasonable options. I'll try to actually instead add a --git-revision-reset which would just be an alternative to --git-revision but that requires the tree to be non-bare, and does a tree reset, and git-paranoia.

Luis