patman: Collect review tags and comments from Patchwork
v2 cover patman: Collect review tags and comments from Patchwork
Patman is a useful tool for creating, checking sending out patches. It automates the creation of patches, simplifies checking them and handles cover letters and change logs.
However once patches are sent and reviewers add Reviewed-by tags, etc., these must be manually added into the commits using git before the next version of the series is sent. Also, the only way to look at patches is in the web interface.
It would be nice if patman could talk to Patchwork and collect the review tags itself. Even better if it could show the review comments in a command-line view patch-by-patch to speed up addressing comments.
This series adds these features to patman, talking directly to the Patchwork REST API. While pwclient seems like it could handle some of this, or at least provide a library to patman, the documentation[1] doesn't really explain what the commands do and it doesn't seem to work with the current Patchwork (e.g. pwclient git-am requires a patch number but Patchwork seems to use an ID now).
With these changes it is possible to type 'patman status' and see a list of new review tags. It is also possible to create a new branch with those tags and list out the review comments as small snippets on the command line.
I have been using these features for a little while and they seem useful, so herewith a series.
Some of the patches here clean up style problems in patman and other minor problems. See the last four patches for the main changes.
You can find it at u-boot-dm/patman-working[2]
[1] https://patchwork.readthedocs.io/projects/pwclient/en/latest/usage/
[2] https://gitlab.denx.de/u-boot/custodians/u-boot-dm/-/tree/patman-working
Changes in v2:
- Adjust the algorithm to handle patch/commit mismatch
- Correct Python style problems
- Use REST API instead of web pages
- Many changes to the whole series, including new patches
- Use of REST API instead of the web site
Simon Glass (30):
patman: Correct operation of -n azure/gitLab/travis: Add pygit2 as a dependency for tests
patman: Update how tests are run
patman: Fix whitespace errors in func_test
patman: Use capture_sys_output() consistently
patman: Fix remaining pylint3 warnings in func_test
patman: Allow linking a series with patchwork
patman: Fix indenting in patchstream
patman: Fix constant style in patchstream
patman: Rename functions in patchstream
patman: Rename variables in patchstream
patman: Drop unused args in patchstream
patman: Fix up argument/return docs in patchstream
patman: Move warning collection to a function
patman: Attach warnings to individual patches
patman: Convert 'Series-xxx' tag errors into warnings
patman: Drop unused signoff member
patman: Add a test for PatchStream tags
patman: Add some tests for warnings
patman: Convert testBasic() to use an interator
patman: Require tags to be before sign-off
patman: Fix spelling of plural for warning
patman: Don't ignore lines starting with hash
patman: Allow showing a Commit as a string
patman: Improve handling of files
patman: Detect missing upstream in CountCommitsToBranch
patman: Support checking for review tags in patchwork
patman: Support updating a branch with review tags
patman: Support parsing of review snippets
patman: Support listing comments from patchwork
.azure-pipelines.yml | 2 +- .gitlab-ci.yml | 2 +- .travis.yml | 1 + tools/buildman/control.py | 6 +- tools/patman/README | 111 +- tools/patman/checkpatch.py | 18 +- tools/patman/commit.py | 5 + tools/patman/control.py | 71 +- tools/patman/func_test.py | 980 ++++++++++++++++-- tools/patman/gitutil.py | 10 +- tools/patman/main.py | 42 +- tools/patman/patchstream.py | 454 +++++--- tools/patman/series.py | 8 +- tools/patman/status.py | 480 +++++++++ tools/patman/terminal.py | 21 +- .../0001-pci-Correct-cast-for-sandbox.patch | 2 +- ...-for-sandbox-in-fdtdec_setup_mem_siz.patch | 2 +- tools/patman/test/test01.txt | 4 +- tools/patman/test_checkpatch.py | 6 +- 19 files changed, 1952 insertions(+), 273 deletions(-) create mode 100644 tools/patman/status.py
Signature · 1 line
2.29.0.rc2.309.g374f81d7ae-goog
PATCH v2 1/30 patman: Correct operation of -n
This operation was unfortunately broken by a recent change. It is now necessary to use -i in addition to -n, if there are errors or warnings in the patches.
Correct this by always showing the summary information.
---
(no changes since v1)
tools/patman/control.py | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)
tools/patman/control.py +5 -6
@@ -170,9 +170,8 @@ def send(args): ok = ok and gitutil.CheckSuppressCCConfig() its_a_go = ok or args.ignore_errors - if its_a_go: - email_patches( - col, series, cover_fname, patch_files, args.process_tags, - its_a_go, args.ignore_bad_tags, args.add_maintainers, - args.limit, args.dry_run, args.in_reply_to, args.thread, - args.smtp_server) + email_patches( + col, series, cover_fname, patch_files, args.process_tags, + its_a_go, args.ignore_bad_tags, args.add_maintainers, + args.limit, args.dry_run, args.in_reply_to, args.thread, + args.smtp_server)
Signature · 1 line
2.29.0.rc2.309.g374f81d7ae-goog
PATCH v2 2/30 azure/gitLab/travis: Add pygit2 as a dependency for tests
In reply to Simon GlassThis lets patman run all of its tests, rather than skipping quite a few.
---
(no changes since v1)
.azure-pipelines.yml | 2 +- .gitlab-ci.yml | 2 +- .travis.yml | 1 + 3 files changed, 3 insertions(+), 2 deletions(-)
.azure-pipelines.yml +1 -1
@@ -140,7 +140,7 @@ jobs: export USER=azure virtualenv -p /usr/bin/python3 /tmp/venv . /tmp/venv/bin/activate - pip install pyelftools pytest + pip install pyelftools pytest pygit2 export UBOOT_TRAVIS_BUILD_DIR=/tmp/sandbox_spl export PYTHONPATH=${UBOOT_TRAVIS_BUILD_DIR}/scripts/dtc/pylibfdt export PATH=${UBOOT_TRAVIS_BUILD_DIR}/scripts/dtc:${PATH}
.gitlab-ci.yml +1 -1
@@ -161,7 +161,7 @@ Run binman, buildman, dtoc, Kconfig and patman testsuites: export USER=gitlab; virtualenv -p /usr/bin/python3 /tmp/venv; . /tmp/venv/bin/activate; - pip install pyelftools pytest; + pip install pyelftools pytest pygit2; export UBOOT_TRAVIS_BUILD_DIR=/tmp/sandbox_spl; export PYTHONPATH="${UBOOT_TRAVIS_BUILD_DIR}/scripts/dtc/pylibfdt"; export PATH="${UBOOT_TRAVIS_BUILD_DIR}/scripts/dtc:${PATH}";
.travis.yml +1 -0
@@ -26,6 +26,7 @@ addons: - python3-sphinx - python3-virtualenv - python3-pip + - python3-pygit2 - swig - libpython-dev - iasl
Signature · 1 line
2.29.0.rc2.309.g374f81d7ae-goog
PATCH v2 3/30 patman: Update how tests are run
In reply to Simon GlassThe current instructions are out-of-date. Fix them.
---
(no changes since v1)
tools/patman/README | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)
tools/patman/README +3 -5
@@ -533,12 +533,10 @@ Most of these are indicated by a TODO in the code. It would be nice if this could handle the In-reply-to side of things. -The tests are incomplete, as is customary. Use the --test flag to run them, -and make sure you are in the tools/patman directory first: +The tests are incomplete, as is customary. Use the 'test' subcommand to run +them: - $ cd /path/to/u-boot - $ cd tools/patman - $ ./patman --test + $ tools/patman/patman test Error handling doesn't always produce friendly error messages - e.g. putting an incorrect tag in a commit may provide a confusing message.
Signature · 1 line
2.29.0.rc2.309.g374f81d7ae-goog
PATCH v2 4/30 patman: Fix whitespace errors in func_test
In reply to Simon GlassFix up various indentation and other minor things to make pylint3 happier.
---
(no changes since v1)
tools/patman/func_test.py | 37 +++++++++++++++++++------------------ 1 file changed, 19 insertions(+), 18 deletions(-)
tools/patman/func_test.py +19 -18
@@ -24,20 +24,20 @@ from patman.test_util import capture_sys_output try: import pygit2 - HAVE_PYGIT2= True + HAVE_PYGIT2 = True except ModuleNotFoundError: HAVE_PYGIT2 = False @contextlib.contextmanager def capture(): - oldout,olderr = sys.stdout, sys.stderr + oldout, olderr = sys.stdout, sys.stderr try: - out=[StringIO(), StringIO()] - sys.stdout,sys.stderr = out + out = [StringIO(), StringIO()] + sys.stdout, sys.stderr = out yield out finally: - sys.stdout,sys.stderr = oldout, olderr + sys.stdout, sys.stderr = oldout, olderr out[0] = out[0].getvalue() out[1] = out[1].getvalue()
@@ -160,10 +160,10 @@ class TestFunctional(unittest.TestCase): in_reply_to = mel count = 2 settings.alias = { - 'fdt': ['simon'], - 'u-boot': ['u-boot at lists.denx.de'], - 'simon': [ed], - 'fred': [fred], + 'fdt': ['simon'], + 'u-boot': ['u-boot at lists.denx.de'], + 'simon': [ed], + 'fred': [fred], } text = self.GetText('test01.txt')
@@ -177,9 +177,9 @@ class TestFunctional(unittest.TestCase): cc_file = series.MakeCcFile(process_tags, cover_fname, not ignore_bad_tags, add_maintainers, None) - cmd = gitutil.EmailPatches(series, cover_fname, args, - dry_run, not ignore_bad_tags, cc_file, - in_reply_to=in_reply_to, thread=None) + cmd = gitutil.EmailPatches( + series, cover_fname, args, dry_run, not ignore_bad_tags, + cc_file, in_reply_to=in_reply_to, thread=None) series.ShowActions(args, cmd, process_tags) cc_lines = open(cc_file, encoding='utf-8').read().splitlines() os.remove(cc_file)
@@ -221,8 +221,9 @@ class TestFunctional(unittest.TestCase): self.assertEqual(('%s %s\0%s' % (args[0], rick, stefan)), tools.ToUnicode(cc_lines[0])) - self.assertEqual(('%s %s\0%s\0%s\0%s' % (args[1], fred, ed, rick, - stefan)), tools.ToUnicode(cc_lines[1])) + self.assertEqual( + '%s %s\0%s\0%s\0%s' % (args[1], fred, ed, rick, stefan), + tools.ToUnicode(cc_lines[1])) expected = ''' This is a test of how the cover
@@ -256,8 +257,8 @@ Simon Glass (2): ''' lines = open(cover_fname, encoding='utf-8').read().splitlines() self.assertEqual( - 'Subject: [RFC PATCH v3 0/2] test: A test patch series', - lines[3]) + 'Subject: [RFC PATCH v3 0/2] test: A test patch series', + lines[3]) self.assertEqual(expected.splitlines(), lines[7:]) for i, fname in enumerate(args):
@@ -310,7 +311,7 @@ Changes in v2: tools.WriteFile(path, text, binary=False) index = self.repo.index index.add(fname) - author = pygit2.Signature('Test user', 'test at email.com') + author = pygit2.Signature('Test user', 'test at email.com') committer = author tree = index.write_tree() message = subject + '\n' + body
@@ -335,7 +336,7 @@ Changes in v2: author = pygit2.Signature('Test user', 'test at email.com') committer = author commit = repo.create_commit('HEAD', author, committer, - 'Created master', new_tree, []) + 'Created master', new_tree, []) self.make_commit_with_file('Initial commit', ''' Add a README
Signature · 1 line
2.29.0.rc2.309.g374f81d7ae-goog
PATCH v2 5/30 patman: Use capture_sys_output() consistently
In reply to Simon GlassOne test still uses its own function for capturing output. Modify it to use the standard one in test_util
---
(no changes since v1)
tools/patman/func_test.py | 20 ++------------------ 1 file changed, 2 insertions(+), 18 deletions(-)
tools/patman/func_test.py +2 -18
@@ -4,7 +4,6 @@ # Copyright 2017 Google, Inc # -import contextlib import os import re import shutil
@@ -12,8 +11,6 @@ import sys import tempfile import unittest -from io import StringIO - from patman import control from patman import gitutil from patman import patchstream
@@ -29,19 +26,6 @@ except ModuleNotFoundError: HAVE_PYGIT2 = False - at contextlib.contextmanager -def capture(): - oldout, olderr = sys.stdout, sys.stderr - try: - out = [StringIO(), StringIO()] - sys.stdout, sys.stderr = out - yield out - finally: - sys.stdout, sys.stderr = oldout, olderr - out[0] = out[0].getvalue() - out[1] = out[1].getvalue() - - class TestFunctional(unittest.TestCase): def setUp(self): self.tmpdir = tempfile.mkdtemp(prefix='patman.')
@@ -169,7 +153,7 @@ class TestFunctional(unittest.TestCase): text = self.GetText('test01.txt') series = patchstream.GetMetaDataForTest(text) cover_fname, args = self.CreatePatchesForTest(series) - with capture() as out: + with capture_sys_output() as out: patchstream.FixPatches(series, args) if cover_fname and series.get('cover'): patchstream.InsertCoverLetter(cover_fname, series, count)
@@ -184,7 +168,7 @@ class TestFunctional(unittest.TestCase): cc_lines = open(cc_file, encoding='utf-8').read().splitlines() os.remove(cc_file) - lines = out[0].splitlines() + lines = out[0].getvalue().splitlines() self.assertEqual('Cleaned %s patches' % len(series.commits), lines[0]) self.assertEqual('Change log missing for v2', lines[1]) self.assertEqual('Change log missing for v3', lines[2])
Signature · 1 line
2.29.0.rc2.309.g374f81d7ae-goog
PATCH v2 6/30 patman: Fix remaining pylint3 warnings in func_test
In reply to Simon GlassThis fixes all but the ones about too many variables/statements.
---
(no changes since v1)
tools/patman/func_test.py | 76 +++++++++++++++++++++++++++++---------- 1 file changed, 58 insertions(+), 18 deletions(-)
tools/patman/func_test.py +58 -18
@@ -4,6 +4,8 @@ # Copyright 2017 Google, Inc # +"""Functional tests for checking that patman behaves correctly""" + import os import re import shutil
@@ -27,6 +29,7 @@ except ModuleNotFoundError: class TestFunctional(unittest.TestCase): + """Functional tests for checking that patman behaves correctly""" def setUp(self): self.tmpdir = tempfile.mkdtemp(prefix='patman.') self.gitdir = os.path.join(self.tmpdir, 'git')
@@ -36,33 +39,69 @@ class TestFunctional(unittest.TestCase): shutil.rmtree(self.tmpdir) @staticmethod - def GetPath(fname): + def _get_path(fname): + """Get the path to a test file + + Args: + fname (str): Filename to obtain + + Returns: + str: Full path to file in the test directory + """ return os.path.join(os.path.dirname(os.path.realpath(sys.argv[0])), 'test', fname) @classmethod - def GetText(self, fname): - return open(self.GetPath(fname), encoding='utf-8').read() + def _get_text(cls, fname): + """Read a file as text + + Args: + fname (str): Filename to read + + Returns: + str: Contents of file + """ + return open(cls._get_path(fname), encoding='utf-8').read() @classmethod - def GetPatchName(self, subject): + def _get_patch_name(cls, subject): + """Get the filename of a patch given its subject + + Args: + subject (str): Patch subject + + Returns: + str: Filename for that patch + """ fname = re.sub('[ :]', '-', subject) return fname.replace('--', '-') - def CreatePatchesForTest(self, series): + def _create_patches_for_test(self, series): + """Create patch files for use by tests + + This copies patch files from the test directory as needed by the series + + Args: + series (Series): Series containing commits to convert + + Returns: + tuple: + str: Cover-letter filename, or None if none + fname_list: list of str, each a patch filename + """ cover_fname = None fname_list = [] for i, commit in enumerate(series.commits): - clean_subject = self.GetPatchName(commit.subject) + clean_subject = self._get_patch_name(commit.subject) src_fname = '%04d-%s.patch' % (i + 1, clean_subject[:52]) fname = os.path.join(self.tmpdir, src_fname) - shutil.copy(self.GetPath(src_fname), fname) + shutil.copy(self._get_path(src_fname), fname) fname_list.append(fname) if series.get('cover'): src_fname = '0000-cover-letter.patch' cover_fname = os.path.join(self.tmpdir, src_fname) fname = os.path.join(self.tmpdir, src_fname) - shutil.copy(self.GetPath(src_fname), fname) + shutil.copy(self._get_path(src_fname), fname) return cover_fname, fname_list
@@ -137,7 +176,8 @@ class TestFunctional(unittest.TestCase): stefan = b'Stefan Br\xc3\xbcns <[email protected]>'.decode('utf-8') rick = 'Richard III <[email protected]>' mel = b'Lord M\xc3\xablchett <[email protected]>'.decode('utf-8') - ed = b'Lond Edmund Blackadd\xc3\xabr <weasel at blackadder.org'.decode('utf-8') + leb = (b'Lond Edmund Blackadd\xc3\xabr <weasel at blackadder.org'. + decode('utf-8')) fred = 'Fred Bloggs <[email protected]>' add_maintainers = [stefan, rick] dry_run = True
@@ -146,13 +186,13 @@ class TestFunctional(unittest.TestCase): settings.alias = { 'fdt': ['simon'], 'u-boot': ['u-boot at lists.denx.de'], - 'simon': [ed], + 'simon': [leb], 'fred': [fred], } - text = self.GetText('test01.txt') + text = self._get_text('test01.txt') series = patchstream.GetMetaDataForTest(text) - cover_fname, args = self.CreatePatchesForTest(series) + cover_fname, args = self._create_patches_for_test(series) with capture_sys_output() as out: patchstream.FixPatches(series, args) if cover_fname and series.get('cover'):
@@ -177,7 +217,7 @@ class TestFunctional(unittest.TestCase): self.assertIn('Dry run', lines[5]) self.assertIn('Send a total of %d patches' % count, lines[7]) line = 8 - for i, commit in enumerate(series.commits): + for i in range(len(series.commits)): self.assertEqual(' %s' % args[i], lines[line + 0]) line += 1 while 'Cc:' in lines[line]:
@@ -190,7 +230,7 @@ class TestFunctional(unittest.TestCase): self.assertEqual('Cover: 4 lines', lines[line + 4]) line += 5 self.assertEqual(' Cc: %s' % fred, lines[line + 0]) - self.assertEqual(' Cc: %s' % tools.FromUnicode(ed), + self.assertEqual(' Cc: %s' % tools.FromUnicode(leb), lines[line + 1]) self.assertEqual(' Cc: %s' % tools.FromUnicode(mel), lines[line + 2])
@@ -206,7 +246,7 @@ class TestFunctional(unittest.TestCase): self.assertEqual(('%s %s\0%s' % (args[0], rick, stefan)), tools.ToUnicode(cc_lines[0])) self.assertEqual( - '%s %s\0%s\0%s\0%s' % (args[1], fred, ed, rick, stefan), + '%s %s\0%s\0%s\0%s' % (args[1], fred, leb, rick, stefan), tools.ToUnicode(cc_lines[1])) expected = '''
@@ -311,7 +351,7 @@ Changes in v2: 'second' has base as upstream and three more: video, serial, bootm Returns: - pygit2 repository + pygit2.Repository: repository """ repo = pygit2.init_repository(self.gitdir) self.repo = repo
@@ -319,8 +359,8 @@ Changes in v2: author = pygit2.Signature('Test user', 'test at email.com') committer = author - commit = repo.create_commit('HEAD', author, committer, - 'Created master', new_tree, []) + _ = repo.create_commit('HEAD', author, committer, 'Created master', + new_tree, []) self.make_commit_with_file('Initial commit', ''' Add a README
Signature · 1 line
2.29.0.rc2.309.g374f81d7ae-goog
PATCH v2 7/30 patman: Allow linking a series with patchwork
In reply to Simon GlassAdd a new Series-links tag to tell patman how to find the series in patchwork. Each item is the series ID optionally preceded by the series version that the link refers to. An empty version indicates this is the latest series.
For example:
Series-links: 209816 1:203302
Documentation is added in a later patch.
---
(no changes since v1)
tools/patman/README | 15 +++++++++++++++ tools/patman/func_test.py | 1 + tools/patman/series.py | 2 +- 3 files changed, 17 insertions(+), 1 deletion(-)
tools/patman/README +15 -0
@@ -187,6 +187,21 @@ Series-name: name patman does not yet use it, but it is convenient to put the branch name here to help you keep track of multiple upstreaming efforts. +Series-links: [id | version:id]... + Set the ID of the series in patchwork. You can set this after you send + out the series and look in patchwork for the resulting series. The + URL you want is the one for the series itself, not any particular patch. + E.g. for http://patchwork.ozlabs.org/project/uboot/list/?series=187331 + the series ID is 187331. This property can have a list of series IDs, + one for each version of the series, e.g. + + Series-links: 1:187331 2:188434 189372 + + Patman always uses the one without a version, since it assumes this is + the latest one. When this tag is provided, patman can compare your local + branch against patchwork to see what new reviews your series has + collected ('patman status'). + Cover-letter: This is the patch set title blah blah
tools/patman/func_test.py +1 -0
@@ -417,6 +417,7 @@ Series for my board This series implements support for my glorious board. END +Series-links: 183237 ''', 'serial.c', '''The code for the serial driver is here''') self.make_commit_with_file('bootm: Make it boot', '''
tools/patman/series.py +1 -1
@@ -16,7 +16,7 @@ from patman import tools # Series-xxx tags that we understand valid_series = ['to', 'cc', 'version', 'changes', 'prefix', 'notes', 'name', - 'cover_cc', 'process_log'] + 'cover_cc', 'process_log', 'links'] class Series(dict): """Holds information about a patch series, including all tags.
Signature · 1 line
2.29.0.rc2.309.g374f81d7ae-goog
PATCH v2 8/30 patman: Fix indenting in patchstream
In reply to Simon GlassUpdate the indenting to keep pylint3 happy.
---
(no changes since v1)
tools/patman/patchstream.py | 29 ++++++++++++++++------------- 1 file changed, 16 insertions(+), 13 deletions(-)
tools/patman/patchstream.py +16 -13
@@ -2,6 +2,8 @@ # Copyright (c) 2011 The Chromium OS Authors. # +"""Handles parsing a stream of commits/emails from 'git log' or other source""" + import datetime import math import os
@@ -15,8 +17,8 @@ from patman import gitutil from patman.series import Series # Tags that we detect and remove -re_remove = re.compile('^BUG=|^TEST=|^BRANCH=|^Review URL:' - '|Reviewed-on:|Commit-\w*:') +re_remove = re.compile(r'^BUG=|^TEST=|^BRANCH=|^Review URL:' + r'|Reviewed-on:|Commit-\w*:') # Lines which are allowed after a TEST= line re_allowed_after_test = re.compile('^Signed-off-by:')
@@ -46,7 +48,7 @@ re_commit = re.compile('^commit ([0-9a-f]*)$') re_space_before_tab = re.compile('^[+].* \t') # Match indented lines for changes -re_leading_whitespace = re.compile('^\s') +re_leading_whitespace = re.compile(r'^\s') # States we can be in - can we use range() and still have comments? STATE_MSG_HEADER = 0 # Still in the message header
@@ -149,7 +151,7 @@ class PatchStream: return int(value) except ValueError as str: raise ValueError("%s: Cannot decode version info '%s'" % - (self.commit.hash, line)) + (self.commit.hash, line)) def FinalizeChange(self): """Finalize a (multi-line) change and add it to the series or commit"""
@@ -271,7 +273,7 @@ class PatchStream: # If we are not in a section, it is an unexpected END elif line == 'END': - raise ValueError("'END' wihout section") + raise ValueError("'END' wihout section") # Detect the commit subject elif not is_blank and self.state == STATE_PATCH_SUBJECT:
@@ -336,8 +338,9 @@ class PatchStream: value = change_id_match.group(1) if self.is_log: if self.commit.change_id: - raise ValueError("%s: Two Change-Ids: '%s' vs. '%s'" % - (self.commit.hash, self.commit.change_id, value)) + raise ValueError( + "%s: Two Change-Ids: '%s' vs. '%s'" % self.commit.hash, + self.commit.change_id, value) self.commit.change_id = value self.skip_blank = True
@@ -353,7 +356,7 @@ class PatchStream: self.change_version = self.ParseVersion(value, line) else: self.warn.append('Line %d: Ignoring Commit-%s' % - (self.linenum, name)) + (self.linenum, name)) # Detect the start of a new commit elif commit_match:
@@ -376,7 +379,7 @@ class PatchStream: # Suppress duplicate signoffs elif signoff_match: if (self.is_log or not self.commit or - self.commit.CheckDuplicateSignoff(signoff_match.group(1))): + self.commit.CheckDuplicateSignoff(signoff_match.group(1))): out = [line] # Well that means this is an ordinary line
@@ -385,7 +388,7 @@ class PatchStream: m = re_space_before_tab.match(line) if m: self.warn.append('Line %d/%d has space before tab' % - (self.linenum, m.start())) + (self.linenum, m.start())) # OK, we have a valid non-blank line out = [line]
@@ -418,7 +421,7 @@ class PatchStream: self.CloseCommit() if self.lines_after_test: self.warn.append('Found %d lines after TEST=' % - self.lines_after_test) + self.lines_after_test) def WriteMessageId(self, outfd): """Write the Message-Id into the output.
@@ -494,7 +497,7 @@ class PatchStream: else: if self.blank_count and (line == '-- ' or match): self.warn.append("Found possible blank line(s) at " - "end of file '%s'" % last_fname) + "end of file '%s'" % last_fname) outfd.write('+\n' * self.blank_count) outfd.write(line + '\n') self.blank_count = 0
@@ -502,7 +505,7 @@ class PatchStream: def GetMetaDataForList(commit_range, git_dir=None, count=None, - series = None, allow_overwrite=False): + series=None, allow_overwrite=False): """Reads out patch series metadata from the commits This does a 'git log' on the relevant commits and pulls out the tags we
Signature · 1 line
2.29.0.rc2.309.g374f81d7ae-goog
PATCH v2 9/30 patman: Fix constant style in patchstream
In reply to Simon GlassThese constants should use upper case. Update them to keep pylint3 happy.
---
(no changes since v1)
tools/patman/patchstream.py | 44 ++++++++++++++++++------------------- 1 file changed, 22 insertions(+), 22 deletions(-)
tools/patman/patchstream.py +22 -22
@@ -17,38 +17,38 @@ from patman import gitutil from patman.series import Series # Tags that we detect and remove -re_remove = re.compile(r'^BUG=|^TEST=|^BRANCH=|^Review URL:' +RE_REMOVE = re.compile(r'^BUG=|^TEST=|^BRANCH=|^Review URL:' r'|Reviewed-on:|Commit-\w*:') # Lines which are allowed after a TEST= line -re_allowed_after_test = re.compile('^Signed-off-by:') +RE_ALLOWED_AFTER_TEST = re.compile('^Signed-off-by:') # Signoffs -re_signoff = re.compile('^Signed-off-by: *(.*)') +RE_SIGNOFF = re.compile('^Signed-off-by: *(.*)') # Cover letter tag -re_cover = re.compile('^Cover-([a-z-]*): *(.*)') +RE_COVER = re.compile('^Cover-([a-z-]*): *(.*)') # Patch series tag -re_series_tag = re.compile('^Series-([a-z-]*): *(.*)') +RE_SERIES_TAG = re.compile('^Series-([a-z-]*): *(.*)') # Change-Id will be used to generate the Message-Id and then be stripped -re_change_id = re.compile('^Change-Id: *(.*)') +RE_CHANGE_ID = re.compile('^Change-Id: *(.*)') # Commit series tag -re_commit_tag = re.compile('^Commit-([a-z-]*): *(.*)') +RE_COMMIT_TAG = re.compile('^Commit-([a-z-]*): *(.*)') # Commit tags that we want to collect and keep -re_tag = re.compile('^(Tested-by|Acked-by|Reviewed-by|Patch-cc|Fixes): (.*)') +RE_TAG = re.compile('^(Tested-by|Acked-by|Reviewed-by|Patch-cc|Fixes): (.*)') # The start of a new commit in the git log -re_commit = re.compile('^commit ([0-9a-f]*)$') +RE_COMMIT = re.compile('^commit ([0-9a-f]*)$') # We detect these since checkpatch doesn't always do it -re_space_before_tab = re.compile('^[+].* \t') +RE_SPACE_BEFORE_TAB = re.compile('^[+].* \t') # Match indented lines for changes -re_leading_whitespace = re.compile(r'^\s') +RE_LEADING_WHITESPACE = re.compile(r'^\s') # States we can be in - can we use range() and still have comments? STATE_MSG_HEADER = 0 # Still in the message header
@@ -195,22 +195,22 @@ class PatchStream: out = [] line = line.rstrip('\n') - commit_match = re_commit.match(line) if self.is_log else None + commit_match = RE_COMMIT.match(line) if self.is_log else None if self.is_log: if line[:4] == ' ': line = line[4:] # Handle state transition and skipping blank lines - series_tag_match = re_series_tag.match(line) - change_id_match = re_change_id.match(line) - commit_tag_match = re_commit_tag.match(line) - cover_match = re_cover.match(line) - signoff_match = re_signoff.match(line) - leading_whitespace_match = re_leading_whitespace.match(line) + series_tag_match = RE_SERIES_TAG.match(line) + change_id_match = RE_CHANGE_ID.match(line) + commit_tag_match = RE_COMMIT_TAG.match(line) + cover_match = RE_COVER.match(line) + signoff_match = RE_SIGNOFF.match(line) + leading_whitespace_match = RE_LEADING_WHITESPACE.match(line) tag_match = None if self.state == STATE_PATCH_HEADER: - tag_match = re_tag.match(line) + tag_match = RE_TAG.match(line) is_blank = not line.strip() if is_blank: if (self.state == STATE_MSG_HEADER
@@ -280,7 +280,7 @@ class PatchStream: self.commit.subject = line # Detect the tags we want to remove, and skip blank lines - elif re_remove.match(line) and not commit_tag_match: + elif RE_REMOVE.match(line) and not commit_tag_match: self.skip_blank = True # TEST= should be the last thing in the commit, so remove
@@ -385,7 +385,7 @@ class PatchStream: # Well that means this is an ordinary line else: # Look for space before tab - m = re_space_before_tab.match(line) + m = RE_SPACE_BEFORE_TAB.match(line) if m: self.warn.append('Line %d/%d has space before tab' % (self.linenum, m.start()))
@@ -410,7 +410,7 @@ class PatchStream: out += self.commit.notes out += [''] + log elif self.found_test: - if not re_allowed_after_test.match(line): + if not RE_ALLOWED_AFTER_TEST.match(line): self.lines_after_test += 1 return out
Signature · 1 line
2.29.0.rc2.309.g374f81d7ae-goog
PATCH v2 10/30 patman: Rename functions in patchstream
In reply to Simon GlassRename these functions to lower case as per PEP8.
---
(no changes since v1)
tools/buildman/control.py | 6 +-- tools/patman/control.py | 6 +-- tools/patman/func_test.py | 6 +-- tools/patman/patchstream.py | 88 ++++++++++++++++----------------- tools/patman/test_checkpatch.py | 6 +-- 5 files changed, 56 insertions(+), 56 deletions(-)
tools/buildman/control.py +3 -3
@@ -276,14 +276,14 @@ def DoBuildman(options, args, toolchains=None, make_func=None, boards=None, options.branch) upstream_commit = gitutil.GetUpstream(options.git_dir, options.branch) - series = patchstream.GetMetaDataForList(upstream_commit, + series = patchstream.get_metadata_for_list(upstream_commit, options.git_dir, 1, series=None, allow_overwrite=True) - series = patchstream.GetMetaDataForList(range_expr, + series = patchstream.get_metadata_for_list(range_expr, options.git_dir, None, series, allow_overwrite=True) else: # Honour the count - series = patchstream.GetMetaDataForList(options.branch, + series = patchstream.get_metadata_for_list(options.branch, options.git_dir, count, series=None, allow_overwrite=True) else: series = None
tools/patman/control.py +3 -3
@@ -54,14 +54,14 @@ def prepare_patches(col, branch, count, start, end, ignore_binary): # Read the metadata from the commits to_do = count - end - series = patchstream.GetMetaData(branch, start, to_do) + series = patchstream.get_metadata(branch, start, to_do) cover_fname, patch_files = gitutil.CreatePatches( branch, start, to_do, ignore_binary, series) # Fix up the patch files to our liking, and insert the cover letter - patchstream.FixPatches(series, patch_files) + patchstream.fix_patches(series, patch_files) if cover_fname and series.get('cover'): - patchstream.InsertCoverLetter(cover_fname, series, to_do) + patchstream.insert_cover_letter(cover_fname, series, to_do) return series, cover_fname, patch_files def check_patches(series, patch_files, run_checkpatch, verbose):
tools/patman/func_test.py +3 -3
@@ -191,12 +191,12 @@ class TestFunctional(unittest.TestCase): } text = self._get_text('test01.txt') - series = patchstream.GetMetaDataForTest(text) + series = patchstream.get_metadata_for_test(text) cover_fname, args = self._create_patches_for_test(series) with capture_sys_output() as out: - patchstream.FixPatches(series, args) + patchstream.fix_patches(series, args) if cover_fname and series.get('cover'): - patchstream.InsertCoverLetter(cover_fname, series, count) + patchstream.insert_cover_letter(cover_fname, series, count) series.DoChecks() cc_file = series.MakeCcFile(process_tags, cover_fname, not ignore_bad_tags, add_maintainers,
tools/patman/patchstream.py +44 -44
266 lines not on this page · raw message
tools/patman/test_checkpatch.py +3 -3
@@ -148,15 +148,15 @@ Signed-off-by: Simon Glass <[email protected]> expfd.write(expected) expfd.close() - # Normally by the time we call FixPatch we've already collected + # Normally by the time we call fix_patch we've already collected # metadata. Here, we haven't, but at least fake up something. - # Set the "count" to -1 which tells FixPatch to use a bogus/fixed + # Set the "count" to -1 which tells fix_patch to use a bogus/fixed # time for generating the Message-Id. com = commit.Commit('') com.change_id = 'I80fe1d0c0b7dd10aa58ce5bb1d9290b6664d5413' com.count = -1 - patchstream.FixPatch(None, inname, series.Series(), com) + patchstream.fix_patch(None, inname, series.Series(), com) rc = os.system('diff -u %s %s' % (inname, expname)) self.assertEqual(rc, 0)
Signature · 1 line
2.29.0.rc2.309.g374f81d7ae-goog
PATCH v2 11/30 patman: Rename variables in patchstream
In reply to Simon GlassSome variables are too short or shadow other variables or types. Fix these to keep pylint3 happy.
---
(no changes since v1)
tools/patman/patchstream.py | 52 ++++++++++++++++++------------------- 1 file changed, 26 insertions(+), 26 deletions(-)
tools/patman/patchstream.py +26 -26
@@ -149,7 +149,7 @@ class PatchStream: """ try: return int(value) - except ValueError as str: + except ValueError: raise ValueError("%s: Cannot decode version info '%s'" % (self.commit.hash, line))
@@ -385,10 +385,10 @@ class PatchStream: # Well that means this is an ordinary line else: # Look for space before tab - m = RE_SPACE_BEFORE_TAB.match(line) - if m: + mat = RE_SPACE_BEFORE_TAB.match(line) + if mat: self.warn.append('Line %d/%d has space before tab' % - (self.linenum, m.start())) + (self.linenum, mat.start())) # OK, we have a valid non-blank line out = [line]
@@ -527,10 +527,10 @@ def get_metadata_for_list(commit_range, git_dir=None, count=None, params = gitutil.LogCmd(commit_range, reverse=True, count=count, git_dir=git_dir) stdout = command.RunPipe([params], capture=True).stdout - ps = PatchStream(series, is_log=True) + pst = PatchStream(series, is_log=True) for line in stdout.splitlines(): - ps.process_line(line) - ps.finalise() + pst.process_line(line) + pst.finalise() return series def get_metadata(branch, start, count):
@@ -554,13 +554,13 @@ def get_metadata_for_test(text): text: """ series = Series() - ps = PatchStream(series, is_log=True) + pst = PatchStream(series, is_log=True) for line in text.splitlines(): - ps.process_line(line) - ps.finalise() + pst.process_line(line) + pst.finalise() return series -def fix_patch(backup_dir, fname, series, commit): +def fix_patch(backup_dir, fname, series, cmt): """Fix up a patch file, by adding/removing as required. We remove our tags from the patch file, insert changes lists, etc.
@@ -571,16 +571,16 @@ def fix_patch(backup_dir, fname, series, commit): Args: fname: Filename to patch file to process series: Series information about this patch set - commit: Commit object for this patch file + cmt: Commit object for this patch file Return: A list of errors, or [] if all ok. """ handle, tmpname = tempfile.mkstemp() outfd = os.fdopen(handle, 'w', encoding='utf-8') infd = open(fname, 'r', encoding='utf-8') - ps = PatchStream(series) - ps.commit = commit - ps.process_stream(infd, outfd) + pst = PatchStream(series) + pst.commit = cmt + pst.process_stream(infd, outfd) infd.close() outfd.close()
@@ -588,7 +588,7 @@ def fix_patch(backup_dir, fname, series, commit): if backup_dir: shutil.copy(fname, os.path.join(backup_dir, os.path.basename(fname))) shutil.move(tmpname, fname) - return ps.warn + return pst.warn def fix_patches(series, fnames): """Fix up a list of patches identified by filenames
@@ -603,10 +603,10 @@ def fix_patches(series, fnames): backup_dir = None #tempfile.mkdtemp('clean-patch') count = 0 for fname in fnames: - commit = series.commits[count] - commit.patch = fname - commit.count = count - result = fix_patch(backup_dir, fname, series, commit) + cmt = series.commits[count] + cmt.patch = fname + cmt.count = count + result = fix_patch(backup_dir, fname, series, cmt) if result: print('%d warnings for %s:' % (len(result), fname)) for warn in result:
@@ -623,11 +623,11 @@ def insert_cover_letter(fname, series, count): series: Series object count: Number of patches in the series """ - fd = open(fname, 'r') - lines = fd.readlines() - fd.close() + fil = open(fname, 'r') + lines = fil.readlines() + fil.close() - fd = open(fname, 'w') + fil = open(fname, 'w') text = series.cover prefix = series.GetPatchPrefix() for line in lines:
@@ -647,5 +647,5 @@ def insert_cover_letter(fname, series, count): # Now the change list out = series.MakeChangeLog(None) line += '\n' + '\n'.join(out) - fd.write(line) - fd.close() + fil.write(line) + fil.close()
Signature · 1 line
2.29.0.rc2.309.g374f81d7ae-goog
PATCH v2 12/30 patman: Drop unused args in patchstream
In reply to Simon GlassDrop a few arguments that are not used in functions.
---
(no changes since v1)
tools/patman/patchstream.py | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)
tools/patman/patchstream.py +3 -5
@@ -64,7 +64,7 @@ class PatchStream: unwanted tags or inject additional ones. These correspond to the two phases of processing. """ - def __init__(self, series, name=None, is_log=False): + def __init__(self, series, is_log=False): self.skip_blank = False # True to skip a single blank line self.found_test = False # Found a TEST= line self.lines_after_test = 0 # Number of lines found after TEST=
@@ -100,15 +100,13 @@ class PatchStream: if self.is_log: self.series.AddTag(self.commit, line, name, value) - def _add_to_commit(self, line, name, value): + def _add_to_commit(self, name): """Add a new Commit-xxx tag. When a Commit-xxx tag is detected, we come here to record it. Args: - line: Source line containing tag (useful for debug/error messages) name: Tag name (part after 'Commit-') - value: Tag value (part after 'Commit-xxx: ') """ if name == 'notes': self.in_section = 'commit-' + name
@@ -349,7 +347,7 @@ class PatchStream: name = commit_tag_match.group(1) value = commit_tag_match.group(2) if name == 'notes': - self._add_to_commit(line, name, value) + self._add_to_commit(name) self.skip_blank = True elif name == 'changes': self.in_change = 'Commit'
Signature · 1 line
2.29.0.rc2.309.g374f81d7ae-goog
PATCH v2 13/30 patman: Fix up argument/return docs in patchstream
In reply to Simon GlassAdd missing documentation and type information. Fix up some missing docs on exceptions also.
---
(no changes since v1)
tools/patman/patchstream.py | 82 ++++++++++++++++++++++--------------- 1 file changed, 50 insertions(+), 32 deletions(-)
tools/patman/patchstream.py +50 -32
@@ -90,9 +90,10 @@ class PatchStream: are scanning a 'git log'. Args: - line: Source line containing tag (useful for debug/error messages) - name: Tag name (part after 'Series-') - value: Tag value (part after 'Series-xxx: ') + line (str): Source line containing tag (useful for debug/error + messages) + name (str): Tag name (part after 'Series-') + value (str): Tag value (part after 'Series-xxx: ') """ if name == 'notes': self.in_section = name
@@ -106,7 +107,7 @@ class PatchStream: When a Commit-xxx tag is detected, we come here to record it. Args: - name: Tag name (part after 'Commit-') + name (str): Tag name (part after 'Commit-') """ if name == 'notes': self.in_section = 'commit-' + name
@@ -116,8 +117,9 @@ class PatchStream: """Add a response tag to the current commit Args: - key: rtag type (e.g. 'Reviewed-by') - who: Person who gave that rtag, e.g. 'Fred Bloggs <[email protected]>' + rtag_type (str): rtag type (e.g. 'Reviewed-by') + who (str): Person who gave that rtag, e.g. + 'Fred Bloggs <[email protected]>' """ self.commit.AddRtag(rtag_type, who)
@@ -139,11 +141,14 @@ class PatchStream: """Parse a version from a *-changes tag Args: - value: Tag value (part after 'xxx-changes: ' - line: Source line containing tag + value (str): Tag value (part after 'xxx-changes: ' + line (str): Source line containing tag Returns: - The version as an integer + int: The version as an integer + + Raises: + ValueError: the value cannot be converted """ try: return int(value)
@@ -184,10 +189,14 @@ class PatchStream: don't want, and add things we think are required. Args: - line: text line to process + line (str): text line to process Returns: - list of output lines, or [] if nothing should be output + list: list of output lines, or [] if nothing should be output + + Raises: + ValueError: a fatal error occurred while parsing, e.g. an END + without a starting tag, or two commits with two change IDs """ # Initially we have no output. Prepare the input line string out = []
@@ -428,7 +437,7 @@ class PatchStream: and the prefix. Args: - outfd: Output stream file object + outfd (io.IOBase): Output stream file object """ if not self.commit.change_id: return
@@ -468,8 +477,8 @@ class PatchStream: This is used to process patch files one at a time. Args: - infd: Input stream file object - outfd: Output stream file object + infd (io.IOBase): Input stream file object + outfd (io.IOBase): Output stream file object """ # Extract the filename from each diff, for nice warnings fname = None
@@ -510,14 +519,15 @@ def get_metadata_for_list(commit_range, git_dir=None, count=None, are interested in. Args: - commit_range: Range of commits to count (e.g. 'HEAD..base') - git_dir: Path to git repositiory (None to use default) - count: Number of commits to list, or None for no limit - series: Series object to add information into. By default a new series + commit_range (str): Range of commits to count (e.g. 'HEAD..base') + git_dir (str): Path to git repositiory (None to use default) + count (int): Number of commits to list, or None for no limit + series (Series): Object to add information into. By default a new series is started. - allow_overwrite: Allow tags to overwrite an existing tag + allow_overwrite (bool): Allow tags to overwrite an existing tag + Returns: - A Series object containing information about the commits. + Series: Object containing information about the commits. """ if not series: series = Series()
@@ -538,9 +548,12 @@ def get_metadata(branch, start, count): are interested in. Args: - branch: Branch to use (None for current branch) - start: Commit to start from: 0=branch HEAD, 1=next one, etc. - count: Number of commits to list + branch (str): Branch to use (None for current branch) + start (int): Commit to start from: 0=branch HEAD, 1=next one, etc. + count (int): Number of commits to list + + Returns: + Series: Object containing information about the commits. """ return get_metadata_for_list( '%s~%d' % (branch if branch else 'HEAD', start), None, count)
@@ -550,6 +563,9 @@ def get_metadata_for_test(text): Args: text: + + Returns: + Series: Object containing information about the commits. """ series = Series() pst = PatchStream(series, is_log=True)
@@ -567,11 +583,13 @@ def fix_patch(backup_dir, fname, series, cmt): A backup file is put into backup_dir (if not None). Args: - fname: Filename to patch file to process - series: Series information about this patch set - cmt: Commit object for this patch file + backup_dir (str): Path to directory to use to backup the file + fname (str): Filename to patch file to process + series (Series): Series information about this patch set + cmt (Commit): Commit object for this patch file + Return: - A list of errors, or [] if all ok. + list: A list of errors, each str, or [] if all ok. """ handle, tmpname = tempfile.mkstemp() outfd = os.fdopen(handle, 'w', encoding='utf-8')
@@ -594,8 +612,8 @@ def fix_patches(series, fnames): The patch files are processed in place, and overwritten. Args: - series: The series object - fnames: List of patch files to process + series (Series): The Series object + fnames (:type: list of str): List of patch files to process """ # Current workflow creates patches, so we shouldn't need a backup backup_dir = None #tempfile.mkdtemp('clean-patch')
@@ -617,9 +635,9 @@ def insert_cover_letter(fname, series, count): """Inserts a cover letter with the required info into patch 0 Args: - fname: Input / output filename of the cover letter file - series: Series object - count: Number of patches in the series + fname (str): Input / output filename of the cover letter file + series (Series): Series object + count (int): Number of patches in the series """ fil = open(fname, 'r') lines = fil.readlines()
Signature · 1 line
2.29.0.rc2.309.g374f81d7ae-goog
PATCH v2 14/30 patman: Move warning collection to a function
In reply to Simon GlassAdd a new function in PatchStream to collect the warnings generated while parsing the stream. This will allow us to adjust the logic, such as dealing with per-commit warnings.
Two of the warnings are in fact internal errors, so change them to raise and exception.
---
(no changes since v1)
tools/patman/patchstream.py | 36 +++++++++++++++++++++++------------- 1 file changed, 23 insertions(+), 13 deletions(-)
tools/patman/patchstream.py +23 -13
@@ -83,6 +83,14 @@ class PatchStream: self.signoff = [] # Contents of signoff line self.commit = None # Current commit + def _add_warn(self, warn): + """Add a new warning to report to the user + + Args: + warn (str): Warning to report + """ + self.warn.append(warn) + def _add_to_series(self, line, name, value): """Add a new Series-xxx tag.
@@ -237,7 +245,7 @@ class PatchStream: # but we are already in a section, this means 'END' is missing # for that section, fix it up. if self.in_section: - self.warn.append("Missing 'END' in section '%s'" % self.in_section) + self._add_warn("Missing 'END' in section '%s'" % self.in_section) if self.in_section == 'cover': self.series.cover = self.section elif self.in_section == 'notes':
@@ -247,14 +255,16 @@ class PatchStream: if self.is_log: self.commit.notes += self.section else: - self.warn.append("Unknown section '%s'" % self.in_section) + # This should not happen + raise ValueError("Unknown section '%s'" % self.in_section) self.in_section = None self.skip_blank = True self.section = [] # but we are already in a change list, that means a blank line # is missing, fix it up. if self.in_change: - self.warn.append("Missing 'blank line' in section '%s-changes'" % self.in_change) + self._add_warn("Missing 'blank line' in section '%s-changes'" % + self.in_change) self._finalise_change() self.in_change = None self.change_version = 0
@@ -271,7 +281,8 @@ class PatchStream: if self.is_log: self.commit.notes += self.section else: - self.warn.append("Unknown section '%s'" % self.in_section) + # This should not happen + raise ValueError("Unknown section '%s'" % self.in_section) self.in_section = None self.skip_blank = True self.section = []
@@ -362,8 +373,8 @@ class PatchStream: self.in_change = 'Commit' self.change_version = self._parse_version(value, line) else: - self.warn.append('Line %d: Ignoring Commit-%s' % - (self.linenum, name)) + self._add_warn('Line %d: Ignoring Commit-%s' % + (self.linenum, name)) # Detect the start of a new commit elif commit_match:
@@ -377,7 +388,7 @@ class PatchStream: # Remove Tested-by self, since few will take much notice if (rtag_type == 'Tested-by' and who.find(os.getenv('USER') + '@') != -1): - self.warn.append("Ignoring %s" % line) + self._add_warn("Ignoring %s" % line) elif rtag_type == 'Patch-cc': self.commit.AddCc(who.split(',')) else:
@@ -394,8 +405,8 @@ class PatchStream: # Look for space before tab mat = RE_SPACE_BEFORE_TAB.match(line) if mat: - self.warn.append('Line %d/%d has space before tab' % - (self.linenum, mat.start())) + self._add_warn('Line %d/%d has space before tab' % + (self.linenum, mat.start())) # OK, we have a valid non-blank line out = [line]
@@ -427,8 +438,7 @@ class PatchStream: self._finalise_change() self._close_commit() if self.lines_after_test: - self.warn.append('Found %d lines after TEST=' % - self.lines_after_test) + self._add_warn('Found %d lines after TEST=' % self.lines_after_test) def _write_message_id(self, outfd): """Write the Message-Id into the output.
@@ -503,8 +513,8 @@ class PatchStream: self.blank_count += 1 else: if self.blank_count and (line == '-- ' or match): - self.warn.append("Found possible blank line(s) at " - "end of file '%s'" % last_fname) + self._add_warn("Found possible blank line(s) at end of file '%s'" % + last_fname) outfd.write('+\n' * self.blank_count) outfd.write(line + '\n') self.blank_count = 0
Signature · 1 line
2.29.0.rc2.309.g374f81d7ae-goog
PATCH v2 15/30 patman: Attach warnings to individual patches
In reply to Simon GlassAt present warnings are produced across the whole set of patches when parsing them. It is more useful to associate each warning with the patch (or commit) that generated it.
Attach warnings to the Commit object and move them out of PatchStream. Also avoid generating duplicate warnings for the same commit.
---
(no changes since v1)
tools/patman/commit.py | 2 ++ tools/patman/patchstream.py | 15 +++++++++++---- 2 files changed, 13 insertions(+), 4 deletions(-)
tools/patman/commit.py +2 -0
@@ -27,6 +27,7 @@ class Commit: rtags: Response tags (e.g. Reviewed-by) collected by the commit, dict: key: rtag type (e.g. 'Reviewed-by') value: Set of people who gave that rtag, each a name/email string + warn: List of warnings for this commit, each a str """ def __init__(self, hash): self.hash = hash
@@ -38,6 +39,7 @@ class Commit: self.notes = [] self.change_id = None self.rtags = collections.defaultdict(set) + self.warn = [] def AddChange(self, version, info): """Add a new change line to the change list for a version.
tools/patman/patchstream.py +11 -4
@@ -68,7 +68,6 @@ class PatchStream: self.skip_blank = False # True to skip a single blank line self.found_test = False # Found a TEST= line self.lines_after_test = 0 # Number of lines found after TEST= - self.warn = [] # List of warnings we have collected self.linenum = 1 # Output line number we are up to self.in_section = None # Name of start...END section we are in self.notes = [] # Series notes
@@ -84,12 +83,20 @@ class PatchStream: self.commit = None # Current commit def _add_warn(self, warn): - """Add a new warning to report to the user + """Add a new warning to report to the user about the current commit + + The new warning is added to the current commit if not already present. Args: warn (str): Warning to report + + Raises: + ValueError: Warning is generated with no commit associated """ - self.warn.append(warn) + if not self.commit: + raise ValueError('Warning outside commit: %s' % warn) + if warn not in self.commit.warn: + self.commit.warn.append(warn) def _add_to_series(self, line, name, value): """Add a new Series-xxx tag.
@@ -614,7 +621,7 @@ def fix_patch(backup_dir, fname, series, cmt): if backup_dir: shutil.copy(fname, os.path.join(backup_dir, os.path.basename(fname))) shutil.move(tmpname, fname) - return pst.warn + return cmt.warn def fix_patches(series, fnames): """Fix up a list of patches identified by filenames
Signature · 1 line
2.29.0.rc2.309.g374f81d7ae-goog
PATCH v2 16/30 patman: Convert 'Series-xxx' tag errors into warnings
In reply to Simon GlassIf the Series-xxx tag is not recognised patman currently reports a fatal error. This is inconvenient if a new feature is later added to patman that an earlier version does not support.
Report a warning instead, to allow the user to take action if needed, but still allow operation to proceed.
---
(no changes since v1)
tools/patman/patchstream.py | 4 +++- tools/patman/series.py | 6 +++++- 2 files changed, 8 insertions(+), 2 deletions(-)
tools/patman/patchstream.py +3 -1
@@ -114,7 +114,9 @@ class PatchStream: self.in_section = name self.skip_blank = False if self.is_log: - self.series.AddTag(self.commit, line, name, value) + warn = self.series.AddTag(self.commit, line, name, value) + if warn: + self.commit.warn.append(warn) def _add_to_commit(self, name): """Add a new Commit-xxx tag.
tools/patman/series.py +5 -1
@@ -59,6 +59,9 @@ class Series(dict): line: Source line containing tag (useful for debug/error messages) name: Tag name (part after 'Series-') value: Tag value (part after 'Series-xxx: ') + + Returns: + String warning if something went wrong, else None """ # If we already have it, then add to our list name = name.replace('-', '_')
@@ -78,9 +81,10 @@ class Series(dict): else: self[name] = value else: - raise ValueError("In %s: line '%s': Unknown 'Series-%s': valid " + return ("In %s: line '%s': Unknown 'Series-%s': valid " "options are %s" % (commit.hash, line, name, ', '.join(valid_series))) + return None def AddCommit(self, commit): """Add a commit into our list of commits
Signature · 1 line
2.29.0.rc2.309.g374f81d7ae-goog
PATCH v2 17/30 patman: Drop unused signoff member
In reply to Simon GlassThis is not used. Drop it.
---
(no changes since v1)
tools/patman/func_test.py | 1 + tools/patman/patchstream.py | 1 - 2 files changed, 1 insertion(+), 1 deletion(-)
tools/patman/func_test.py +1 -0
@@ -16,6 +16,7 @@ import unittest from patman import control from patman import gitutil from patman import patchstream +from patman.patchstream import PatchStream from patman import settings from patman import terminal from patman import tools
tools/patman/patchstream.py +0 -1
@@ -79,7 +79,6 @@ class PatchStream: self.change_lines = [] # Lines of the current change self.blank_count = 0 # Number of blank lines stored up self.state = STATE_MSG_HEADER # What state are we in? - self.signoff = [] # Contents of signoff line self.commit = None # Current commit def _add_warn(self, warn):
Signature · 1 line
2.29.0.rc2.309.g374f81d7ae-goog
PATCH v2 18/30 patman: Add a test for PatchStream tags
In reply to Simon GlassThe current functional tests run most of patman. Add a smaller test that just checks tag handling with the PatchStream class.
---
(no changes since v1)
tools/patman/func_test.py | 26 +++++++++++++++++++++----- tools/patman/patchstream.py | 23 +++++++++++++++++++++++ 2 files changed, 44 insertions(+), 5 deletions(-)
tools/patman/func_test.py +21 -5
@@ -31,6 +31,9 @@ except ModuleNotFoundError: class TestFunctional(unittest.TestCase): """Functional tests for checking that patman behaves correctly""" + leb = (b'Lord Edmund Blackadd\xc3\xabr <[email protected]>'. + decode('utf-8')) + def setUp(self): self.tmpdir = tempfile.mkdtemp(prefix='patman.') self.gitdir = os.path.join(self.tmpdir, 'git')
@@ -177,8 +180,6 @@ class TestFunctional(unittest.TestCase): stefan = b'Stefan Br\xc3\xbcns <[email protected]>'.decode('utf-8') rick = 'Richard III <[email protected]>' mel = b'Lord M\xc3\xablchett <[email protected]>'.decode('utf-8') - leb = (b'Lond Edmund Blackadd\xc3\xabr <weasel at blackadder.org'. - decode('utf-8')) fred = 'Fred Bloggs <[email protected]>' add_maintainers = [stefan, rick] dry_run = True
@@ -187,7 +188,7 @@ class TestFunctional(unittest.TestCase): settings.alias = { 'fdt': ['simon'], 'u-boot': ['u-boot at lists.denx.de'], - 'simon': [leb], + 'simon': [self.leb], 'fred': [fred], }
@@ -231,7 +232,7 @@ class TestFunctional(unittest.TestCase): self.assertEqual('Cover: 4 lines', lines[line + 4]) line += 5 self.assertEqual(' Cc: %s' % fred, lines[line + 0]) - self.assertEqual(' Cc: %s' % tools.FromUnicode(leb), + self.assertEqual(' Cc: %s' % tools.FromUnicode(self.leb), lines[line + 1]) self.assertEqual(' Cc: %s' % tools.FromUnicode(mel), lines[line + 2])
@@ -247,7 +248,7 @@ class TestFunctional(unittest.TestCase): self.assertEqual(('%s %s\0%s' % (args[0], rick, stefan)), tools.ToUnicode(cc_lines[0])) self.assertEqual( - '%s %s\0%s\0%s\0%s' % (args[1], fred, leb, rick, stefan), + '%s %s\0%s\0%s\0%s' % (args[1], fred, self.leb, rick, stefan), tools.ToUnicode(cc_lines[1])) expected = '''
@@ -480,3 +481,18 @@ complicated as possible''') self.assertEqual(2, len(patch_files)) finally: os.chdir(orig_dir) + + def testTags(self): + """Test collection of tags in a patchstream""" + text = '''This is a patch + +Signed-off-by: Terminator +Reviewed-by: Joe Bloggs <[email protected]> +Reviewed-by: Mary Bloggs <[email protected]> +Tested-by: %s +''' % self.leb + pstrm = PatchStream.process_text(text) + self.assertEqual(pstrm.commit.rtags, { + 'Reviewed-by': {'Mary Bloggs <[email protected]>', + 'Joe Bloggs <[email protected]>'}, + 'Tested-by': {self.leb}})
tools/patman/patchstream.py +23 -0
@@ -5,6 +5,7 @@ """Handles parsing a stream of commits/emails from 'git log' or other source""" import datetime +import io import math import os import re
@@ -81,6 +82,28 @@ class PatchStream: self.state = STATE_MSG_HEADER # What state are we in? self.commit = None # Current commit + @staticmethod + def process_text(text, is_comment=False): + """Process some text through this class using a default Commit/Series + + Args: + text (str): Text to parse + is_comment (bool): True if this is a comment rather than a patch. + If True, PatchStream doesn't expect a patch subject at the + start, but jumps straight into the body + + Returns: + PatchStream: object with results + """ + pstrm = PatchStream(Series()) + pstrm.commit = commit.Commit(None) + infd = io.StringIO(text) + outfd = io.StringIO() + if is_comment: + pstrm.state = STATE_PATCH_HEADER + pstrm.process_stream(infd, outfd) + return pstrm + def _add_warn(self, warn): """Add a new warning to report to the user about the current commit
Signature · 1 line
2.29.0.rc2.309.g374f81d7ae-goog
PATCH v2 19/30 patman: Add some tests for warnings
In reply to Simon GlassAdd tests that check that warnings are generated when expected.
---
(no changes since v1)
tools/patman/func_test.py | 111 +++++++++++++++++++++++++++++++++--- tools/patman/main.py | 16 +++++- tools/patman/patchstream.py | 2 +- 3 files changed, 116 insertions(+), 13 deletions(-)
tools/patman/func_test.py +102 -9
@@ -33,6 +33,9 @@ class TestFunctional(unittest.TestCase): """Functional tests for checking that patman behaves correctly""" leb = (b'Lord Edmund Blackadd\xc3\xabr <[email protected]>'. decode('utf-8')) + fred = 'Fred Bloggs <[email protected]>' + joe = 'Joe Bloggs <[email protected]>' + mary = 'Mary Bloggs <[email protected]>' def setUp(self): self.tmpdir = tempfile.mkdtemp(prefix='patman.')
@@ -180,7 +183,6 @@ class TestFunctional(unittest.TestCase): stefan = b'Stefan Br\xc3\xbcns <[email protected]>'.decode('utf-8') rick = 'Richard III <[email protected]>' mel = b'Lord M\xc3\xablchett <[email protected]>'.decode('utf-8') - fred = 'Fred Bloggs <[email protected]>' add_maintainers = [stefan, rick] dry_run = True in_reply_to = mel
@@ -189,7 +191,7 @@ class TestFunctional(unittest.TestCase): 'fdt': ['simon'], 'u-boot': ['u-boot at lists.denx.de'], 'simon': [self.leb], - 'fred': [fred], + 'fred': [self.fred], } text = self._get_text('test01.txt')
@@ -231,7 +233,7 @@ class TestFunctional(unittest.TestCase): self.assertEqual('Prefix:\t RFC', lines[line + 3]) self.assertEqual('Cover: 4 lines', lines[line + 4]) line += 5 - self.assertEqual(' Cc: %s' % fred, lines[line + 0]) + self.assertEqual(' Cc: %s' % self.fred, lines[line + 0]) self.assertEqual(' Cc: %s' % tools.FromUnicode(self.leb), lines[line + 1]) self.assertEqual(' Cc: %s' % tools.FromUnicode(mel),
@@ -248,7 +250,7 @@ class TestFunctional(unittest.TestCase): self.assertEqual(('%s %s\0%s' % (args[0], rick, stefan)), tools.ToUnicode(cc_lines[0])) self.assertEqual( - '%s %s\0%s\0%s\0%s' % (args[1], fred, self.leb, rick, stefan), + '%s %s\0%s\0%s\0%s' % (args[1], self.fred, self.leb, rick, stefan), tools.ToUnicode(cc_lines[1])) expected = '''
@@ -487,12 +489,103 @@ complicated as possible''') text = '''This is a patch Signed-off-by: Terminator -Reviewed-by: Joe Bloggs <[email protected]> -Reviewed-by: Mary Bloggs <[email protected]> +Reviewed-by: %s +Reviewed-by: %s Tested-by: %s -''' % self.leb +''' % (self.joe, self.mary, self.leb) pstrm = PatchStream.process_text(text) self.assertEqual(pstrm.commit.rtags, { - 'Reviewed-by': {'Mary Bloggs <[email protected]>', - 'Joe Bloggs <[email protected]>'}, + 'Reviewed-by': {self.joe, self.mary}, 'Tested-by': {self.leb}}) + + def testMissingEnd(self): + """Test a missing END tag""" + text = '''This is a patch + +Cover-letter: +This is the title +missing END after this line +Signed-off-by: Fred +''' + pstrm = PatchStream.process_text(text) + self.assertEqual(["Missing 'END' in section 'cover'"], + pstrm.commit.warn) + + def testMissingBlankLine(self): + """Test a missing blank line after a tag""" + text = '''This is a patch + +Series-changes: 2 +- First line of changes +- Missing blank line after this line +Signed-off-by: Fred +''' + pstrm = PatchStream.process_text(text) + self.assertEqual(["Missing 'blank line' in section 'Series-changes'"], + pstrm.commit.warn) + + def testInvalidCommitTag(self): + """Test an invalid Commit-xxx tag""" + text = '''This is a patch + +Commit-fred: testing +''' + pstrm = PatchStream.process_text(text) + self.assertEqual(["Line 3: Ignoring Commit-fred"], pstrm.commit.warn) + + def testSelfTest(self): + """Test a tested by tag by this user""" + test_line = 'Tested-by: %s at napier.com' % os.getenv('USER') + text = '''This is a patch + +%s +''' % test_line + pstrm = PatchStream.process_text(text) + self.assertEqual(["Ignoring '%s'" % test_line], pstrm.commit.warn) + + def testSpaceBeforeTab(self): + """Test a space before a tab""" + text = '''This is a patch + ++ \tSomething +''' + pstrm = PatchStream.process_text(text) + self.assertEqual(["Line 3/0 has space before tab"], pstrm.commit.warn) + + def testLinesAfterTest(self): + """Test detecting lines after TEST= line""" + text = '''This is a patch + +TEST=sometest +more lines +here +''' + pstrm = PatchStream.process_text(text) + self.assertEqual(["Found 2 lines after TEST="], pstrm.commit.warn) + + def testBlankLineAtEnd(self): + """Test detecting a blank line at the end of a file""" + text = '''This is a patch + +diff --git a/lib/fdtdec.c b/lib/fdtdec.c +index c072e54..942244f 100644 +--- a/lib/fdtdec.c ++++ b/lib/fdtdec.c +@@ -1200,7 +1200,8 @@ int fdtdec_setup_mem_size_base(void) + } + + gd->ram_size = (phys_size_t)(res.end - res.start + 1); +- debug("%s: Initial DRAM size %llx\n", __func__, (u64)gd->ram_size); ++ debug("%s: Initial DRAM size %llx\n", __func__, ++ (unsigned long long)gd->ram_size); ++ +diff --git a/lib/efi_loader/efi_memory.c b/lib/efi_loader/efi_memory.c + +-- +2.7.4 + + ''' + pstrm = PatchStream.process_text(text) + self.assertEqual( + ["Found possible blank line(s) at end of file 'lib/fdtdec.c'"], + pstrm.commit.warn)
tools/patman/main.py +13 -3
@@ -86,6 +86,8 @@ AddCommonArgs(send) send.add_argument('patchfiles', nargs='*') test_parser = subparsers.add_parser('test', help='Run tests') +test_parser.add_argument('testname', type=str, default=None, nargs='?', + help="Specify the test to run") AddCommonArgs(test_parser) # Parse options twice: first to get the project and second to handle
@@ -111,15 +113,23 @@ if args.cmd == 'test': sys.argv = [sys.argv[0]] result = unittest.TestResult() + suite = unittest.TestSuite() + loader = unittest.TestLoader() for module in (test_checkpatch.TestPatch, func_test.TestFunctional): - suite = unittest.TestLoader().loadTestsFromTestCase(module) - suite.run(result) + if args.testname: + try: + suite.addTests(loader.loadTestsFromName(args.testname, module)) + except AttributeError: + continue + else: + suite.addTests(loader.loadTestsFromTestCase(module)) + suite.run(result) for module in ['gitutil', 'settings', 'terminal']: suite = doctest.DocTestSuite(module) suite.run(result) - sys.exit(test_util.ReportResult('patman', None, result)) + sys.exit(test_util.ReportResult('patman', args.testname, result)) # Process commits, produce patches files, check them, email them elif args.cmd == 'send':
tools/patman/patchstream.py +1 -1
@@ -419,7 +419,7 @@ class PatchStream: # Remove Tested-by self, since few will take much notice if (rtag_type == 'Tested-by' and who.find(os.getenv('USER') + '@') != -1): - self._add_warn("Ignoring %s" % line) + self._add_warn("Ignoring '%s'" % line) elif rtag_type == 'Patch-cc': self.commit.AddCc(who.split(',')) else:
Signature · 1 line
2.29.0.rc2.309.g374f81d7ae-goog
PATCH v2 20/30 patman: Convert testBasic() to use an interator
In reply to Simon GlassOn balance it is easier to use an iterator here, particularly if we need to insert lines due to new functionality. The only niggle is the need to keep the previous iterator value around in one case.
Convert this test to use iter().
---
(no changes since v1)
tools/patman/func_test.py | 55 +++++++++++++++++++-------------------- 1 file changed, 27 insertions(+), 28 deletions(-)
tools/patman/func_test.py +27 -28
@@ -212,40 +212,39 @@ class TestFunctional(unittest.TestCase): cc_lines = open(cc_file, encoding='utf-8').read().splitlines() os.remove(cc_file) - lines = out[0].getvalue().splitlines() - self.assertEqual('Cleaned %s patches' % len(series.commits), lines[0]) - self.assertEqual('Change log missing for v2', lines[1]) - self.assertEqual('Change log missing for v3', lines[2]) - self.assertEqual('Change log for unknown version v4', lines[3]) - self.assertEqual("Alias 'pci' not found", lines[4]) - self.assertIn('Dry run', lines[5]) - self.assertIn('Send a total of %d patches' % count, lines[7]) - line = 8 - for i in range(len(series.commits)): - self.assertEqual(' %s' % args[i], lines[line + 0]) - line += 1 - while 'Cc:' in lines[line]: - line += 1 - self.assertEqual('To: u-boot at lists.denx.de', lines[line]) - self.assertEqual('Cc: %s' % tools.FromUnicode(stefan), - lines[line + 1]) - self.assertEqual('Version: 3', lines[line + 2]) - self.assertEqual('Prefix:\t RFC', lines[line + 3]) - self.assertEqual('Cover: 4 lines', lines[line + 4]) - line += 5 - self.assertEqual(' Cc: %s' % self.fred, lines[line + 0]) + lines = iter(out[0].getvalue().splitlines()) + self.assertEqual('Cleaned %s patches' % len(series.commits), + next(lines)) + self.assertEqual('Change log missing for v2', next(lines)) + self.assertEqual('Change log missing for v3', next(lines)) + self.assertEqual('Change log for unknown version v4', next(lines)) + self.assertEqual("Alias 'pci' not found", next(lines)) + self.assertIn('Dry run', next(lines)) + self.assertEqual('', next(lines)) + self.assertIn('Send a total of %d patches' % count, next(lines)) + prev = next(lines) + for i, commit in enumerate(series.commits): + self.assertEqual(' %s' % args[i], prev) + while True: + prev = next(lines) + if 'Cc:' not in prev: + break + self.assertEqual('To: u-boot at lists.denx.de', prev) + self.assertEqual('Cc: %s' % tools.FromUnicode(stefan), next(lines)) + self.assertEqual('Version: 3', next(lines)) + self.assertEqual('Prefix:\t RFC', next(lines)) + self.assertEqual('Cover: 4 lines', next(lines)) + self.assertEqual(' Cc: %s' % self.fred, next(lines)) self.assertEqual(' Cc: %s' % tools.FromUnicode(self.leb), - lines[line + 1]) - self.assertEqual(' Cc: %s' % tools.FromUnicode(mel), - lines[line + 2]) - self.assertEqual(' Cc: %s' % rick, lines[line + 3]) + next(lines)) + self.assertEqual(' Cc: %s' % tools.FromUnicode(mel), next(lines)) + self.assertEqual(' Cc: %s' % rick, next(lines)) expected = ('Git command: git send-email --annotate ' '--in-reply-to="%s" --to "u-boot at lists.denx.de" ' '--cc "%s" --cc-cmd "%s --cc-cmd %s" %s %s' % (in_reply_to, stefan, sys.argv[0], cc_file, cover_fname, ' '.join(args))) - line += 4 - self.assertEqual(expected, tools.ToUnicode(lines[line])) + self.assertEqual(expected, tools.ToUnicode(next(lines))) self.assertEqual(('%s %s\0%s' % (args[0], rick, stefan)), tools.ToUnicode(cc_lines[0]))
Signature · 1 line
2.29.0.rc2.309.g374f81d7ae-goog
PATCH v2 21/30 patman: Require tags to be before sign-off
In reply to Simon GlassAt present it is possible to put sign-off tags before the change log. This works but then it is hard for patman to add its own tags to a commit. Also if the commit has a Change-Id (e.g. for Gerrit) the commit becomes invalid if there is anything after it.
Report a warning if patman tags are in the wrong place.
One test patch already has the sign-off in the wrong place. Update the second one to avoid warnings.
---
(no changes since v1)
tools/patman/README | 3 ++- tools/patman/func_test.py | 23 ++++++++++++++++++- tools/patman/patchstream.py | 22 ++++++++++++++++++ .../0001-pci-Correct-cast-for-sandbox.patch | 2 +- ...-for-sandbox-in-fdtdec_setup_mem_siz.patch | 2 +- tools/patman/test/test01.txt | 4 ++-- 6 files changed, 50 insertions(+), 6 deletions(-)
tools/patman/README +2 -1
@@ -161,7 +161,8 @@ How to add tags =============== To make this script useful you must add tags like the following into any -commit. Most can only appear once in the whole series. +commit. Most can only appear once in the whole series. Tags should be placed +before the sign-off line / Change-Id. Series-to: email / alias Email address / alias to send patch series to (you can add this
tools/patman/func_test.py +22 -1
@@ -194,6 +194,9 @@ class TestFunctional(unittest.TestCase): 'fred': [self.fred], } + # NOTE: If you change this file you must also change the patch files in + # the same directory, since we assume that the metadata file matches the + # patched. If it doesn't, this test will fail. text = self._get_text('test01.txt') series = patchstream.get_metadata_for_test(text) cover_fname, args = self._create_patches_for_test(series)
@@ -213,6 +216,10 @@ class TestFunctional(unittest.TestCase): os.remove(cc_file) lines = iter(out[0].getvalue().splitlines()) + self.assertIn('1 warnings for', next(lines)) + self.assertEqual( + "\t Tag 'Commit-notes' should be before sign-off / Change-Id", + next(lines)) self.assertEqual('Cleaned %s patches' % len(series.commits), next(lines)) self.assertEqual('Change log missing for v2', next(lines))
@@ -223,7 +230,7 @@ class TestFunctional(unittest.TestCase): self.assertEqual('', next(lines)) self.assertIn('Send a total of %d patches' % count, next(lines)) prev = next(lines) - for i, commit in enumerate(series.commits): + for i in range(len(series.commits)): self.assertEqual(' %s' % args[i], prev) while True: prev = next(lines)
@@ -588,3 +595,17 @@ diff --git a/lib/efi_loader/efi_memory.c b/lib/efi_loader/efi_memory.c self.assertEqual( ["Found possible blank line(s) at end of file 'lib/fdtdec.c'"], pstrm.commit.warn) + + def testTagsAfterSignoff(self): + """Test detection of tags after the signoff""" + text = '''This is a patch + +Signed-off-by: Terminator 2 +Series-changes: 2 +- A change + +''' + pstrm = PatchStream.process_text(text) + self.assertEqual( + ["Tag 'Series-changes' should be before sign-off / Change-Id"], + pstrm.commit.warn)
tools/patman/patchstream.py +22 -0
@@ -81,6 +81,8 @@ class PatchStream: self.blank_count = 0 # Number of blank lines stored up self.state = STATE_MSG_HEADER # What state are we in? self.commit = None # Current commit + self.saw_signoff = False # Found sign-off line for this commit + self.saw_change_id = False # Found Change-Id for this commit @staticmethod def process_text(text, is_comment=False):
@@ -176,6 +178,9 @@ class PatchStream: self.skip_blank = True self.section = [] + self.saw_signoff = False + self.saw_change_id = False + def _parse_version(self, value, line): """Parse a version from a *-changes tag
@@ -343,6 +348,10 @@ class PatchStream: elif cover_match: name = cover_match.group(1) value = cover_match.group(2) + if self.saw_signoff or self.saw_change_id: + self._add_warn( + "Tag 'Cover-%s' should be before sign-off / Change-Id" % + name) if name == 'letter': self.in_section = 'cover' self.skip_blank = False
@@ -374,6 +383,10 @@ class PatchStream: elif series_tag_match: name = series_tag_match.group(1) value = series_tag_match.group(2) + if self.saw_signoff or self.saw_change_id: + self._add_warn( + "Tag 'Series-%s' should be before sign-off / Change-Id" % + name) if name == 'changes': # value is the version number: e.g. 1, or 2 self.in_change = 'Series'
@@ -385,6 +398,7 @@ class PatchStream: # Detect Change-Id tags elif change_id_match: value = change_id_match.group(1) + self.saw_change_id = True if self.is_log: if self.commit.change_id: raise ValueError(
@@ -397,6 +411,10 @@ class PatchStream: elif commit_tag_match: name = commit_tag_match.group(1) value = commit_tag_match.group(2) + if self.saw_signoff or self.saw_change_id: + self._add_warn( + "Tag 'Commit-%s' should be before sign-off / Change-Id" % + name) if name == 'notes': self._add_to_commit(name) self.skip_blank = True
@@ -427,6 +445,7 @@ class PatchStream: # Suppress duplicate signoffs elif signoff_match: + self.saw_signoff = True if (self.is_log or not self.commit or self.commit.CheckDuplicateSignoff(signoff_match.group(1))): out = [line]
@@ -527,6 +546,8 @@ class PatchStream: re_fname = re.compile('diff --git a/(.*) b/.*') self._write_message_id(outfd) + self.saw_signoff = False + self.saw_change_id = False while True: line = infd.readline()
@@ -637,6 +658,7 @@ def fix_patch(backup_dir, fname, series, cmt): infd = open(fname, 'r', encoding='utf-8') pst = PatchStream(series) pst.commit = cmt + cmt.warn = [] pst.process_stream(infd, outfd) infd.close() outfd.close()
tools/patman/test/0001-pci-Correct-cast-for-sandbox.patch +1 -1
@@ -14,7 +14,6 @@ cmd/pci.c:152:11: warning: format ?%llx? expects argument of type Fix it with a cast. -Signed-off-by: Simon Glass <[email protected]> Commit-changes: 2 - Changes only for this commit
@@ -24,6 +23,7 @@ about some things from the first commit END +Signed-off-by: Simon Glass <[email protected]> Commit-notes: Some notes about the first commit
tools/patman/test/0002-fdt-Correct-cast-for-sandbox-in-fdtdec_setup_mem_siz.patch +1 -1
@@ -14,7 +14,6 @@ lib/fdtdec.c:1203:8: warning: format ?%llx? expects argument of type Fix it with a cast. -Signed-off-by: Simon Glass <[email protected]> Series-to: u-boot Series-prefix: RFC Series-cc: Stefan Br?ns <[email protected]>
@@ -40,6 +39,7 @@ This is a test of how the cover letter works END +Signed-off-by: Simon Glass <[email protected]> --- fs/fat/fat.c | 1 + lib/efi_loader/efi_memory.c | 1 +
tools/patman/test/test01.txt +2 -2
@@ -12,7 +12,6 @@ Date: Sat Apr 15 15:39:08 2017 -0600 Fix it with a cast. - Signed-off-by: Simon Glass <[email protected]> Commit-changes: 2 - second revision change
@@ -22,6 +21,7 @@ Date: Sat Apr 15 15:39:08 2017 -0600 from the first commit END + Signed-off-by: Simon Glass <[email protected]> Commit-notes: Some notes about the first commit
@@ -41,7 +41,6 @@ Date: Sat Apr 15 15:39:08 2017 -0600 Fix it with a cast. - Signed-off-by: Simon Glass <[email protected]> Series-to: u-boot Series-prefix: RFC Series-cc: Stefan Br?ns <[email protected]>
@@ -67,3 +66,4 @@ Date: Sat Apr 15 15:39:08 2017 -0600 letter works END + Signed-off-by: Simon Glass <[email protected]>
Signature · 1 line
2.29.0.rc2.309.g374f81d7ae-goog
PATCH v2 21/30 patman: Require tags to be before sign-off
Quoted 4 lines · Simon Glass
At present it is possible to put sign-off tags before the change log. This works but then it is hard for patman to add its own tags to a commit. Also if the commit has a Change-Id (e.g. for Gerrit) the commit becomes invalid if there is anything after it.
NAK. I put SOBs before changelogs in all my series. Doing it like this makes the pre-patman commit more-closely match what is sent to the mailing list, and what will eventually be committed. What is most important is the summary, the description, who has signed off, and lastly any changelogs which will be removed by git am.
I also like to put SOBs on top because it groups all the commit-specific information before any patman-specific tags. The last commit (the HEAD) usually has several tags (Series-to/cc, Series-process-log, Cover-letter, etc.) which are unrelated to the commit itself.
The tool should do what is best for us humans, not what is convenient for the tool.
--Sean
Quoted 248 lines
Report a warning if patman tags are in the wrong place.
One test patch already has the sign-off in the wrong place. Update the second one to avoid warnings.
Signed-off-by Simon Glass <[email protected]>---
(no changes since v1)
tools/patman/README | 3 ++- tools/patman/func_test.py | 23 ++++++++++++++++++- tools/patman/patchstream.py | 22 ++++++++++++++++++ .../0001-pci-Correct-cast-for-sandbox.patch | 2 +- ...-for-sandbox-in-fdtdec_setup_mem_siz.patch | 2 +- tools/patman/test/test01.txt | 4 ++-- 6 files changed, 50 insertions(+), 6 deletions(-)tools/patman/README +2
-1@@ -161,7 +161,8 @@ How to add tags =============== To make this script useful you must add tags like the following into any -commit. Most can only appear once in the whole series. +commit. Most can only appear once in the whole series. Tags should be placed +before the sign-off line / Change-Id. Series-to: email / alias Email address / alias to send patch series to (you can add thistools/patman/func_test.py +22
-1@@ -194,6 +194,9 @@ class TestFunctional(unittest.TestCase): 'fred': [self.fred], } + # NOTE: If you change this file you must also change the patch files in + # the same directory, since we assume that the metadata file matches the + # patched. If it doesn't, this test will fail. text = self._get_text('test01.txt') series = patchstream.get_metadata_for_test(text) cover_fname, args = self._create_patches_for_test(series)@@ -213,6 +216,10 @@ class TestFunctional(unittest.TestCase): os.remove(cc_file) lines = iter(out[0].getvalue().splitlines()) + self.assertIn('1 warnings for', next(lines)) + self.assertEqual( + "\t Tag 'Commit-notes' should be before sign-off / Change-Id", + next(lines)) self.assertEqual('Cleaned %s patches' % len(series.commits), next(lines)) self.assertEqual('Change log missing for v2', next(lines))@@ -223,7 +230,7 @@ class TestFunctional(unittest.TestCase): self.assertEqual('', next(lines)) self.assertIn('Send a total of %d patches' % count, next(lines)) prev = next(lines) - for i, commit in enumerate(series.commits): + for i in range(len(series.commits)): self.assertEqual(' %s' % args[i], prev) while True: prev = next(lines)@@ -588,3 +595,17 @@ diff --git a/lib/efi_loader/efi_memory.c b/lib/efi_loader/efi_memory.c self.assertEqual( ["Found possible blank line(s) at end of file 'lib/fdtdec.c'"], pstrm.commit.warn) + + def testTagsAfterSignoff(self): + """Test detection of tags after the signoff""" + text = '''This is a patch + +Signed-off-by: Terminator 2 +Series-changes: 2 +- A change + +''' + pstrm = PatchStream.process_text(text) + self.assertEqual( + ["Tag 'Series-changes' should be before sign-off / Change-Id"], + pstrm.commit.warn)tools/patman/patchstream.py +22
-0@@ -81,6 +81,8 @@ class PatchStream: self.blank_count = 0 # Number of blank lines stored up self.state = STATE_MSG_HEADER # What state are we in? self.commit = None # Current commit + self.saw_signoff = False # Found sign-off line for this commit + self.saw_change_id = False # Found Change-Id for this commit @staticmethod def process_text(text, is_comment=False):@@ -176,6 +178,9 @@ class PatchStream: self.skip_blank = True self.section = [] + self.saw_signoff = False + self.saw_change_id = False + def _parse_version(self, value, line): """Parse a version from a *-changes tag@@ -343,6 +348,10 @@ class PatchStream: elif cover_match: name = cover_match.group(1) value = cover_match.group(2) + if self.saw_signoff or self.saw_change_id: + self._add_warn( + "Tag 'Cover-%s' should be before sign-off / Change-Id" % + name) if name == 'letter': self.in_section = 'cover' self.skip_blank = False@@ -374,6 +383,10 @@ class PatchStream: elif series_tag_match: name = series_tag_match.group(1) value = series_tag_match.group(2) + if self.saw_signoff or self.saw_change_id: + self._add_warn( + "Tag 'Series-%s' should be before sign-off / Change-Id" % + name) if name == 'changes': # value is the version number: e.g. 1, or 2 self.in_change = 'Series'@@ -385,6 +398,7 @@ class PatchStream: # Detect Change-Id tags elif change_id_match: value = change_id_match.group(1) + self.saw_change_id = True if self.is_log: if self.commit.change_id: raise ValueError(@@ -397,6 +411,10 @@ class PatchStream: elif commit_tag_match: name = commit_tag_match.group(1) value = commit_tag_match.group(2) + if self.saw_signoff or self.saw_change_id: + self._add_warn( + "Tag 'Commit-%s' should be before sign-off / Change-Id" % + name) if name == 'notes': self._add_to_commit(name) self.skip_blank = True@@ -427,6 +445,7 @@ class PatchStream: # Suppress duplicate signoffs elif signoff_match: + self.saw_signoff = True if (self.is_log or not self.commit or self.commit.CheckDuplicateSignoff(signoff_match.group(1))): out = [line]@@ -527,6 +546,8 @@ class PatchStream: re_fname = re.compile('diff --git a/(.*) b/.*') self._write_message_id(outfd) + self.saw_signoff = False + self.saw_change_id = False while True: line = infd.readline()@@ -637,6 +658,7 @@ def fix_patch(backup_dir, fname, series, cmt): infd = open(fname, 'r', encoding='utf-8') pst = PatchStream(series) pst.commit = cmt + cmt.warn = [] pst.process_stream(infd, outfd) infd.close() outfd.close()tools/patman/test/0001-pci-Correct-cast-for-sandbox.patch +1
-1@@ -14,7 +14,6 @@ cmd/pci.c:152:11: warning: format ?%llx? expects argument of type Fix it with a cast. -Signed-off-by: Simon Glass <[email protected]> Commit-changes: 2 - Changes only for this commit@@ -24,6 +23,7 @@ about some things from the first commit END +Signed-off-by: Simon Glass <[email protected]> Commit-notes: Some notes about the first committools/patman/test/0002-fdt-Correct-cast-for-sandbox-in-fdtdec_setup_mem_siz.patch +1
-1@@ -14,7 +14,6 @@ lib/fdtdec.c:1203:8: warning: format ?%llx? expects argument of type Fix it with a cast. -Signed-off-by: Simon Glass <[email protected]> Series-to: u-boot Series-prefix: RFC Series-cc: Stefan Br?ns <[email protected]>@@ -40,6 +39,7 @@ This is a test of how the cover letter works END +Signed-off-by: Simon Glass <[email protected]> --- fs/fat/fat.c | 1 + lib/efi_loader/efi_memory.c | 1 +tools/patman/test/test01.txt +2
-2@@ -12,7 +12,6 @@ Date: Sat Apr 15 15:39:08 2017 -0600 Fix it with a cast. - Signed-off-by: Simon Glass <[email protected]> Commit-changes: 2 - second revision change@@ -22,6 +21,7 @@ Date: Sat Apr 15 15:39:08 2017 -0600 from the first commit END + Signed-off-by: Simon Glass <[email protected]> Commit-notes: Some notes about the first commit@@ -41,7 +41,6 @@ Date: Sat Apr 15 15:39:08 2017 -0600 Fix it with a cast. - Signed-off-by: Simon Glass <[email protected]> Series-to: u-boot Series-prefix: RFC Series-cc: Stefan Br?ns <[email protected]>@@ -67,3 +66,4 @@ Date: Sat Apr 15 15:39:08 2017 -0600 letter works END + Signed-off-by: Simon Glass <[email protected]>
PATCH v2 21/30 patman: Require tags to be before sign-off
Hi Sean,
Quoted 13 lines · Sean Anderson
Quoted 4 lines · Simon Glass
At present it is possible to put sign-off tags before the change log. This works but then it is hard for patman to add its own tags to a commit. Also if the commit has a Change-Id (e.g. for Gerrit) the commit becomes invalid if there is anything after it.
NAK. I put SOBs before changelogs in all my series. Doing it like this makes the pre-patman commit more-closely match what is sent to the mailing list, and what will eventually be committed. What is most important is the summary, the description, who has signed off, and lastly any changelogs which will be removed by git am.
Quoted 7 lines
I also like to put SOBs on top because it groups all the commit-specific information before any patman-specific tags. The last commit (the HEAD) usually has several tags (Series-to/cc, Series-process-log, Cover-letter, etc.) which are unrelated to the commit itself.
The tool should do what is best for us humans, not what is convenient for the tool.
I suppose it can be done. But we have the same problem with Change-Id. Do you use gerit?
Regards,
SImon
PATCH v2 21/30 patman: Require tags to be before sign-off
Quoted 7 lines · Simon Glass
Quoted 5 lines · Sean Anderson
I also like to put SOBs on top because it groups all the commit-specific information before any patman-specific tags. The last commit (the HEAD) usually has several tags (Series-to/cc, Series-process-log, Cover-letter, etc.) which are unrelated to the commit itself.
Quoted 5 lines
Quoted 2 lines
The tool should do what is best for us humans, not what is convenient for the tool.
I suppose it can be done. But we have the same problem with Change-Id. Do you use gerit?
I assume this is mostly about Commit-Notes, when used without following the rfc-style format that has subsequent lines indented.
If so, then this issue isn't specific to Gerrit: it applies when using Git directly (the "git interpret-trailers" command, "git commit --amend --signoff", and so on).
As long as you're following rfc822 syntax, all is fine, so in some sense the issue here is the Commit-Notes tag. Should it go in the commit message body instead of the trailer paragraph?
Thanks,
Jonathan
PATCH v2 21/30 patman: Require tags to be before sign-off
Quoted 8 lines · Jonathan Nieder
Quoted 7 lines · Simon Glass
Quoted 5 lines · Sean Anderson
I also like to put SOBs on top because it groups all the commit-specific information before any patman-specific tags. The last commit (the HEAD) usually has several tags (Series-to/cc, Series-process-log, Cover-letter, etc.) which are unrelated to the commit itself.
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
The tool should do what is best for us humans, not what is convenient for the tool.
I suppose it can be done. But we have the same problem with Change-Id. Do you use gerit?
I have occasionally used gerrit, but not on any projects where I also would like to use patman. I don't care about the Change-Id requirement.
Using the following commit as an example,
Quoted 6 lines
doc: Update logging documentation
This updates logging documentation with some examples of the new commands added in the previous commits. It also removes some items from the to-do list which have been implemented.
to confirm, you would like to force the next two lines to be placed last?
Quoted 2 lines
Signed-off-by Sean Anderson <[email protected]>Reviewed-by Simon Glass <[email protected]>
I like them in their current location, because everything above this line will be present in the final commit as it is applied.
Quoted 10 lines
Series-changes: 2
- Clarify wording of filter documentation
- Reorganize log documentation; related sections should now be more proximate
- Include enum definitions instead of re-documenting them
- Add a few informational commandsSeries-changes: 3
- Fix heading level of Filters section
- Remove a few more already-implemented features from the TODO list
Everything above this line will be present in the patch as it is emailed.
Quoted 15 lines
Series-version: 3
Series-process-log: sort
Series-to: u-boot
Series-cc: sjg
Series-cc: trini
Series-cc: xypron
Cover-letter:
log: Add commands for manipulating filters
This series adds several commands for adding, listing, and removing log filters. It also adds getopt, since the filter-add command needs to have several optional arguments to be complete, and positional specification of those arguments would have been difficult.
END
Quoted 2 lines
I assume this is mostly about Commit-Notes, when used without following the rfc-style format that has subsequent lines indented.
Can you elaborate a bit on what you're referring to?
Quoted 11 lines
If so, then this issue isn't specific to Gerrit: it applies when using Git directly (the "git interpret-trailers" command, "git commit --amend --signoff", and so on).
As long as you're following rfc822 syntax, all is fine, so in some sense the issue here is the Commit-Notes tag. Should it go in the commit message body instead of the trailer paragraph?
Thanks,
Jonathan
--Sean
PATCH v2 22/30 patman: Fix spelling of plural for warning
In reply to Simon GlassTidy up the extra 's' when there is only a single warning. Fix the empty print statement also.
---
(no changes since v1)
tools/patman/func_test.py | 5 +++-- tools/patman/patchstream.py | 9 +++++---- 2 files changed, 8 insertions(+), 6 deletions(-)
tools/patman/func_test.py +3 -2
@@ -216,10 +216,11 @@ class TestFunctional(unittest.TestCase): os.remove(cc_file) lines = iter(out[0].getvalue().splitlines()) - self.assertIn('1 warnings for', next(lines)) + self.assertIn('1 warning for', next(lines)) self.assertEqual( - "\t Tag 'Commit-notes' should be before sign-off / Change-Id", + "\tTag 'Commit-notes' should be before sign-off / Change-Id", next(lines)) + self.assertEqual('', next(lines)) self.assertEqual('Cleaned %s patches' % len(series.commits), next(lines)) self.assertEqual('Change log missing for v2', next(lines))
tools/patman/patchstream.py +5 -4
@@ -687,12 +687,13 @@ def fix_patches(series, fnames): cmt.count = count result = fix_patch(backup_dir, fname, series, cmt) if result: - print('%d warnings for %s:' % (len(result), fname)) + print('%d warning%s for %s:' % + (len(result), 's' if len(result) > 1 else '', fname)) for warn in result: - print('\t', warn) - print + print('\t%s' % warn) + print() count += 1 - print('Cleaned %d patches' % count) + print('Cleaned %d patche%s' % (count, 's' if count > 1 else '')) def insert_cover_letter(fname, series, count): """Inserts a cover letter with the required info into patch 0
Signature · 1 line
2.29.0.rc2.309.g374f81d7ae-goog
PATCH v2 23/30 patman: Don't ignore lines starting with hash
In reply to Simon GlassThese lines can indicate a continuation of an error and should not be ignored. Fix this.
---
(no changes since v1)
tools/patman/checkpatch.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
tools/patman/checkpatch.py +5 -2
@@ -95,6 +95,7 @@ def CheckPatch(fname, verbose=False, show_types=False): re_check = re.compile('CHECK:%s (.*)' % type_name) re_file = re.compile('#\d+: FILE: ([^:]*):(\d+):') re_note = re.compile('NOTE: (.*)') + re_new_file = re.compile('new file mode .*') indent = ' ' * 6 for line in result.stdout.splitlines(): if verbose:
@@ -111,8 +112,10 @@ def CheckPatch(fname, verbose=False, show_types=False): # Skip lines which quote code if line.startswith(indent): continue - # Skip code quotes and #<n> - if line.startswith('+') or line.startswith('#'): + # Skip code quotes + if line.startswith('+'): + continue + if re_new_file.match(line): continue match = re_stats_full.match(line) if not match:
Signature · 1 line
2.29.0.rc2.309.g374f81d7ae-goog
PATCH v2 24/30 patman: Allow showing a Commit as a string
In reply to Simon GlassUse the subject of the Commit object when printing it out.
---
(no changes since v1)
tools/patman/commit.py | 3 +++ 1 file changed, 3 insertions(+)
tools/patman/commit.py +3 -0
@@ -41,6 +41,9 @@ class Commit: self.rtags = collections.defaultdict(set) self.warn = [] + def __str__(self): + return self.subject + def AddChange(self, version, info): """Add a new change line to the change list for a version.
Signature · 1 line
2.29.0.rc2.309.g374f81d7ae-goog
PATCH v2 25/30 patman: Improve handling of files
In reply to Simon GlassSometimes warnings are associated with a file and sometimes with the patch as a whole. Update the regular expression to handle both cases, even in emacs mode. Also add support for detecting new files.
---
(no changes since v1)
tools/patman/checkpatch.py | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)
tools/patman/checkpatch.py +8 -3
@@ -93,7 +93,7 @@ def CheckPatch(fname, verbose=False, show_types=False): re_error = re.compile('ERROR:%s (.*)' % type_name) re_warning = re.compile(emacs_prefix + 'WARNING:%s (.*)' % type_name) re_check = re.compile('CHECK:%s (.*)' % type_name) - re_file = re.compile('#\d+: FILE: ([^:]*):(\d+):') + re_file = re.compile('#(\d+): (FILE: ([^:]*):(\d+):)?') re_note = re.compile('NOTE: (.*)') re_new_file = re.compile('new file mode .*') indent = ' ' * 6
@@ -153,8 +153,13 @@ def CheckPatch(fname, verbose=False, show_types=False): item['msg'] = check_match.group(2) item['type'] = 'check' elif file_match: - item['file'] = file_match.group(1) - item['line'] = int(file_match.group(2)) + err_fname = file_match.group(3) + if err_fname: + item['file'] = err_fname + item['line'] = int(file_match.group(4)) + else: + item['file'] = '<patch>' + item['line'] = int(file_match.group(1)) elif subject_match: item['file'] = '<patch subject>' item['line'] = None
Signature · 1 line
2.29.0.rc2.309.g374f81d7ae-goog
PATCH v2 26/30 patman: Detect missing upstream in CountCommitsToBranch
In reply to Simon GlassAt present if we fail to find the upstream then the error output is piped to wc, resulting in bogus results. Avoid the pipe and check the output directly.
---
(no changes since v1)
tools/patman/func_test.py | 19 +++++++++++++++++++ tools/patman/gitutil.py | 10 +++++++--- 2 files changed, 26 insertions(+), 3 deletions(-)
tools/patman/func_test.py +19 -0
@@ -610,3 +610,22 @@ Series-changes: 2 self.assertEqual( ["Tag 'Series-changes' should be before sign-off / Change-Id"], pstrm.commit.warn) + + @unittest.skipIf(not HAVE_PYGIT2, 'Missing python3-pygit2') + def testNoUpstream(self): + """Test CountCommitsToBranch when there is no upstream""" + repo = self.make_git_tree() + target = repo.lookup_reference('refs/heads/base') + self.repo.checkout(target, strategy=pygit2.GIT_CHECKOUT_FORCE) + + # Check that it can detect the current branch + try: + orig_dir = os.getcwd() + os.chdir(self.gitdir) + with self.assertRaises(ValueError) as exc: + gitutil.CountCommitsToBranch(None) + self.assertIn( + "Failed to determine upstream: fatal: no upstream configured for branch 'base'", + str(exc.exception)) + finally: + os.chdir(orig_dir)
tools/patman/gitutil.py +7 -3
@@ -66,9 +66,13 @@ def CountCommitsToBranch(branch): rev_range = '%s..%s' % (us, branch) else: rev_range = '@{upstream}..' - pipe = [LogCmd(rev_range, oneline=True), ['wc', '-l']] - stdout = command.RunPipe(pipe, capture=True, oneline=True).stdout - patch_count = int(stdout) + pipe = [LogCmd(rev_range, oneline=True)] + result = command.RunPipe(pipe, capture=True, capture_stderr=True, + oneline=True, raise_on_error=False) + if result.return_code: + raise ValueError('Failed to determine upstream: %s' % + result.stderr.strip()) + patch_count = len(result.stdout.splitlines()) return patch_count def NameRevision(commit_hash):
Signature · 1 line
2.29.0.rc2.309.g374f81d7ae-goog
PATCH v2 27/30 patman: Support checking for review tags in patchwork
In reply to Simon GlassBefore sending out a new version of a series for review, it is important to add any review tags (e.g. Reviewed-by, Acked-by) collected by patchwork. Otherwise people waste time reviewing the same patch repeatedly, become frustrated and stop reviewing your patches.
To help with this, add a new 'status' subcommand that checks patchwork for review tags, showing those which are not present in the local branch.
This allows users to see what new review tags have been received and then add them.
Sample output:
$ patman status
1 Subject 1
Reviewed-by: Joe Bloggs <[email protected]>
2 Subject 2
Tested-by: Lord Edmund Blackadd?r <[email protected]>
Reviewed-by: Fred Bloggs <[email protected]>
+ Reviewed-by: Mary Bloggs <[email protected]>
1 new response available in patchworkThe '+' indicates a new tag. Colours are used to make it easier to read.
---
Changes in v2:
- Adjust the algorithm to handle patch/commit mismatch
- Correct Python style problems
- Use REST API instead of web pages
tools/patman/README | 34 ++++ tools/patman/control.py | 46 +++++ tools/patman/func_test.py | 315 +++++++++++++++++++++++++++++++++ tools/patman/main.py | 18 ++ tools/patman/status.py | 356 ++++++++++++++++++++++++++++++++++++++ tools/patman/terminal.py | 21 ++- 6 files changed, 784 insertions(+), 6 deletions(-) create mode 100644 tools/patman/status.py
tools/patman/README +34 -0
@@ -11,6 +11,8 @@ This tool is a Python script which: - Runs the patches through checkpatch.pl and its own checks - Optionally emails them out to selected people +It also shows review tags from Patchwork so you can update your local patches. + It is intended to automate patch creation and make it a less error-prone process. It is useful for U-Boot and Linux work so far, since it uses the checkpatch.pl script.
@@ -353,6 +355,38 @@ These people will get the cover letter even if they are not on the To/Cc list for any of the patches. +Patchwork Integration +===================== + +Patman has a very basic integration with Patchwork. If you point patman to +your series on patchwork it can show you what new reviews have appears since +you sent your series. + +To set this up, add a Series-link tag to one of the commits in your series +(see above). + +Then you can type + + patman status + +and patman will show you each patch and what review tags have been collected, +for example: + +... + 21 x86: mtrr: Update the command to use the new mtrr + Reviewed-by: Wolfgang Wallner <[email protected]> + + Reviewed-by: Bin Meng <[email protected]> + 22 x86: mtrr: Restructure so command execution is in + Reviewed-by: Wolfgang Wallner <[email protected]> + + Reviewed-by: Bin Meng <[email protected]> +... + +This shows that patch 21 and 22 were sent out with one review but have since +attracted another review each. If the series needs changes, you can update +these commits with the new review tag before sending the next version of the +series. + + Example Work Flow =================
tools/patman/control.py +46 -0
49 lines not on this page · raw message
tools/patman/func_test.py +315 -0
346 lines not on this page · raw message
tools/patman/main.py +18 -0
@@ -90,6 +90,10 @@ test_parser.add_argument('testname', type=str, default=None, nargs='?', help="Specify the test to run") AddCommonArgs(test_parser) +status = subparsers.add_parser('status', + help='Check status of patches in patchwork') +AddCommonArgs(status) + # Parse options twice: first to get the project and second to handle # defaults properly (which depends on project). argv = sys.argv[1:]
@@ -157,3 +161,17 @@ elif args.cmd == 'send': else: control.send(args) + +# Check status of patches in patchwork +elif args.cmd == 'status': + ret_code = 0 + try: + control.patchwork_status(args.branch, args.count, args.start, args.end) + except Exception as e: + terminal.Print('patman: %s: %s' % (type(e).__name__, e), + colour=terminal.Color.RED) + if args.debug: + print() + traceback.print_exc() + ret_code = 1 + sys.exit(ret_code)
tools/patman/status.py new file +356 -0
356 lines not on this page · raw message
tools/patman/terminal.py +15 -6
@@ -34,14 +34,22 @@ class PrintLine: newline: True to output a newline after the text colour: Text colour to use """ - def __init__(self, text, newline, colour): + def __init__(self, text, colour, newline=True, bright=True): self.text = text self.newline = newline self.colour = colour + self.bright = bright + + def __eq__(self, other): + return (self.text == other.text and + self.newline == other.newline and + self.colour == other.colour and + self.bright == other.bright) def __str__(self): - return 'newline=%s, colour=%s, text=%s' % (self.newline, self.colour, - self.text) + return ("newline=%s, colour=%s, bright=%d, text='%s'" % + (self.newline, self.colour, self.bright, self.text)) + def CalcAsciiLen(text): """Calculate the length of a string, ignoring any ANSI sequences
@@ -136,7 +144,7 @@ def Print(text='', newline=True, colour=None, limit_to_line=False, bright=True): global last_print_len if print_test_mode: - print_test_list.append(PrintLine(text, newline, colour)) + print_test_list.append(PrintLine(text, colour, newline, bright)) else: if colour: col = Color()
@@ -159,11 +167,12 @@ def PrintClear(): print('\r%s\r' % (' '* last_print_len), end='', flush=True) last_print_len = None -def SetPrintTestMode(): +def SetPrintTestMode(enable=True): """Go into test mode, where all printing is recorded""" global print_test_mode - print_test_mode = True + print_test_mode = enable + GetPrintTestLines() def GetPrintTestLines(): """Get a list of all lines output through Print()
Signature · 1 line
2.29.0.rc2.309.g374f81d7ae-goog
PATCH v2 28/30 patman: Support updating a branch with review tags
In reply to Simon GlassIt is tedious to add review tags into the local branch and errors can sometimes be made. Add an option to create a new branch with the review tags obtained from patchwork.
---
(no changes since v1)
tools/patman/README | 17 ++++++- tools/patman/control.py | 9 ++-- tools/patman/func_test.py | 100 +++++++++++++++++++++++++++++++++++++- tools/patman/main.py | 7 ++- tools/patman/status.py | 89 +++++++++++++++++++++++++++++++-- 5 files changed, 210 insertions(+), 12 deletions(-)
tools/patman/README +15 -2
@@ -11,11 +11,13 @@ This tool is a Python script which: - Runs the patches through checkpatch.pl and its own checks - Optionally emails them out to selected people -It also shows review tags from Patchwork so you can update your local patches. +It also has some Patchwork features: +- shows review tags from Patchwork so you can update your local patches +- pulls these down into a new branch on request It is intended to automate patch creation and make it a less error-prone process. It is useful for U-Boot and Linux work so far, -since it uses the checkpatch.pl script. +since they use the checkpatch.pl script. It is configured almost entirely by tags it finds in your commits. This means that you can work on a number of different branches at
@@ -386,6 +388,17 @@ attracted another review each. If the series needs changes, you can update these commits with the new review tag before sending the next version of the series. +To automatically pull into these tags into a new branch, use the -d option: + + patman status -d mtrr4 + +This will create a new 'mtrr4' branch which is the same as your current branch +but has the new review tags in it. You can check that this worked with: + + patman -b mtrr4 status + +which should show that there are no new responses compared to this new branch. + Example Work Flow =================
tools/patman/control.py +6 -3
@@ -176,11 +176,11 @@ def send(args): args.limit, args.dry_run, args.in_reply_to, args.thread, args.smtp_server) -def patchwork_status(branch, count, start, end): +def patchwork_status(branch, count, start, end, dest_branch, force): """Check the status of patches in patchwork This finds the series in patchwork using the Series-link tag, checks for new - comments / review tags and displays them + review tags, displays then and creates a new branch with the review tags. Args: branch (str): Branch to create patches from (None = current)
@@ -189,6 +189,9 @@ def patchwork_status(branch, count, start, end): start (int): Start partch to use (0=first / top of branch) end (int): End patch to use (0=last one in series, 1=one before that, etc.) + dest_branch (str): Name of new branch to create with the updated tags + (None to not create a branch) + force (bool): With dest_branch, force overwriting an existing branch Raises: ValueError: if the branch has no Series-link value
@@ -220,4 +223,4 @@ def patchwork_status(branch, count, start, end): # Import this here to avoid failing on other commands if the dependencies # are not present from patman import status - status.check_patchwork_status(series, found[0]) + status.check_patchwork_status(series, found[0], branch, dest_branch, force)
tools/patman/func_test.py +98 -2
109 lines not on this page · raw message
tools/patman/main.py +6 -1
@@ -92,6 +92,10 @@ AddCommonArgs(test_parser) status = subparsers.add_parser('status', help='Check status of patches in patchwork') +status.add_argument('-d', '--dest-branch', type=str, + help='Name of branch to create with collected responses') +status.add_argument('-f', '--force', action='store_true', + help='Force overwriting an existing branch') AddCommonArgs(status) # Parse options twice: first to get the project and second to handle
@@ -166,7 +170,8 @@ elif args.cmd == 'send': elif args.cmd == 'status': ret_code = 0 try: - control.patchwork_status(args.branch, args.count, args.start, args.end) + control.patchwork_status(args.branch, args.count, args.start, args.end, + args.dest_branch, args.force) except Exception as e: terminal.Print('patman: %s: %s' % (type(e).__name__, e), colour=terminal.Color.RED)
tools/patman/status.py +85 -4
118 lines not on this page · raw message
Signature · 1 line
2.29.0.rc2.309.g374f81d7ae-goog
PATCH v2 29/30 patman: Support parsing of review snippets
In reply to Simon GlassAdd support for parsing the contents of a patchwork 'patch' web page containing comments received from reviewers. This allows patman to show these comments in a simple 'snippets' format.
A snippet is some quoted code plus some unquoted comments below it. Each review is from a unique person/email and can produce multiple snippets, one for each part of the code that attracts a comment.
Show the file and line-number info at the top of each snippet if available.
---
(no changes since v1)
tools/patman/func_test.py | 83 ++++++++++++++++++++++++++++++++++++ tools/patman/patchstream.py | 84 +++++++++++++++++++++++++++++++++++++ 2 files changed, 167 insertions(+)
tools/patman/func_test.py +83 -0
@@ -1040,3 +1040,86 @@ Series-changes: 2 {'Tested-by': {self.leb}, 'Reviewed-by': {self.fred, self.mary}}, new_series.commits[1].rtags) + + def testParseSnippets(self): + """Test parsing of review snippets""" + text = '''Hi Fred, + +This is a comment from someone. + +Something else + +On some recent date, Fred wrote: +> This is why I wrote the patch +> so here it is + +Now a comment about the commit message +A little more to say + +Even more + +> diff --git a/file.c b/file.c +> Some more code +> Code line 2 +> Code line 3 +> Code line 4 +> Code line 5 +> Code line 6 +> Code line 7 +> Code line 8 +> Code line 9 + +And another comment + +> @@ -153,8 +143,13 @@ def CheckPatch(fname, show_types=False): +> further down on the file +> and more code +> +Addition here +> +Another addition here +> codey +> more codey + +and another thing in same file + +> @@ -253,8 +243,13 @@ +> with no function context + +one more thing + +> diff --git a/tools/patman/main.py b/tools/patman/main.py +> +line of code +now a very long comment in a different file +line2 +line3 +line4 +line5 +line6 +line7 +line8 +''' + pstrm = PatchStream.process_text(text, True) + self.assertEqual([], pstrm.commit.warn) + + # We expect to the filename and up to 5 lines of code context before + # each comment. The 'On xxx wrote:' bit should be removed. + self.assertEqual( + [['Hi Fred,', + 'This is a comment from someone.', + 'Something else'], + ['> This is why I wrote the patch', + '> so here it is', + 'Now a comment about the commit message', + 'A little more to say', 'Even more'], + ['> File: file.c', '> Code line 5', '> Code line 6', + '> Code line 7', '> Code line 8', '> Code line 9', + 'And another comment'], + ['> File: file.c', + '> Line: 153 / 143: def CheckPatch(fname, show_types=False):', + '> and more code', '> +Addition here', '> +Another addition here', + '> codey', '> more codey', 'and another thing in same file'], + ['> File: file.c', '> Line: 253 / 243', + '> with no function context', 'one more thing'], + ['> File: tools/patman/main.py', '> +line of code', + 'now a very long comment in a different file', + 'line2', 'line3', 'line4', 'line5', 'line6', 'line7', 'line8']], + pstrm.snippets)
tools/patman/patchstream.py +84 -0
137 lines not on this page · raw message
Signature · 1 line
2.29.0.rc2.309.g374f81d7ae-goog
PATCH v2 30/30 patman: Support listing comments from patchwork
In reply to Simon GlassWhile reviewing feedback it is helpful to see the review comments on the command line to check that each has been addressed. Add an option to support that.
Update the workflow documentation to describe the new features.
---
Changes in v2:
- Many changes to the whole series, including new patches
- Use of REST API instead of the web site
tools/patman/README | 36 +++++-- tools/patman/control.py | 11 ++- tools/patman/func_test.py | 186 +++++++++++++++++++++++++++++++++--- tools/patman/main.py | 5 +- tools/patman/patchstream.py | 7 +- tools/patman/status.py | 59 ++++++++++-- 6 files changed, 268 insertions(+), 36 deletions(-)
tools/patman/README +29 -7
@@ -14,6 +14,7 @@ This tool is a Python script which: It also has some Patchwork features: - shows review tags from Patchwork so you can update your local patches - pulls these down into a new branch on request +- lists comments received on a series It is intended to automate patch creation and make it a less error-prone process. It is useful for U-Boot and Linux work so far,
@@ -399,6 +400,8 @@ but has the new review tags in it. You can check that this worked with: which should show that there are no new responses compared to this new branch. +There is also a -C option to list the comments received for each patch. + Example Work Flow =================
@@ -483,17 +486,33 @@ people on the list don't see your secret info. Of course patches often attract comments and you need to make some updates. Let's say one person sent comments and you get an Acked-by: on one patch. Also, the patch on the list that you were waiting for has been merged, -so you can drop your wip commit. So you resync with upstream: +so you can drop your wip commit. + +Take a look on patchwork and find out the URL of the series. This will be +something like http://patchwork.ozlabs.org/project/uboot/list/?series=187331 +Add this to a tag in your top commit: + + Series-link: http://patchwork.ozlabs.org/project/uboot/list/?series=187331 + +You can use then patman to collect the Acked-by tag to the correct commit, +creating a new 'version 2' branch for us-cmd: + + patman status -d us-cmd2 + git checkout us-cmd2 + +You can look at the comments in Patchwork or with: + + patman status -C + +Then you can resync with upstream: git fetch origin (or whatever upstream is called) git rebase origin/master -and use git rebase -i to edit the commits, dropping the wip one. You add -the ack tag to one commit: - - Acked-by: Heiko Schocher <[email protected]> +and use git rebase -i to edit the commits, dropping the wip one. -update the Series-cc: in the top commit: +Then update the Series-cc: in the top commit to add the person who reviewed +the v1 series: Series-cc: bfin, marex, Heiko Schocher <[email protected]>
@@ -532,7 +551,9 @@ so to send them: and it will create and send the version 2 series. -General points: + +General points +============== 1. When you change back to the us-cmd branch days or weeks later all your information is still there, safely stored in the commits. You don't need
@@ -612,3 +633,4 @@ a bad thing. Simon Glass <[email protected]> v1, v2, 19-Oct-11 revised v3 24-Nov-11 +revised v4 Independence Day 2020, with Patchwork integration
tools/patman/control.py +8 -3
@@ -176,11 +176,13 @@ def send(args): args.limit, args.dry_run, args.in_reply_to, args.thread, args.smtp_server) -def patchwork_status(branch, count, start, end, dest_branch, force): +def patchwork_status(branch, count, start, end, dest_branch, force, + show_comments): """Check the status of patches in patchwork This finds the series in patchwork using the Series-link tag, checks for new - review tags, displays then and creates a new branch with the review tags. + comments and review tags, displays then and creates a new branch with the + review tags. Args: branch (str): Branch to create patches from (None = current)
@@ -192,6 +194,8 @@ def patchwork_status(branch, count, start, end, dest_branch, force): dest_branch (str): Name of new branch to create with the updated tags (None to not create a branch) force (bool): With dest_branch, force overwriting an existing branch + show_comments (bool): True to display snippets from the comments + provided by reviewers Raises: ValueError: if the branch has no Series-link value
@@ -223,4 +227,5 @@ def patchwork_status(branch, count, start, end, dest_branch, force): # Import this here to avoid failing on other commands if the dependencies # are not present from patman import status - status.check_patchwork_status(series, found[0], branch, dest_branch, force) + status.check_patchwork_status(series, found[0], branch, dest_branch, force, + show_comments)
tools/patman/func_test.py +172 -14
242 lines not on this page · raw message
tools/patman/main.py +4 -1
@@ -92,6 +92,8 @@ AddCommonArgs(test_parser) status = subparsers.add_parser('status', help='Check status of patches in patchwork') +status.add_argument('-C', '--show-comments', action='store_true', + help='Show comments from each patch') status.add_argument('-d', '--dest-branch', type=str, help='Name of branch to create with collected responses') status.add_argument('-f', '--force', action='store_true',
@@ -171,7 +173,8 @@ elif args.cmd == 'status': ret_code = 0 try: control.patchwork_status(args.branch, args.count, args.start, args.end, - args.dest_branch, args.force) + args.dest_branch, args.force, + args.show_comments) except Exception as e: terminal.Print('patman: %s: %s' % (type(e).__name__, e), colour=terminal.Color.RED)
tools/patman/patchstream.py +4 -3
@@ -89,11 +89,12 @@ class PatchStream: self.blank_count = 0 # Number of blank lines stored up self.state = STATE_MSG_HEADER # What state are we in? self.commit = None # Current commit - self.snippets = [] # List of unquoted test blocks + # List of unquoted test blocks, each a list of str lines + self.snippets = [] self.cur_diff = None # Last 'diff' line seen (str) self.cur_line = None # Last context (@@) line seen (str) - self.recent_diff= None # 'diff' line for current snippet (str) - self.recent_line= None # '@@' line for current snippet (str) + self.recent_diff = None # 'diff' line for current snippet (str) + self.recent_line = None # '@@' line for current snippet (str) self.recent_quoted = collections.deque([], 5) self.recent_unquoted = queue.Queue() self.was_quoted = None
tools/patman/status.py +51 -8
133 lines not on this page · raw message
Signature · 1 line
2.29.0.rc2.309.g374f81d7ae-goog