Add 'patman' patch generation, checking and submission script

What is this?
=============

This tool is a Python script which:
- Creates patch directly from your branch
- Cleans them up by removing unwanted tags
- Inserts a cover letter with change lists
- Runs the patches through checkpatch.pl and its own checks
- Optionally emails them out to selected people

It is intended to automate patch creation and make it a less error-prone process. It is useful for U-Boot and Linux work so far, since it uses the checkpatch.pl script.

It is configured almost entirely by tags it finds in your commits. This means that you can work on a number of different branches at once, and keep the settings with each branch rather than having to git format-patch, git send-email, etc. with the correct parameters each time. So for example if you put:

in one of your commits, the series will be sent there.

See the README file for full details.

Hi,

Hmmm patman found a tag in this commit and tried to send it to Fred Bloggs. I have added the line below - sorry for the confusion.

Quoted 9 lines · Simon Glass

What is this?
=============

This tool is a Python script which:
- Creates patch directly from your branch
- Cleans them up by removing unwanted tags
- Inserts a cover letter with change lists
- Runs the patches through checkpatch.pl and its own checks
- Optionally emails them out to selected people

Hi Simon,

Le 15/01/2012 02:20, Simon Glass a ?crit :

Quoted 16 lines

Hi,

Hmmm patman found a tag in this commit and tried to send it to Fred Bloggs. I have added the line below - sorry for the confusion.

Quoted 10 lines · On Sat, Jan 14, 2012 at 5:12 PM, Simon Glass<[email protected]> ?wrote:

What is this?
=============

This tool is a Python script which:
- Creates patch directly from your branch
- Cleans them up by removing unwanted tags
- Inserts a cover letter with change lists
- Runs the patches through checkpatch.pl and its own checks
- Optionally emails them out to selected people

It seems to me that this is not specific to u-boot, and as such, has no reason to be managed along with U-Boot.

Why should it not simply have its own project and resources?

Amicalement,
--
Albert.

Hi Albert,

Quoted 21 lines · Albert ARIBAUD

Hi Simon,

Le 15/01/2012 02:20, Simon Glass a ?crit :

Quoted 16 lines

Hi,

Hmmm patman found a tag in this commit and tried to send it to Fred Bloggs. I have added the line below - sorry for the confusion.

Quoted 10 lines · Simon Glass

What is this?
=============

This tool is a Python script which:
- Creates patch directly from your branch
- Cleans them up by removing unwanted tags
- Inserts a cover letter with change lists
- Runs the patches through checkpatch.pl and its own checks
- Optionally emails them out to selected people

Quoted 4 lines

It seems to me that this is not specific to u-boot, and as such, has no reason to be managed along with U-Boot.

Why should it not simply have its own project and resources?

It was created in response to Wolfgang's comments that the sometimes low quality of patches on the U-Boot ML chews up a lot of time.

It is not technically specific to U-Boot (since it could also be used with the Linux kernel), but that was my intent when creating it. If this goes in then I plan to look at how to automate build testing also so we can more easily see that patches are MAKEALL-clean.

The discussion at the time was here:

http://lists.denx.de/pipermail/u-boot/2011-October/105862.html

and the many messages which followed. My patch was in response to this.

Regards,
Simon

Hi Simon,

Le 03/02/2012 20:30, Simon Glass a ?crit :

Quoted 42 lines

Hi Albert,

Quoted 24 lines · On Fri, Feb 3, 2012 at 11:00 AM, Albert ARIBAUD <[email protected]>…

Hi Simon,

Le 15/01/2012 02:20, Simon Glass a ?crit :

Quoted 17 lines

Hi,

Hmmm patman found a tag in this commit and tried to send it to Fred Bloggs. I have added the line below - sorry for the confusion.

Quoted 11 lines · On Sat, Jan 14, 2012 at 5:12 PM, Simon Glass<[email protected]> ? ?wrote:

What is this?
=============

This tool is a Python script which:
- Creates patch directly from your branch
- Cleans them up by removing unwanted tags
- Inserts a cover letter with change lists
- Runs the patches through checkpatch.pl and its own checks
- Optionally emails them out to selected people

Quoted 4 lines

It seems to me that this is not specific to u-boot, and as such, has no reason to be managed along with U-Boot.

Why should it not simply have its own project and resources?

It was created in response to Wolfgang's comments that the sometimes low quality of patches on the U-Boot ML chews up a lot of time.

It is not technically specific to U-Boot (since it could also be used with the Linux kernel), but that was my intent when creating it. If this goes in then I plan to look at how to automate build testing also so we can more easily see that patches are MAKEALL-clean.

Quoted 5 lines

The discussion at the time was here:

http://lists.denx.de/pipermail/u-boot/2011-October/105862.html

and the many messages which followed. My patch was in response to this.

This aptly explains why the tool was created and what it intends to attain, and that its use in U-Boot was discussed -- things that I did not question in the first place, and still do not.

What I question is the relevance of tying together the development, source code management, release management etc of U-Boot and patman. Is there any reason that a release of U-Boot should also be a release of patman, or the other way around? Is there any reason even to have similar development models for both projects?

As an example, the creation of git was strongly motivated by, and tied to, source code management requirements of Linux, but git is not integrated in the Linux source tree and its development is fully independent from that of Linux.

My point is that patman, not being tied to a given project and being certainly just as useful to may other git-based projects, it should have its own project, be able to evolve at its own pace, etc.

Amicalement,
--
Albert.

One advantage of having it in the U-Boot source tree tools directory is that people might find it and use it (at least at the early stages). If it outgrows its space there due to extra features or more generic use then it could move perhaps.

Regards,
Simon

What is this?
=============

This tool is a Python script which:
- Creates patch directly from your branch
- Cleans them up by removing unwanted tags
- Inserts a cover letter with change lists
- Runs the patches through checkpatch.pl and its own checks
- Optionally emails them out to selected people

It is intended to automate patch creation and make it a less error-prone process. It is useful for U-Boot and Linux work so far, since it uses the checkpatch.pl script.

It is configured almost entirely by tags it finds in your commits. This means that you can work on a number of different branches at once, and keep the settings with each branch rather than having to git format-patch, git send-email, etc. with the correct parameters each time. So for example if you put:

Hi Simon,

OK I haven't tried it yet, but this sounds awesome.
I wonder how people manage to send and rework their patches without such tool. Even one single patchset has been giving me strong enough headaches so far, not to mention the massive waste of time.

I pretty much agree with Albert, this should eventually move out of u-boot. But you need to start somewhere, and this is perhaps a good testbed to get people to use it. I believe it should perhaps eventually be integrated into git as it makes for a wonderful enhancement (or wrapper) over git format-patch and git send-email.

As I said I haven't tested it yet, but I would like to contribute a couple questions / suggestions for enhancements out of your README:

1) Marking the test setup commits using tags as well. Something like

?Series-exclude: true

I mean, I tend to forget (and make mistakes) pretty easily. Not having to remember that a given commit is for testing only makes it more difficult for me to go wrong. Even that extra "-s1" I could easily miss... Also, it *might* be also useful to have those test commits somewhere in the middle of the patch series, perhaps.

2) Do you think it would be possible to write the cover letter on a commit of its own? I believe git doesn't allow you to create a commit not touching any file, but perhaps one might find some way arount it as well.

Maybe the cover letter itself could be written as an added file to such commit, and then tagged with something like:

?Cover-letter-file: wonderfulpatchset.txt

This might turn out useful, as one could easily edit the file while reworking the patchset from the top commit, and then attribute it to such commit, wherever it is located in the tree.

What do you think?

Thanks again for the tool!

Gerlando

Quoted 20 lines · Simon Glass

What is this?
=============

This tool is a Python script which:
- Creates patch directly from your branch
- Cleans them up by removing unwanted tags
- Inserts a cover letter with change lists
- Runs the patches through checkpatch.pl and its own checks
- Optionally emails them out to selected people

It is intended to automate patch creation and make it a less error-prone process. It is useful for U-Boot and Linux work so far, since it uses the checkpatch.pl script.

It is configured almost entirely by tags it finds in your commits. This means that you can work on a number of different branches at once, and keep the settings with each branch rather than having to git format-patch, git send-email, etc. with the correct parameters each time. So for example if you put:

[...]

Hi Gerlando,

Quoted 6 lines · Gerlando Falauto

Hi Simon,

OK I haven't tried it yet, but this sounds awesome.
I wonder how people manage to send and rework their patches without such tool. Even one single patchset has been giving me strong enough headaches so far, not to mention the massive waste of time.

Yes I wrote it when I had to do a few revisions of a relatively small series. It was very difficult and time-consuming to get everything right for submission. Now it is mostly automatic.

Quoted 1 line

Quoted 5 lines

I pretty much agree with Albert, this should eventually move out of u-boot. But you need to start somewhere, and this is perhaps a good testbed to get people to use it. I believe it should perhaps eventually be integrated into git as it makes for a wonderful enhancement (or wrapper) over git format-patch and git send-email.

Yes, the only thing that would need to be sorted out is the hooks for checkpatch.

Quoted 5 lines

As I said I haven't tested it yet, but I would like to contribute a couple questions / suggestions for enhancements out of your README:

1) Marking the test setup commits using tags as well. Something like

Quoted 7 lines

Series-exclude: true

I mean, I tend to forget (and make mistakes) pretty easily. Not having to remember that a given commit is for testing only makes it more difficult for me to go wrong. Even that extra "-s1" I could easily miss... Also, it *might* be also useful to have those test commits somewhere in the middle of the patch series, perhaps.

Yes I think that is useful, and it fits with the idea of not needed any args in the normal case. I will stick it on the list.

Quoted 4 lines

2) Do you think it would be possible to write the cover letter on a commit of its own? I believe git doesn't allow you to create a commit not touching any file, but perhaps one might find some way arount it as well.

You can put it in any commit, and in principle in its own commit, but 'git rebase -i' doesn't like empty commits in my experience.

Quoted 9 lines

Maybe the cover letter itself could be written as an added file to such commit, and then tagged with something like:

Cover-letter-file: wonderfulpatchset.txt

This might turn out useful, as one could easily edit the file while reworking the patchset from the top commit, and then attribute it to such commit, wherever it is located in the tree.

Quoted 1 line

What do you think?

Easy to do - I wonder if it might be better to commit that file to a separate commit (which is marked Series-exclude:). Otherwise you have a dangling file that might hang around for weeks and is very vulnerable to accidents.

Quoted 1 line

Thanks again for the tool!

Thanks for looking at it. We will see if this goes into U-Boot this time.

Regards,
Simon

Quoted 26 lines

Gerlando

Quoted 20 lines · Simon Glass

What is this?
=============

This tool is a Python script which:
- Creates patch directly from your branch
- Cleans them up by removing unwanted tags
- Inserts a cover letter with change lists
- Runs the patches through checkpatch.pl and its own checks
- Optionally emails them out to selected people

It is intended to automate patch creation and make it a less error-prone process. It is useful for U-Boot and Linux work so far, since it uses the checkpatch.pl script.

It is configured almost entirely by tags it finds in your commits. This means that you can work on a number of different branches at once, and keep the settings with each branch rather than having to git format-patch, git send-email, etc. with the correct parameters each time. So for example if you put:

[...]

Quoted 30 lines · Simon Glass

Hi Gerlando,

Quoted 8 lines · On Wed, Apr 4, 2012 at 3:35 AM, Gerlando Falauto…

Hi Simon,

OK I haven't tried it yet, but this sounds awesome.
I wonder how people manage to send and rework their patches without such tool. Even one single patchset has been giving me strong enough headaches so
far, not to mention the massive waste of time.

Yes I wrote it when I had to do a few revisions of a relatively small series. It was very difficult and time-consuming to get everything right for submission. Now it is mostly automatic.

Quoted 9 lines

I pretty much agree with Albert, this should eventually move out of u-boot.
But you need to start somewhere, and this is perhaps a good testbed to get
people to use it. I believe it should perhaps eventually be integrated into
git as it makes for a wonderful enhancement (or wrapper) over git format-patch and git send-email.

Quoted 2 lines

Yes, the only thing that would need to be sorted out is the hooks for checkpatch.

Wouldn't something in ~/.gitconfig do the trick?

Quoted 42 lines

Quoted 10 lines

As I said I haven't tested it yet, but I would like to contribute a couple
questions / suggestions for enhancements out of your README:

1) Marking the test setup commits using tags as well. Something like

?Series-exclude: true

Quoted 7 lines

I mean, I tend to forget (and make mistakes) pretty easily. Not having to remember that a given commit is for testing only makes it more difficult for
me to go wrong. Even that extra "-s1" I could easily miss... Also, it *might* be also useful to have those test commits somewhere in the middle of
the patch series, perhaps.

Yes I think that is useful, and it fits with the idea of not needed any args in the normal case. I will stick it on the list.

Quoted 6 lines

2) Do you think it would be possible to write the cover letter on a commit
of its own? I believe git doesn't allow you to create a commit not touching
any file, but perhaps one might find some way arount it as well.

You can put it in any commit, and in principle in its own commit, but 'git rebase -i' doesn't like empty commits in my experience.

Quoted 9 lines

Maybe the cover letter itself could be written as an added file to such commit, and then tagged with something like:

?Cover-letter-file: wonderfulpatchset.txt

This might turn out useful, as one could easily edit the file while reworking the patchset from the top commit, and then attribute it to such commit, wherever it is located in the tree.

Quoted 7 lines

Quoted 1 line

What do you think?

Easy to do - I wonder if it might be better to commit that file to a separate commit (which is marked Series-exclude:). Otherwise you have a dangling file that might hang around for weeks and is very vulnerable to accidents.

You mean not committing the cover letter file being the alternative? I believe you definitely *don't* want files hanging (not under revision control) anywhere, at any time. At least, _I don't_.

So the cover letter commit could contain all the needed information, (like recipients, revision etc) including a file with the text of the cover letter. Having it marked as Series-exclude (or whatever) would just not count it as a patch file to send out.
Not quite sure whether it's trivial to deal with the numbering though.

Quoted 6 lines

Quoted 3 lines

Thanks again for the tool!

Thanks for looking at it. We will see if this goes into U-Boot this time.

Thanks,
Gerlando

Quoted 31 lines

Regards,
Simon

Quoted 27 lines

Gerlando

Quoted 21 lines · Simon Glass

What is this?
=============

This tool is a Python script which:
- Creates patch directly from your branch
- Cleans them up by removing unwanted tags
- Inserts a cover letter with change lists
- Runs the patches through checkpatch.pl and its own checks
- Optionally emails them out to selected people

It is intended to automate patch creation and make it a less error-prone process. It is useful for U-Boot and Linux work so far, since it uses the checkpatch.pl script.

It is configured almost entirely by tags it finds in your commits. This means that you can work on a number of different branches at once, and keep the settings with each branch rather than having to git format-patch, git send-email, etc. with the correct parameters each time. So for example if you put:

[...]

What is this?
=============

This tool is a Python script which:
- Creates patch directly from your branch
- Cleans them up by removing unwanted tags
- Inserts a cover letter with change lists
- Runs the patches through checkpatch.pl and its own checks
- Optionally emails them out to selected people

It is intended to automate patch creation and make it a less error-prone process. It is useful for U-Boot and Linux work so far, since it uses the checkpatch.pl script.

It is configured almost entirely by tags it finds in your commits. This means that you can work on a number of different branches at once, and keep the settings with each branch rather than having to git format-patch, git send-email, etc. with the correct parameters each time. So for example if you put:

in one of your commits, the series will be sent there.

See the README file for full details.
END

Signed-off-by Simon Glass <[email protected]>

---
Changes in v2:
- Fix up use of Color in series.py (avoid runtime error)

Changes in v3:
- Add a few more tests (but still very incomplete)
- Add option to display the README
- Change cc-list tmpfile name to start with patman
- Expand the README to include info about suggested workflow
- Find checkpatch.pl in U-Boot tree
- Fix generation of cc list to match patch names correctly
- Support git's alias file if found in config sendemail.aliasesfile
- Support recursive aliases
- Tidy up gitutil and settings to separate code better
- Use -M with git format-patch to so that file moves are tracked
- Use @{upstream} instead of the previous mess (thanks Anton Staff)

Changes in v4:
- Add --no-tags options to not Cc maintainers on tags
- Add per-commit change log (automatically puts them all in cover letter)
- Allow multiple Signed-off-by tags
- Append valid tags to the patch
- Make aliases additive so you can add more to an alias later
- Sort change lists

 tools/patman/.gitignore     |    1 +
 tools/patman/README         |  408 +++++++++++++++++++++++++++++++++++++++
 tools/patman/checkpatch.py  |  161 ++++++++++++++++
 tools/patman/command.py     |   72 +++++++
 tools/patman/commit.py      |   87 +++++++++
 tools/patman/gitutil.py     |  372 ++++++++++++++++++++++++++++++++++++
 tools/patman/patchstream.py |  444 +++++++++++++++++++++++++++++++++++++++++++
 tools/patman/patman         |    1 +
 tools/patman/patman.py      |  153 +++++++++++++++
 tools/patman/series.py      |  238 +++++++++++++++++++++++
 tools/patman/settings.py    |   81 ++++++++
 tools/patman/terminal.py    |   86 +++++++++
 tools/patman/test.py        |  250 ++++++++++++++++++++++++
 13 files changed, 2354 insertions(+), 0 deletions(-)
 create mode 100644 tools/patman/.gitignore
 create mode 100644 tools/patman/README
 create mode 100644 tools/patman/checkpatch.py
 create mode 100644 tools/patman/command.py
 create mode 100644 tools/patman/commit.py

 create mode 100644 tools/patman/gitutil.py
 create mode 100644 tools/patman/patchstream.py
 create mode 120000 tools/patman/patman
 create mode 100755 tools/patman/patman.py
 create mode 100644 tools/patman/series.py
 create mode 100644 tools/patman/settings.py
 create mode 100644 tools/patman/terminal.py
 create mode 100644 tools/patman/test.py