checkpatch: improve email parsing
Subsystems (2)
PATCH v3 1/1 checkpatch: improve email parsing
checkpatch doesn't report warnings for many common mistakes in emails. Some of which are trailing commas and incorrect use of email comments.
At the same time several false positives are reported due to incorrect handling of mail comments. The most common of which is due to the pattern:
<[email protected]> # X.X
Improve email parsing in checkpatch.
Some general comment rules are defined:
- Multiple name comments should not be allowed.
- Comments inside address should not be allowed.
- In general comments should be enclosed within parentheses. Exception for [email protected] # X.X
Improvements to parsing:
- Detect and report unexpected content after email.
- Quoted names are excluded from comment parsing.
- Trailing dots or commas in email are removed during formatting. Correspondingly a BAD_SIGN_OFF warning is emitted.
- Improperly quoted email like '"name <address>"' are now warned about.
In addition, added fixes for all the possible rules.
--- scripts/checkpatch.pl | 88 ++++++++++++++++++++++++++++++++++--------- 1 file changed, 71 insertions(+), 17 deletions(-)
scripts/checkpatch.pl +71 -17
@@ -1152,6 +1152,7 @@ sub parse_email { my ($formatted_email) = @_; my $name = ""; + my $quoted = ""; my $name_comment = ""; my $address = ""; my $comment = "";
@@ -1183,14 +1184,20 @@ sub parse_email { } } - $comment = trim($comment); - $name = trim($name); - $name =~ s/^\"|\"$//g; - if ($name =~ s/(\s*\([^\)]+\))\s*//) { - $name_comment = trim($1); + # Extract comments from names excluding quoted parts + # "John A. (Kennedy)" - Do not extract + if ($name =~ s/\"(.+)\"//) { + $quoted = $1; + } + while ($name =~ s/\s*($balanced_parens)\s*/ /) { + $name_comment .= trim($1); } + $name =~ s/^[ \"]+|[ \"]+$//g; + $name = trim("$quoted $name"); + $address = trim($address); $address =~ s/^\<|\>$//g; + $comment = trim($comment); if ($name =~ /[^\w \-]/i) { ##has "must quote" chars $name =~ s/(?<!\\)"/\\"/g; ##escape quotes
@@ -1205,17 +1212,20 @@ sub format_email { my $formatted_email; - $name_comment = trim($name_comment); - $comment = trim($comment); - $name = trim($name); - $name =~ s/^\"|\"$//g; + $name =~ s/^[ \"]+|[ \"]+$//g; $address = trim($address); + $address =~ s/(?:\.|\,)+$//; ##trailing commas or dots if ($name =~ /[^\w \-]/i) { ##has "must quote" chars $name =~ s/(?<!\\)"/\\"/g; ##escape quotes $name = "\"$name\""; } + $name_comment = trim($name_comment); + $name_comment = " $name_comment" if length($name_comment) > 0; + $comment = trim($comment); + $comment = " $comment" if length($comment) > 0; + if ("$name" eq "") { $formatted_email = "$address"; } else {
@@ -1233,15 +1243,11 @@ sub reformat_email { } sub same_email_addresses { - my ($email1, $email2, $match_comment) = @_; + my ($email1, $email2) = @_; my ($email1_name, $name1_comment, $email1_address, $comment1) = parse_email($email1); my ($email2_name, $name2_comment, $email2_address, $comment2) = parse_email($email2); - if ($match_comment != 1) { - return $email1_name eq $email2_name && - $email1_address eq $email2_address; - } return $email1_name eq $email2_name && $email1_address eq $email2_address && $name1_comment eq $name2_comment &&
@@ -2704,7 +2710,7 @@ sub process { $signoff++; $in_commit_log = 0; if ($author ne '' && $authorsignoff != 1) { - if (same_email_addresses($1, $author, 1)) { + if (same_email_addresses($1, $author)) { $authorsignoff = 1; } else { my $ctx = $1;
@@ -2800,9 +2806,57 @@ sub process { $dequoted =~ s/" </ </; # Don't force email to have quotes # Allow just an angle bracketed address - if (!same_email_addresses($email, $suggested_email, 0)) { + if (!same_email_addresses($email, $suggested_email)) { + if (WARN("BAD_SIGN_OFF", + "email address '$email' might be better as '$suggested_email'\n" . $herecurr) && + $fix) { + $fixed[$fixlinenr] =~ s/\Q$email\E/$suggested_email/; + } + } + + # Address part shouldn't have comments + my $stripped_address = $email_address; + $stripped_address =~ s/\([^\(\)]*\)//g; + if ($email_address ne $stripped_address) { + if (WARN("BAD_SIGN_OFF", + "address part of email should not have comments: '$email_address'\n" . $herecurr) && + $fix) { + $fixed[$fixlinenr] =~ s/\Q$email_address\E/$stripped_address/; + } + } + + # Only one name comment should be allowed + my $comment_count = () = $name_comment =~ /\([^\)]+\)/g; + if ($comment_count > 1) { WARN("BAD_SIGN_OFF", - "email address '$email' might be better as '$suggested_email'\n" . $herecurr); + "Use a single name comment in email: '$email'\n" . $herecurr); + } + + # Comments must begin only with ( + # or # in case of [email protected] + if ($email =~ /^.*stable\@vger/) { + if ($comment ne "" && $comment !~ /^#.+/) { + if (WARN("BAD_SIGN_OFF", + "Invalid comment format for stable: '$email', prefer parentheses\n" . $herecurr) && + $fix) { + my $new_comment = $comment; + $new_comment =~ s/^[ \(\[]+|[ \)\]]+$//g; + $new_comment = " # $new_comment" if length($new_comment) > 0; + $fixed[$fixlinenr] =~ s/\s*\Q$comment\E$/$new_comment/; + } + } + } else { + if ($comment ne "" && $comment !~ /^\(.+\)$/) { + if (WARN("BAD_SIGN_OFF", + "Unexpected content after email: '$email'\n" . $herecurr) && + $fix) { + my $new_comment = $comment; + $new_comment =~ s/^(?:\#|\/\*|\.|\,)//g; + $new_comment =~ s/^[ \{\[]+|[ \}\]]+$//g; + $new_comment = " ($new_comment)" if length($new_comment) > 0; + $fixed[$fixlinenr] =~ s/\s*\Q$comment\E$/$new_comment/; + } + } } }
Signature · 1 line
2.27.0
Joe Perches
maintainer(adding stable and Greg KH for additional review)
Quoted 3 lines · Dwaipayan Ray
checkpatch doesn't report warnings for many common mistakes in emails. Some of which are trailing commas and incorrect use of email comments.
I presume you've tested this against the git tree.
Can you send me a file with the BAD_SIGN_OFF messages generated and if possible the git SHA-1s of the commits?
Quoted 6 lines
At the same time several false positives are reported due to incorrect handling of mail comments. The most common of which is due to the pattern:
<[email protected]> # X.X
Quoted 8 lines
Improve email parsing in checkpatch.
Some general comment rules are defined:
- Multiple name comments should not be allowed.
- Comments inside address should not be allowed.
- In general comments should be enclosed within parentheses. Exception for [email protected] # X.X
not just vger.kernel.org, but this should also allow [email protected] and only allow cc: and not any other -by: type for that email address.
A process preference question for Greg and the stable team:
The most common stable forms are
[email protected] # version info
then
[email protected] [ version info ]
with some other relatively infrequently used outlier styles, some that use parentheses, but this is not frequent.
It might be sensible to standardize on the "# version info" trailer comment version info style and warn on any other form.
A somewhat common style for the stable address is to use a name before the stable address which describes the version info:
Perhaps any name before stable should be warned and the version should be a comment.
Here's a list of the stable addresses with "version name" then stable address in the git tree and other outlier styles.
24 linux-stable <[email protected]> 21 5.4+ <[email protected]> 14 All applicable <[email protected]> 6 3.10+ <[email protected]> 5 5.9+ <[email protected]> 5 5.3+ <[email protected]> 5 5.1+ <[email protected]> 4 5.6+ <[email protected]> 4 4.20+ <[email protected]> 3 Stable Team <[email protected]> 3 4.19+ <[email protected]> 3 4.15+ <[email protected]> 3 4.10+ <[email protected]> 2 [email protected] (v2.6.12+) 2 5.2+ <[email protected]> 2 4.16+ <[email protected]> 1 v5.8+ <[email protected]> 1 v5.7+ <[email protected]> 1 v5.6+ <[email protected]> 1 v5.3+ <[email protected]> 1 v5.0+ <[email protected]> 1 v4.9+ <[email protected]> 1 <[email protected]> v5.0+ 1 <[email protected]> +v4.18 1 [email protected] (3.14+) 1 5.8+ <[email protected]> 1 5.5+ <[email protected]> 1 5.0+ <[email protected]> 1 4.18+ <[email protected]> 1 4.14+ <[email protected]> 1 4.13+ <[email protected]> 1 4.0+ <[email protected]> 1 3.15+ <[email protected]> 1 3.11+ <[email protected]>
Quoted 9 lines
Improvements to parsing:
- Detect and report unexpected content after email.
- Quoted names are excluded from comment parsing.
- Trailing dots or commas in email are removed during formatting. Correspondingly aBAD_SIGN_OFFwarning is emitted.
- Improperly quoted email like '"name <address>"' are now warned about.
All of the above seems right but perhaps the comment style for any <foo>-by: lines should also allow # comments.
The below is just comments on the patch itself.
scripts/checkpatch.pl
Quoted 1 line
scripts/checkpatch.pl +0
-0
[]
scripts/checkpatch.pl sub process { · line 2812
Quoted 9 lines
scripts/checkpatch.pl +4
-1@@ -2800,9 +2806,57 @@ sub process { $dequoted =~ s/" </ </; # Don't force email to have quotes # Allow just an angle bracketed address - if (!same_email_addresses($email, $suggested_email, 0)) { + if (!same_email_addresses($email, $suggested_email)) { + if (WARN("BAD_SIGN_OFF", + "email address '$email' might be better as '$suggested_email'\n" . $herecurr) && + $fix) {
trivia:
Please always align $fix with tabs to the if and then 4 spaces to the open parenthesis.
scripts/checkpatch.pl sub process { · line 2837
Quoted 3 lines
scripts/checkpatch.pl +3
-0+ # Comments must begin only with ( + # or # in case of [email protected] + if ($email =~ /^.*stable\@vger/) {
I believe this should be
if ($email =~ /^stable\@(?:vger\.)?kernel.org$/) {scripts/checkpatch.pl sub process { · line 2840
Quoted 3 lines
scripts/checkpatch.pl +3
-0+ if ($comment ne "" && $comment !~ /^#.+/) { + if (WARN("BAD_SIGN_OFF", + "Invalid comment format for stable: '$email', prefer parentheses\n" . $herecurr) &&
Prefer #
scripts/checkpatch.pl sub process { · line 2843
Quoted 3 lines
scripts/checkpatch.pl +3
-0+ $fix) { + my $new_comment = $comment; + $new_comment =~ s/^[ \(\[]+|[ \)\]]+$//g;
Does the comment include any leading whitespace here? I presumed not given the $comment !~ /^#/ test above.
Greg KH
maintainerQuoted 26 lines · Joe Perches
(adding stable and Greg KH for additional review)
Quoted 3 lines · Dwaipayan Ray
checkpatch doesn't report warnings for many common mistakes in emails. Some of which are trailing commas and incorrect use of email comments.
I presume you've tested this against the git tree.
Can you send me a file with the
BAD_SIGN_OFFmessages generated and if possible the git SHA-1s of the commits?Quoted 6 lines
At the same time several false positives are reported due to incorrect handling of mail comments. The most common of which is due to the pattern:
<[email protected]> # X.X
Quoted 8 lines
Improve email parsing in checkpatch.
Some general comment rules are defined:
- Multiple name comments should not be allowed.
- Comments inside address should not be allowed.
- In general comments should be enclosed within parentheses. Exception for [email protected] # X.X
Quoted 8 lines
not just vger.kernel.org, but this should also allow [email protected] and only allow cc: and not any other -by: type for that email address.
A process preference question for Greg and the stable team:
The most common stable forms are
[email protected] # version info
That is what is documented it should be, yes.
Quoted 2 lines
then
[email protected] [ version info ]
Really? Ick, no wonder my email parsing scripts choke on that :)
Quoted 2 lines
with some other relatively infrequently used outlier styles, some that use parentheses, but this is not frequent.
The one with '#' should be preferred. If not, we need to change our documentation.
thanks,
greg k-h
A message that is not in the archive (sent off-list or not yet received): CABJPP5BTRtAjdNQdRCx-3HjmyJ0AnoXBzB95YiPRMoF0njOOaA@mail.gmail.com
Joe Perches
maintainerQuoted 4 lines · Dwaipayan Ray
Quoted 2 lines
Can you send me a file with the
BAD_SIGN_OFFmessages generated and if possible the git SHA-1s of the commits?Yes sure, am attaching the file for data tested on about 27k commits from v5.4.
Thanks.
Quoted 11 lines
Excluded the duplicate signatures, tags and spacing ones for simplicity.
Quoted 6 lines
24 linux-stable <[email protected]>
21 5.4+ <[email protected]>
14 All applicable <[email protected]>
6 3.10+ <[email protected]>
5 5.9+ <[email protected]>
5 5.3+ <[email protected]>Do I also convert these then for the fix?
Yes.
Ideally removing any case insensitive name like linux-stable or stable and also removing any leading < or trailing > from the email address.
And the stable email address should be forced to lower case only.
Any other name should be moved as a comment after the email address then #.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Improvements to parsing:
- Detect and report unexpected content after email.
- Quoted names are excluded from comment parsing.
- Trailing dots or commas in email are removed during formatting. Correspondingly aBAD_SIGN_OFFwarning is emitted.
- Improperly quoted email like '"name <address>"' are now warned about.
[]
Quoted 10 lines
Quoted 6 lines
scripts/checkpatch.pl sub process { · line 2843
Quoted 2 lines
scripts/checkpatch.pl +2
-0+ my $new_comment = $comment; + $new_comment =~ s/^[ \(\[]+|[ \)\]]+$//g;Does the comment include any leading whitespace here? I presumed not given the $comment !~ /^#/ test above.
I added it to discard empty spaces before or after the first brackets are unwrapped. Something like ( v5.7 ) would
be better if translated to # v5.7. (The extra spaces would be purged). It just looks good :). Should I change it to keep it as it is?
No. Converting the comment style is very sensible and a good change. I did not understand why the regex included a space. I do now. I suggest changing the space to use \s in case there are tabs.