Skip to contentSkip to conversation

checkpatch: improve email parsing

PATCH Superseded Dwaipayan Ray 3 participants 4 messages Nov 5, 2020 – Nov 6, 2020 1 message not in the archive

Subsystems (2)

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 checkpatch: improve email parsing

checkpatch doesn't report warnings for many common mistakes in emails. Some of which are trailing commas and incorrect use of email comments.

At the same time several false positives are reported due to incorrect handling of mail comments. The most common of which is due to the pattern:

<[email protected]> # X.X

Improve email parsing in checkpatch.

Some general comment rules are defined:

- Multiple name comments should not be allowed.
- Comments inside address should not be allowed.
- In general comments should be enclosed within parentheses. Exception for [email protected] # X.X

Improvements to parsing:

- Detect and report unexpected content after email.
- Quoted names are excluded from comment parsing.
- Trailing dots or commas in email are removed during formatting. Correspondingly a BAD_SIGN_OFF warning is emitted.
- Improperly quoted email like '"name <address>"' are now warned about.

In addition, added fixes for all the possible rules.

---
 scripts/checkpatch.pl | 88 ++++++++++++++++++++++++++++++++++---------
 1 file changed, 71 insertions(+), 17 deletions(-)
scripts/checkpatch.pl +71 -17
index fab38b493cef..dc8b664b7de1 100755
@@ -1152,6 +1152,7 @@ sub parse_email {
 	my ($formatted_email) = @_;
 
 	my $name = "";
+	my $quoted = "";
 	my $name_comment = "";
 	my $address = "";
 	my $comment = "";
@@ -1183,14 +1184,20 @@ sub parse_email {
 		}
 	}
 
-	$comment = trim($comment);
-	$name = trim($name);
-	$name =~ s/^\"|\"$//g;
-	if ($name =~ s/(\s*\([^\)]+\))\s*//) {
-		$name_comment = trim($1);
+	# Extract comments from names excluding quoted parts
+	# "John A. (Kennedy)" - Do not extract
+	if ($name =~ s/\"(.+)\"//) {
+		$quoted = $1;
+	}
+	while ($name =~ s/\s*($balanced_parens)\s*/ /) {
+		$name_comment .= trim($1);
 	}
+	$name =~ s/^[ \"]+|[ \"]+$//g;
+	$name = trim("$quoted $name");
+
 	$address = trim($address);
 	$address =~ s/^\<|\>$//g;
+	$comment = trim($comment);
 
 	if ($name =~ /[^\w \-]/i) { ##has "must quote" chars
 		$name =~ s/(?<!\\)"/\\"/g; ##escape quotes
@@ -1205,17 +1212,20 @@ sub format_email {
 
 	my $formatted_email;
 
-	$name_comment = trim($name_comment);
-	$comment = trim($comment);
-	$name = trim($name);
-	$name =~ s/^\"|\"$//g;
+	$name =~ s/^[ \"]+|[ \"]+$//g;
 	$address = trim($address);
+	$address =~ s/(?:\.|\,)+$//; ##trailing commas or dots
 
 	if ($name =~ /[^\w \-]/i) { ##has "must quote" chars
 		$name =~ s/(?<!\\)"/\\"/g; ##escape quotes
 		$name = "\"$name\"";
 	}
 
+	$name_comment = trim($name_comment);
+	$name_comment = " $name_comment" if length($name_comment) > 0;
+	$comment = trim($comment);
+	$comment = " $comment" if length($comment) > 0;
+
 	if ("$name" eq "") {
 		$formatted_email = "$address";
 	} else {
@@ -1233,15 +1243,11 @@ sub reformat_email {
 }
 
 sub same_email_addresses {
-	my ($email1, $email2, $match_comment) = @_;
+	my ($email1, $email2) = @_;
 
 	my ($email1_name, $name1_comment, $email1_address, $comment1) = parse_email($email1);
 	my ($email2_name, $name2_comment, $email2_address, $comment2) = parse_email($email2);
 
-	if ($match_comment != 1) {
-		return $email1_name eq $email2_name &&
-		       $email1_address eq $email2_address;
-	}
 	return $email1_name eq $email2_name &&
 	       $email1_address eq $email2_address &&
 	       $name1_comment eq $name2_comment &&
@@ -2704,7 +2710,7 @@ sub process {
 			$signoff++;
 			$in_commit_log = 0;
 			if ($author ne ''  && $authorsignoff != 1) {
-				if (same_email_addresses($1, $author, 1)) {
+				if (same_email_addresses($1, $author)) {
 					$authorsignoff = 1;
 				} else {
 					my $ctx = $1;
@@ -2800,9 +2806,57 @@ sub process {
 				$dequoted =~ s/" </ </;
 				# Don't force email to have quotes
 				# Allow just an angle bracketed address
-				if (!same_email_addresses($email, $suggested_email, 0)) {
+				if (!same_email_addresses($email, $suggested_email)) {
+					if (WARN("BAD_SIGN_OFF",
+					    "email address '$email' might be better as '$suggested_email'\n" . $herecurr) &&
+						$fix) {
+						$fixed[$fixlinenr] =~ s/\Q$email\E/$suggested_email/;
+					}
+				}
+
+				# Address part shouldn't have comments
+				my $stripped_address = $email_address;
+				$stripped_address =~ s/\([^\(\)]*\)//g;
+				if ($email_address ne $stripped_address) {
+					if (WARN("BAD_SIGN_OFF",
+					    "address part of email should not have comments: '$email_address'\n" . $herecurr) &&
+						$fix) {
+						$fixed[$fixlinenr] =~ s/\Q$email_address\E/$stripped_address/;
+					}
+				}
+
+				# Only one name comment should be allowed
+				my $comment_count = () = $name_comment =~ /\([^\)]+\)/g;
+				if ($comment_count > 1) {
 					WARN("BAD_SIGN_OFF",
-					     "email address '$email' might be better as '$suggested_email'\n" . $herecurr);
+					     "Use a single name comment in email: '$email'\n" . $herecurr);
+				}
+
+				# Comments must begin only with (
+				# or # in case of [email protected]
+				if ($email =~ /^.*stable\@vger/) {
+					if ($comment ne "" && $comment !~ /^#.+/) {
+						if (WARN("BAD_SIGN_OFF",
+						    "Invalid comment format for stable: '$email', prefer parentheses\n" . $herecurr) &&
+							$fix) {
+							my $new_comment = $comment;
+							$new_comment =~ s/^[ \(\[]+|[ \)\]]+$//g;
+							$new_comment = " # $new_comment" if length($new_comment) > 0;
+							$fixed[$fixlinenr] =~ s/\s*\Q$comment\E$/$new_comment/;
+						}
+					}
+				} else {
+					if ($comment ne "" && $comment !~ /^\(.+\)$/) {
+						if (WARN("BAD_SIGN_OFF",
+						    "Unexpected content after email: '$email'\n" . $herecurr) &&
+							$fix) {
+							my $new_comment = $comment;
+							$new_comment =~ s/^(?:\#|\/\*|\.|\,)//g;
+							$new_comment =~ s/^[ \{\[]+|[ \}\]]+$//g;
+							$new_comment = " ($new_comment)" if length($new_comment) > 0;
+							$fixed[$fixlinenr] =~ s/\s*\Q$comment\E$/$new_comment/;
+						}
+					}
 				}
 			}
 
Signature · 1 line
2.27.0
maintainer
#1 lore raw

(adding stable and Greg KH for additional review)

Quoted 3 lines · Dwaipayan Ray

checkpatch doesn't report warnings for many common mistakes in emails. Some of which are trailing commas and incorrect use of email comments.

I presume you've tested this against the git tree.

Can you send me a file with the BAD_SIGN_OFF messages generated and if possible the git SHA-1s of the commits?

Quoted 6 lines

At the same time several false positives are reported due to incorrect handling of mail comments. The most common of which is due to the pattern:

<[email protected]> # X.X

Quoted 8 lines

Improve email parsing in checkpatch.

Some general comment rules are defined:

- Multiple name comments should not be allowed.
- Comments inside address should not be allowed.
- In general comments should be enclosed within parentheses. Exception for [email protected] # X.X

not just vger.kernel.org, but this should also allow [email protected] and only allow cc: and not any other -by: type for that email address.

A process preference question for Greg and the stable team:

The most common stable forms are

[email protected] # version info
then
[email protected] [ version info ]

with some other relatively infrequently used outlier styles, some that use parentheses, but this is not frequent.

It might be sensible to standardize on the "# version info" trailer comment version info style and warn on any other form.

A somewhat common style for the stable address is to use a name before the stable address which describes the version info:

Perhaps any name before stable should be warned and the version should be a comment.

Here's a list of the stable addresses with "version name" then stable address in the git tree and other outlier styles.

     24 linux-stable <[email protected]>
     21 5.4+ <[email protected]>
     14 All applicable <[email protected]>
      6 3.10+ <[email protected]>
      5 5.9+ <[email protected]>
      5 5.3+ <[email protected]>
      5 5.1+ <[email protected]>
      4 5.6+ <[email protected]>
      4 4.20+ <[email protected]>
      3 Stable Team <[email protected]>
      3 4.19+ <[email protected]>
      3 4.15+ <[email protected]>
      3 4.10+ <[email protected]>
      2 [email protected] (v2.6.12+)
      2 5.2+ <[email protected]>
      2 4.16+ <[email protected]>
      1 v5.8+ <[email protected]>
      1 v5.7+ <[email protected]>
      1 v5.6+ <[email protected]>
      1 v5.3+ <[email protected]>
      1 v5.0+ <[email protected]>
      1 v4.9+ <[email protected]>
      1 <[email protected]> v5.0+
      1 <[email protected]> +v4.18
      1 [email protected] (3.14+)
      1 5.8+ <[email protected]>
      1 5.5+ <[email protected]>
      1 5.0+ <[email protected]>
      1 4.18+ <[email protected]>
      1 4.14+ <[email protected]>
      1 4.13+ <[email protected]>
      1 4.0+ <[email protected]>
      1 3.15+ <[email protected]>
      1 3.11+ <[email protected]>
Quoted 9 lines

Improvements to parsing:

- Detect and report unexpected content after email.
- Quoted names are excluded from comment parsing.
- Trailing dots or commas in email are removed during formatting. Correspondingly a BAD_SIGN_OFF warning is emitted.
- Improperly quoted email like '"name <address>"' are now warned about.

All of the above seems right but perhaps the comment style for any <foo>-by: lines should also allow # comments.

The below is just comments on the patch itself.

scripts/checkpatch.pl

Quoted 1 line
scripts/checkpatch.pl +0 -0

[]

scripts/checkpatch.pl sub process { · line 2812

Quoted 9 lines
scripts/checkpatch.pl +4 -1
@@ -2800,9 +2806,57 @@ sub process {
  				$dequoted =~ s/" </ </;
  				# Don't force email to have quotes
  				# Allow just an angle bracketed address
-				if (!same_email_addresses($email, $suggested_email, 0)) {
+				if (!same_email_addresses($email, $suggested_email)) {
+					if (WARN("BAD_SIGN_OFF",
+					    "email address '$email' might be better as '$suggested_email'\n" . $herecurr) &&
+						$fix) {

trivia:

Please always align $fix with tabs to the if and then 4 spaces to the open parenthesis.

scripts/checkpatch.pl sub process { · line 2837

Quoted 3 lines
scripts/checkpatch.pl +3 -0
+				# Comments must begin only with (
+				# or # in case of [email protected]
+				if ($email =~ /^.*stable\@vger/) {

I believe this should be

				if ($email =~ /^stable\@(?:vger\.)?kernel.org$/) {

scripts/checkpatch.pl sub process { · line 2840

Quoted 3 lines
scripts/checkpatch.pl +3 -0
+					if ($comment ne "" && $comment !~ /^#.+/) {
+						if (WARN("BAD_SIGN_OFF",
+						    "Invalid comment format for stable: '$email', prefer parentheses\n" . $herecurr) &&

Prefer #

scripts/checkpatch.pl sub process { · line 2843

Quoted 3 lines
scripts/checkpatch.pl +3 -0
+							$fix) {
+							my $new_comment = $comment;
+							$new_comment =~ s/^[ \(\[]+|[ \)\]]+$//g;

Does the comment include any leading whitespace here? I presumed not given the $comment !~ /^#/ test above.

maintainer
#2 lore raw
Quoted 26 lines · Joe Perches

(adding stable and Greg KH for additional review)

Quoted 3 lines · Dwaipayan Ray

checkpatch doesn't report warnings for many common mistakes in emails. Some of which are trailing commas and incorrect use of email comments.

I presume you've tested this against the git tree.

Can you send me a file with the BAD_SIGN_OFF messages generated and if possible the git SHA-1s of the commits?

Quoted 6 lines

At the same time several false positives are reported due to incorrect handling of mail comments. The most common of which is due to the pattern:

<[email protected]> # X.X

Quoted 8 lines

Improve email parsing in checkpatch.

Some general comment rules are defined:

- Multiple name comments should not be allowed.
- Comments inside address should not be allowed.
- In general comments should be enclosed within parentheses. Exception for [email protected] # X.X

Quoted 8 lines

not just vger.kernel.org, but this should also allow [email protected] and only allow cc: and not any other -by: type for that email address.

A process preference question for Greg and the stable team:

The most common stable forms are

[email protected] # version info

That is what is documented it should be, yes.

Quoted 2 lines

then
[email protected] [ version info ]

Really? Ick, no wonder my email parsing scripts choke on that :)

Quoted 2 lines

with some other relatively infrequently used outlier styles, some that use parentheses, but this is not frequent.

The one with '#' should be preferred. If not, we need to change our documentation.

thanks,

greg k-h

A message that is not in the archive (sent off-list or not yet received): CABJPP5BTRtAjdNQdRCx-3HjmyJ0AnoXBzB95YiPRMoF0njOOaA@mail.gmail.com

maintainer
#4 lore raw
Quoted 4 lines · Dwaipayan Ray
Quoted 2 lines

Can you send me a file with the BAD_SIGN_OFF messages generated and if possible the git SHA-1s of the commits?

Yes sure, am attaching the file for data tested on about 27k commits from v5.4.

Thanks.

Quoted 11 lines

Excluded the duplicate signatures, tags and spacing ones for simplicity.

Quoted 6 lines

     24 linux-stable <[email protected]>
     21 5.4+ <[email protected]>
     14 All applicable <[email protected]>
      6 3.10+ <[email protected]>
      5 5.9+ <[email protected]>
      5 5.3+ <[email protected]>

Do I also convert these then for the fix?

Yes.

Ideally removing any case insensitive name like linux-stable or stable and also removing any leading < or trailing > from the email address.

And the stable email address should be forced to lower case only.

Any other name should be moved as a comment after the email address then #.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

Improvements to parsing:

- Detect and report unexpected content after email.
- Quoted names are excluded from comment parsing.
- Trailing dots or commas in email are removed during formatting. Correspondingly a BAD_SIGN_OFF warning is emitted.
- Improperly quoted email like '"name <address>"' are now warned about.

[]

Quoted 10 lines
Quoted 6 lines

scripts/checkpatch.pl sub process { · line 2843

Quoted 2 lines
scripts/checkpatch.pl +2 -0
+                                                     my $new_comment = $comment;
+                                                     $new_comment =~ s/^[ \(\[]+|[ \)\]]+$//g;

Does the comment include any leading whitespace here? I presumed not given the $comment !~ /^#/ test above.

I added it to discard empty spaces before or after the first brackets are unwrapped. Something like ( v5.7 ) would
be better if translated to # v5.7. (The extra spaces would be purged). It just looks good :). Should I change it to keep it as it is?

No. Converting the comment style is very sensible and a good change. I did not understand why the regex included a space. I do now. I suggest changing the space to use \s in case there are tabs.