checkpatch: improve handling of email comments
RFC PATCH 1/1 checkpatch: improve handling of email comments
checkpatch has limited support for parsing email comments. It only support single name comments or single after address comments. Whereas, RFC 5322 specifies that comments can be inserted in between any tokens of the email fields.
On analyzing 50,000 commits from v5.4 it was found that there were about 370 false positives resulting from wrong parsing of comments.
Improve comment parsing mechanism in checkpatch.
What is handled now:
- Multiple name/address comments
- Comments anywhere in between name/address
- Multi level comments like (John (Doe) )
--- scripts/checkpatch.pl | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-)
scripts/checkpatch.pl +14 -5
@@ -1183,14 +1183,20 @@ sub parse_email { } } - $comment = trim($comment); + # Comments in between name like John(A nice chap) Doe + while ($name =~ s/\s*($balanced_parens)\s*/ /) { + $name_comment .= trim($1); + } $name = trim($name); $name =~ s/^\"|\"$//g; - if ($name =~ s/(\s*\([^\)]+\))\s*//) { - $name_comment = trim($1); + + # Comments in between address like <john(his account)@doe.com> + while ($address =~ s/\s*($balanced_parens)\s*//) { + $comment .= trim($1); } $address = trim($address); $address =~ s/^\<|\>$//g; + $comment = trim($comment); if ($name =~ /[^\w \-]/i) { ##has "must quote" chars $name =~ s/(?<!\\)"/\\"/g; ##escape quotes
@@ -1205,8 +1211,6 @@ sub format_email { my $formatted_email; - $name_comment = trim($name_comment); - $comment = trim($comment); $name = trim($name); $name =~ s/^\"|\"$//g; $address = trim($address);
@@ -1216,6 +1220,11 @@ sub format_email { $name = "\"$name\""; } + $name_comment = trim($name_comment); + $name_comment =~ s/(.+)/ $1/; + $comment = trim($comment); + $comment =~ s/(.+)/ $1/; + if ("$name" eq "") { $formatted_email = "$address"; } else {
Signature · 6 lines
2.27.0 _______________________________________________ Linux-kernel-mentees mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/linux-kernel-mentees
Quoted 72 lines · Dwaipayan Ray
checkpatch has limited support for parsing email comments. It only support single name comments or single after address comments. Whereas, RFC 5322 specifies that comments can be inserted in between any tokens of the email fields.
On analyzing 50,000 commits from v5.4 it was found that there were about 370 false positives resulting from wrong parsing of comments.
Improve comment parsing mechanism in checkpatch.
What is handled now:
- Multiple name/address comments
- Comments anywhere in between name/address
- Multi level comments like (John (Doe) )Signed-off-by Dwaipayan Ray <[email protected]>--- scripts/checkpatch.pl | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-)scripts/checkpatch.pl +14
-6@@ -1183,14 +1183,20 @@ sub parse_email { } } - $comment = trim($comment); + # Comments in between name like John(A nice chap) Doe + while ($name =~ s/\s*($balanced_parens)\s*/ /) { + $name_comment .= trim($1); + } $name = trim($name); $name =~ s/^\"|\"$//g; - if ($name =~ s/(\s*\([^\)]+\))\s*//) { - $name_comment = trim($1); + + # Comments in between address like <john(his account)@doe.com> + while ($address =~ s/\s*($balanced_parens)\s*//) { + $comment .= trim($1); } $address = trim($address); $address =~ s/^\<|\>$//g; + $comment = trim($comment); if ($name =~ /[^\w \-]/i) { ##has "must quote" chars $name =~ s/(?<!\\)"/\\"/g; ##escape quotes@@ -1205,8 +1211,6 @@ sub format_email { my $formatted_email; - $name_comment = trim($name_comment); - $comment = trim($comment); $name = trim($name); $name =~ s/^\"|\"$//g; $address = trim($address);@@ -1216,6 +1220,11 @@ sub format_email { $name = "\"$name\""; } + $name_comment = trim($name_comment); + $name_comment =~ s/(.+)/ $1/; + $comment = trim($comment); + $comment =~ s/(.+)/ $1/; + if ("$name" eq "") { $formatted_email = "$address"; } else { --
Quoted 2 lines
2.27.0
Hi,
This patch is a follow up to our discussion earlier about improving the comment handling at:
https://lore.kernel.org/linux-kernel-mentees/alpine.DEB.2.21.2010251022060.25172@felia/
I have only tested it on a few patterns. But I will run an evaluation again on those 50k commits to see what changes.
What do you think of this?
Thanks,
Dwaipayan.
_______________________________________________
Linux-kernel-mentees mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/linux-kernel-mentees
Quoted 81 lines · Dwaipayan Ray
Quoted 72 lines · Dwaipayan Ray
checkpatch has limited support for parsing email comments. It only support single name comments or single after address comments. Whereas, RFC 5322 specifies that comments can be inserted in between any tokens of the email fields.
On analyzing 50,000 commits from v5.4 it was found that there were about 370 false positives resulting from wrong parsing of comments.
Improve comment parsing mechanism in checkpatch.
What is handled now:
- Multiple name/address comments
- Comments anywhere in between name/address
- Multi level comments like (John (Doe) )Signed-off-by Dwaipayan Ray <[email protected]>--- scripts/checkpatch.pl | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-)scripts/checkpatch.pl +14
-6@@ -1183,14 +1183,20 @@ sub parse_email { } } - $comment = trim($comment); + # Comments in between name like John(A nice chap) Doe + while ($name =~ s/\s*($balanced_parens)\s*/ /) { + $name_comment .= trim($1); + } $name = trim($name); $name =~ s/^\"|\"$//g; - if ($name =~ s/(\s*\([^\)]+\))\s*//) { - $name_comment = trim($1); + + # Comments in between address like <john(his account)@doe.com> + while ($address =~ s/\s*($balanced_parens)\s*//) { + $comment .= trim($1); } $address = trim($address); $address =~ s/^\<|\>$//g; + $comment = trim($comment); if ($name =~ /[^\w \-]/i) { ##has "must quote" chars $name =~ s/(?<!\\)"/\\"/g; ##escape quotes@@ -1205,8 +1211,6 @@ sub format_email { my $formatted_email; - $name_comment = trim($name_comment); - $comment = trim($comment); $name = trim($name); $name =~ s/^\"|\"$//g; $address = trim($address);@@ -1216,6 +1220,11 @@ sub format_email { $name = "\"$name\""; } + $name_comment = trim($name_comment); + $name_comment =~ s/(.+)/ $1/; + $comment = trim($comment); + $comment =~ s/(.+)/ $1/; + if ("$name" eq "") { $formatted_email = "$address"; } else { --Quoted 2 lines
2.27.0
Hi,
This patch is a follow up to our discussion earlier about improving the comment handling at:
https://lore.kernel.org/linux-kernel-mentees/alpine.DEB.2.21.2010251022060.25172@felia/
Quoted 5 lines
I have only tested it on a few patterns. But I will run an evaluation again on those 50k commits to see what changes.
What do you think of this?
That was exactly the comment I wanted to make.
Commit message is clear; code addition also looks pretty clear.
We are just missing the evaluation to make sure how much we improve and that we did not add some stupid bug on the way.
Aditya, can you help us here with evaluation?
E.g., Dwaipayan takes from v5.4 into the future, e.g., until v5.8/v5.9 or so.
Aditya, you could then check e.g., v5.0..v5.4, so v5.4 down to the past as far as your computer and scripts handles within roughly half a day...
An evaluation on 100,000 commits is certainly a good basis.
Of course, every mentorship candidate can join here as well and show off their own evaluation script.
Lukas
_______________________________________________
Linux-kernel-mentees mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/linux-kernel-mentees
Quoted 81 lines · Dwaipayan Ray
Quoted 72 lines · Dwaipayan Ray
checkpatch has limited support for parsing email comments. It only support single name comments or single after address comments. Whereas, RFC 5322 specifies that comments can be inserted in between any tokens of the email fields.
On analyzing 50,000 commits from v5.4 it was found that there were about 370 false positives resulting from wrong parsing of comments.
Improve comment parsing mechanism in checkpatch.
What is handled now:
- Multiple name/address comments
- Comments anywhere in between name/address
- Multi level comments like (John (Doe) )Signed-off-by Dwaipayan Ray <[email protected]>--- scripts/checkpatch.pl | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-)scripts/checkpatch.pl +14
-6@@ -1183,14 +1183,20 @@ sub parse_email { } } - $comment = trim($comment); + # Comments in between name like John(A nice chap) Doe + while ($name =~ s/\s*($balanced_parens)\s*/ /) { + $name_comment .= trim($1); + } $name = trim($name); $name =~ s/^\"|\"$//g; - if ($name =~ s/(\s*\([^\)]+\))\s*//) { - $name_comment = trim($1); + + # Comments in between address like <john(his account)@doe.com> + while ($address =~ s/\s*($balanced_parens)\s*//) { + $comment .= trim($1); } $address = trim($address); $address =~ s/^\<|\>$//g; + $comment = trim($comment); if ($name =~ /[^\w \-]/i) { ##has "must quote" chars $name =~ s/(?<!\\)"/\\"/g; ##escape quotes@@ -1205,8 +1211,6 @@ sub format_email { my $formatted_email; - $name_comment = trim($name_comment); - $comment = trim($comment); $name = trim($name); $name =~ s/^\"|\"$//g; $address = trim($address);@@ -1216,6 +1220,11 @@ sub format_email { $name = "\"$name\""; } + $name_comment = trim($name_comment); + $name_comment =~ s/(.+)/ $1/; + $comment = trim($comment); + $comment =~ s/(.+)/ $1/; + if ("$name" eq "") { $formatted_email = "$address"; } else { --Quoted 2 lines
2.27.0
Hi,
This patch is a follow up to our discussion earlier about improving the comment handling at:
https://lore.kernel.org/linux-kernel-mentees/alpine.DEB.2.21.2010251022060.25172@felia/
Quoted 5 lines
I have only tested it on a few patterns. But I will run an evaluation again on those 50k commits to see what changes.
What do you think of this?
That was exactly the comment I wanted to make.
Commit message is clear; code addition also looks pretty clear.
We are just missing the evaluation to make sure how much we improve and that we did not add some stupid bug on the way.
Aditya, can you help us here with evaluation?
E.g., Dwaipayan takes from v5.4 into the future, e.g., until v5.8/v5.9 or so.
Aditya, you could then check e.g., v5.0..v5.4, so v5.4 down to the past as far as your computer and scripts handles within roughly half a day...
An evaluation on 100,000 commits is certainly a good basis.
Of course, every mentorship candidate can join here as well and show off their own evaluation script.
Lukas
_______________________________________________
Linux-kernel-mentees mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/linux-kernel-mentees
Quoted 106 lines · Lukas Bulwahn
Quoted 81 lines · Dwaipayan Ray
Quoted 72 lines · Dwaipayan Ray
checkpatch has limited support for parsing email comments. It only support single name comments or single after address comments. Whereas, RFC 5322 specifies that comments can be inserted in between any tokens of the email fields.
On analyzing 50,000 commits from v5.4 it was found that there were about 370 false positives resulting from wrong parsing of comments.
Improve comment parsing mechanism in checkpatch.
What is handled now:
- Multiple name/address comments
- Comments anywhere in between name/address
- Multi level comments like (John (Doe) )Signed-off-by Dwaipayan Ray <[email protected]>--- scripts/checkpatch.pl | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-)scripts/checkpatch.pl +14
-6@@ -1183,14 +1183,20 @@ sub parse_email { } } - $comment = trim($comment); + # Comments in between name like John(A nice chap) Doe + while ($name =~ s/\s*($balanced_parens)\s*/ /) { + $name_comment .= trim($1); + } $name = trim($name); $name =~ s/^\"|\"$//g; - if ($name =~ s/(\s*\([^\)]+\))\s*//) { - $name_comment = trim($1); + + # Comments in between address like <john(his account)@doe.com> + while ($address =~ s/\s*($balanced_parens)\s*//) { + $comment .= trim($1); } $address = trim($address); $address =~ s/^\<|\>$//g; + $comment = trim($comment); if ($name =~ /[^\w \-]/i) { ##has "must quote" chars $name =~ s/(?<!\\)"/\\"/g; ##escape quotes@@ -1205,8 +1211,6 @@ sub format_email { my $formatted_email; - $name_comment = trim($name_comment); - $comment = trim($comment); $name = trim($name); $name =~ s/^\"|\"$//g; $address = trim($address);@@ -1216,6 +1220,11 @@ sub format_email { $name = "\"$name\""; } + $name_comment = trim($name_comment); + $name_comment =~ s/(.+)/ $1/; + $comment = trim($comment); + $comment =~ s/(.+)/ $1/; + if ("$name" eq "") { $formatted_email = "$address"; } else { --Quoted 2 lines
2.27.0
Hi,
This patch is a follow up to our discussion earlier about improving the comment handling at:
https://lore.kernel.org/linux-kernel-mentees/alpine.DEB.2.21.2010251022060.25172@felia/Quoted 5 lines
I have only tested it on a few patterns. But I will run an evaluation again on those 50k commits to see what changes.
What do you think of this?
That was exactly the comment I wanted to make.
Commit message is clear; code addition also looks pretty clear.
We are just missing the evaluation to make sure how much we improve and that we did not add some stupid bug on the way.
Aditya, can you help us here with evaluation?
E.g., Dwaipayan takes from v5.4 into the future, e.g., until v5.8/v5.9 or so.
Aditya, you could then check e.g., v5.0..v5.4, so v5.4 down to the past as far as your computer and scripts handles within roughly half a day...
Quoted 7 lines
An evaluation on 100,000 commits is certainly a good basis.
Of course, every mentorship candidate can join here as well and show off their own evaluation script.
Lukas
Sure, would love to help.
Just to clear, I have to apply this patch over my last patch right? Or do I need to apply any more patch before it?
Thanks
Aditya
_______________________________________________
Linux-kernel-mentees mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/linux-kernel-mentees
Quoted 13 lines
Quoted 13 lines
Aditya, can you help us here with evaluation?
E.g., Dwaipayan takes from v5.4 into the future, e.g., until v5.8/v5.9 or so.
Aditya, you could then check e.g., v5.0..v5.4, so v5.4 down to the past as far as your computer and scripts handles within roughly half a day...
An evaluation on 100,000 commits is certainly a good basis.
Of course, every mentorship candidate can join here as well and show off their own evaluation script.
Quoted 7 lines
Quoted 2 lines
Lukas
Sure, would love to help.
Just to clear, I have to apply this patch over my last patch right? Or do I need to apply any more patch before it?
Thanks Aditya.
I think you can apply the patch directly on the latest official tree.
The patch is here:
https://lore.kernel.org/linux-kernel-mentees/[email protected]/
Regards,
Dwaipayan.
_______________________________________________
Linux-kernel-mentees mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/linux-kernel-mentees
Quoted 23 lines · Dwaipayan Ray
Quoted 13 lines
Quoted 13 lines
Aditya, can you help us here with evaluation?
E.g., Dwaipayan takes from v5.4 into the future, e.g., until v5.8/v5.9 or so.
Aditya, you could then check e.g., v5.0..v5.4, so v5.4 down to the past as far as your computer and scripts handles within roughly half a day...
An evaluation on 100,000 commits is certainly a good basis.
Of course, every mentorship candidate can join here as well and show off their own evaluation script.
Quoted 7 lines
Quoted 2 lines
Lukas
Sure, would love to help.
Just to clear, I have to apply this patch over my last patch right? Or do I need to apply any more patch before it?Thanks Aditya.
Quoted 8 lines
I think you can apply the patch directly on the latest official tree.
The patch is here:
https://lore.kernel.org/linux-kernel-mentees/[email protected]/Regards,
Dwaipayan.
Hi Dwaipayan,
I have generated reports on v5.0..v5.4.
Relative change in warnings/errors before/after your patch:
https://github.com/AdityaSrivast/kernel-tasks/blob/master/Task4/relative_summary/summary_relative.txt
Dropped warnings:
https://github.com/AdityaSrivast/kernel-tasks/tree/master/Task4/relative_summary/dropped_warnings
There may be some discrepancies due to multi-threading. But it should give us a rough idea.
Thanks
Aditya
_______________________________________________
Linux-kernel-mentees mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/linux-kernel-mentees
Quoted 4 lines
Hi Dwaipayan,
I have generated reports on v5.0..v5.4.
Quoted 9 lines
Relative change in warnings/errors before/after your patch:
https://github.com/AdityaSrivast/kernel-tasks/blob/master/Task4/relative_summary/summary_relative.txtDropped warnings:
https://github.com/AdityaSrivast/kernel-tasks/tree/master/Task4/relative_summary/dropped_warningsThere may be some discrepancies due to multi-threading. But it should give us a rough idea.
Thanks Aditya.
Seems like 763 false positives were resolved successfully. So I guess the patch is working as intended.
My evaluation is currently running and might need a few hours more. I will share it once it completes.
Thanks,
Dwaipayan.
_______________________________________________
Linux-kernel-mentees mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/linux-kernel-mentees
Quoted 15 lines · Dwaipayan Ray
Quoted 5 lines
Hi Dwaipayan,
I have generated reports on v5.0..v5.4.Quoted 9 lines
Relative change in warnings/errors before/after your patch:
https://github.com/AdityaSrivast/kernel-tasks/blob/master/Task4/relative_summary/summary_relative.txtDropped warnings:
https://github.com/AdityaSrivast/kernel-tasks/tree/master/Task4/relative_summary/dropped_warningsThere may be some discrepancies due to multi-threading. But it should give us a rough idea.
Quoted 8 lines
Thanks Aditya.
Seems like 763 false positives were resolved successfully. So I guess the patch is working as intended.
My evaluation is currently running and might need a few hours more. I will share it once it completes.
The summary of the analysis is as follows:
Commits checked: 47, 377 from v 5.4
Before the patch:BAD_SIGN_OFF: 1516FROM_SIGN_OFF_MISMATCH: 931NO_AUTHOR_SIGN_OFF: 54
After the patch:BAD_SIGN_OFF: 756FROM_SIGN_OFF_MISMATCH: 898NO_AUTHOR_SIGN_OFF: 54
A total of 793 false positives were resolved.
I think this patch does the job.
Lukas, what do you think?
Thanks,
Dwaipayan.
_______________________________________________
Linux-kernel-mentees mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/linux-kernel-mentees
Quoted 40 lines · Dwaipayan Ray
Quoted 15 lines · Dwaipayan Ray
Quoted 5 lines
Hi Dwaipayan,
I have generated reports on v5.0..v5.4.Quoted 9 lines
Relative change in warnings/errors before/after your patch:
https://github.com/AdityaSrivast/kernel-tasks/blob/master/Task4/relative_summary/summary_relative.txtDropped warnings:
https://github.com/AdityaSrivast/kernel-tasks/tree/master/Task4/relative_summary/dropped_warningsThere may be some discrepancies due to multi-threading. But it should give us a rough idea.
Quoted 8 lines
Thanks Aditya.
Seems like 763 false positives were resolved successfully. So I guess the patch is working as intended.
My evaluation is currently running and might need a few hours more. I will share it once it completes.
The summary of the analysis is as follows:
Commits checked: 47, 377 from v 5.4
Before the patch:
BAD_SIGN_OFF: 1516FROM_SIGN_OFF_MISMATCH: 931NO_AUTHOR_SIGN_OFF: 54After the patch:
BAD_SIGN_OFF: 756FROM_SIGN_OFF_MISMATCH: 898NO_AUTHOR_SIGN_OFF: 54
Quoted 5 lines
A total of 793 false positives were resolved.
I think this patch does the job.
Lukas, what do you think?
Agree. The numbers and my quick spot checks suggest it is a good fix to get merged.
Update the commit message to provide the overall summary of the two evaluations (by adding them up for the combined git range) and send the patch to Joe for a quick check.
By the way, I guess it is a good time to share the checkpatch.pl evaluation scripts with each other, compare them and settle among us for one common script to use.
(That gives away the mentee intro coding challenge but I am sure we can come up with a new one...)
Lukas
_______________________________________________
Linux-kernel-mentees mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/linux-kernel-mentees
Quoted 58 lines · Lukas Bulwahn
Quoted 40 lines · Dwaipayan Ray
Quoted 15 lines · Dwaipayan Ray
Quoted 5 lines
Hi Dwaipayan,
I have generated reports on v5.0..v5.4.Quoted 9 lines
Relative change in warnings/errors before/after your patch:
https://github.com/AdityaSrivast/kernel-tasks/blob/master/Task4/relative_summary/summary_relative.txtDropped warnings:
https://github.com/AdityaSrivast/kernel-tasks/tree/master/Task4/relative_summary/dropped_warningsThere may be some discrepancies due to multi-threading. But it should give us a rough idea.
Quoted 8 lines
Thanks Aditya.
Seems like 763 false positives were resolved successfully. So I guess the patch is working as intended.
My evaluation is currently running and might need a few hours more. I will share it once it completes.
The summary of the analysis is as follows:
Commits checked: 47, 377 from v 5.4
Before the patch:
BAD_SIGN_OFF: 1516FROM_SIGN_OFF_MISMATCH: 931NO_AUTHOR_SIGN_OFF: 54After the patch:
BAD_SIGN_OFF: 756FROM_SIGN_OFF_MISMATCH: 898NO_AUTHOR_SIGN_OFF: 54Quoted 5 lines
A total of 793 false positives were resolved.
I think this patch does the job.
Lukas, what do you think?Agree. The numbers and my quick spot checks suggest it is a good fix to get merged.
Update the commit message to provide the overall summary of the two evaluations (by adding them up for the combined git range) and send the patch to Joe for a quick check.
By the way, I guess it is a good time to share the checkpatch.pl evaluation scripts with each other, compare them and settle among us for one common script to use.
Quoted 6 lines
(That gives away the mentee intro coding challenge but I am sure we can come up with a new one...)
Lukas
Sure. The scripts I am using are:
For generating checkpatch reports over commit ids:
https://github.com/AdityaSrivast/kernel-tasks/blob/master/Task2/reports/before_commit/script.pl
For finding difference in errors/warning messages with count:
https://github.com/AdityaSrivast/kernel-tasks/blob/master/Task4/relative_summary/script.pl
For generating dropped warnings:
https://github.com/AdityaSrivast/kernel-tasks/blob/master/Task4/relative_summary/dropped_warnings/from_sign_off_mismatch/script.pl
_______________________________________________
Linux-kernel-mentees mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/linux-kernel-mentees
Quoted 9 lines
By the way, I guess it is a good time to share the checkpatch.pl evaluation scripts with each other, compare them and settle among us for one common script to use.
(That gives away the mentee intro coding challenge but I am sure we can come up with a new one...)
Sure that sounds good to me. We could even combine the scripts to create something more efficient probably.
Here is the one I use for generating the checkpatch output data over the supplied commits:
https://gist.github.com/raydwaipayan/57cc003e74618fe8de8917bc343fd234
(Multithreaded and semaphores to eliminate races between threads to write to file)
For generating summary:
https://gist.github.com/raydwaipayan/9e1f340e3e892bd48a2c972f0517698d
For comparisons and individual warnings I usually rely on diff and grep.
Thanks,
Dwaipayan.
_______________________________________________
Linux-kernel-mentees mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/linux-kernel-mentees