checkpatch: Avoid missing typo suggestions
PATCH 1/1 checkpatch: Avoid missing typo suggestions
My codespell dictionary has a lot of capitalized words. For example:
MSDOS->MS-DOS
Since checkpatch uses case-insensitive matching, I get an undefined variable warning and then empty suggestions for things like this:
Use of uninitialized value $typo_fix in concatenation (.) or string at ./scripts/checkpatch.pl line 2958.
WARNING: 'msdos' may be misspelled - perhaps ''?
+ struct msdos_dir_entry *de;
This fixes the matcher to avoid the warning, but it's still a rather silly suggestion:
WARNING: 'msdos' may be misspelled - perhaps 'MS-DOS'?
+ struct msdos_dir_entry *de;
So I'm not really sure what to do with this ... filter out bad suggestions instead?
--- scripts/checkpatch.pl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/checkpatch.pl +1 -1
@@ -727,7 +727,7 @@ if ($codespell) { my ($suspect, $fix) = split(/->/, $line); - $spelling_fix{$suspect} = $fix; + $spelling_fix{lc($suspect)} = $fix; } close($spelling); } else {
Signature · 5 lines
2.25.1 -- Kees Cook
Joe Perches
maintainerQuoted 12 lines · Kees Cook
My codespell dictionary has a lot of capitalized words. For example:
MSDOS->MS-DOS
Since checkpatch uses case-insensitive matching, I get an undefined variable warning and then empty suggestions for things like this:
Use of uninitialized value $typo_fix in concatenation (.) or string at
./scripts/checkpatch.plline 2958.WARNING: 'msdos' may be misspelled - perhaps ''?
+ structmsdos_dir_entry*de;
Quoted 8 lines
This fixes the matcher to avoid the warning, but it's still a rather silly suggestion:
WARNING: 'msdos' may be misspelled - perhaps 'MS-DOS'?
+ structmsdos_dir_entry*de;So I'm not really sure what to do with this ... filter out bad suggestions instead?
Hey Kees.
Maybe this?
btw: My codespell dictionary file moved to /usr/lib/python3/dist-packages/codespell_lib/data/dictionary.txt
and I had to use --codespell --codespellfile=(above) so maybe there should be multiple lookups for this file like the array below.
Are there other standard codespell dictionary locations?
--- scripts/checkpatch.pl | 31 ++++++++++++++++++++++++++----- 1 file changed, 26 insertions(+), 5 deletions(-)
scripts/checkpatch.pl +26 -5
@@ -59,7 +59,7 @@ my $minimum_perl_version = 5.10.0; my $min_conf_desc_length = 4; my $spelling_file = "$D/spelling.txt"; my $codespell = 0; -my $codespellfile = "/usr/share/codespell/dictionary.txt"; +my $codespellfile; my $conststructsfile = "$D/const_structs.checkpatch"; my $typedefsfile = ""; my $color = "auto";
@@ -716,7 +716,20 @@ if (open(my $spelling, '<', $spelling_file)) { } if ($codespell) { - if (open(my $spelling, '<', $codespellfile)) { + if (!defined($codespellfile)) { + my @csfiles = ("/usr/share/codespell/dictionary.txt", + "/usr/lib/python3/dist-packages/codespell_lib/data/dictionary.txt"); + foreach my $csfile (@csfiles) { + if (-f $csfile) { + $codespellfile = $csfile; + last; + } + } + } + + if (!defined($codespellfile)) { + warn "No codespell typos will be found - codespell dictionary not found\n"; + } elsif (open(my $spelling, '<', $codespellfile)) { while (<$spelling>) { my $line = $_;
@@ -2963,13 +2976,21 @@ sub process { while ($rawline =~ /(?:^|[^a-z@])($misspellings)(?:\b|$|[^a-z@])/gi) { my $typo = $1; my $typo_fix = $spelling_fix{lc($typo)}; - $typo_fix = ucfirst($typo_fix) if ($typo =~ /^[A-Z]/); - $typo_fix = uc($typo_fix) if ($typo =~ /^[A-Z]+$/); + $typo_fix = $spelling_fix{$typo} if (!defined($typo_fix)); + $typo_fix = $spelling_fix{uc($typo)} if (!defined($typo_fix)); + $typo_fix = 'unknown typo fix' if (!defined($typo_fix)); + if ($typo =~ /^[A-Z]+$/) { + $typo_fix = uc($typo_fix); + } elsif ($typo =~ /^[A-Z]/) { + $typo_fix = ucfirst($typo_fix); + } + my $msg_level = \&WARN; $msg_level = \&CHK if ($file); if (&{$msg_level}("TYPO_SPELLING", "'$typo' may be misspelled - perhaps '$typo_fix'?\n" . $herecurr) && - $fix) { + $fix && + $typo_fix ne 'unknown typo fix') { $fixed[$fixlinenr] =~ s/(^|[^A-Za-z@])($typo)($|[^A-Za-z@])/$1$typo_fix$3/; } }
Quoted 27 lines · Joe Perches
Quoted 12 lines · Kees Cook
My codespell dictionary has a lot of capitalized words. For example:
MSDOS->MS-DOS
Since checkpatch uses case-insensitive matching, I get an undefined variable warning and then empty suggestions for things like this:
Use of uninitialized value $typo_fix in concatenation (.) or string at
./scripts/checkpatch.plline 2958.WARNING: 'msdos' may be misspelled - perhaps ''?
+ structmsdos_dir_entry*de;Quoted 8 lines
This fixes the matcher to avoid the warning, but it's still a rather silly suggestion:
WARNING: 'msdos' may be misspelled - perhaps 'MS-DOS'?
+ structmsdos_dir_entry*de;So I'm not really sure what to do with this ... filter out bad suggestions instead?
Hey Kees.
Maybe this?
Quoted 8 lines
btw: My codespell dictionary file moved to
/usr/lib/python3/dist-packages/codespell_lib/data/dictionary.txtand I had to use --codespell --codespellfile=(above) so maybe there should be multiple lookups for this file like the array below.
Are there other standard codespell dictionary locations?
It might be better to support standard and non standard locations. I think it's better to request from codespell where his dictionary is. I created ticket for this:
https://github.com/codespell-project/codespell/issues/1540
However this patch is good as a temporary solution.
Regards,
Maxim.
Quoted 65 lines
--- scripts/checkpatch.pl | 31 ++++++++++++++++++++++++++----- 1 file changed, 26 insertions(+), 5 deletions(-)scripts/checkpatch.pl +26
-5@@ -59,7 +59,7 @@ my $minimum_perl_version = 5.10.0; my $min_conf_desc_length = 4; my $spelling_file = "$D/spelling.txt"; my $codespell = 0; -my $codespellfile = "/usr/share/codespell/dictionary.txt"; +my $codespellfile; my $conststructsfile = "$D/const_structs.checkpatch"; my $typedefsfile = ""; my $color = "auto";@@ -716,7 +716,20 @@ if (open(my $spelling, '<', $spelling_file)) { } if ($codespell) { - if (open(my $spelling, '<', $codespellfile)) { + if (!defined($codespellfile)) { + my @csfiles = ("/usr/share/codespell/dictionary.txt", + "/usr/lib/python3/dist-packages/codespell_lib/data/dictionary.txt"); + foreach my $csfile (@csfiles) { + if (-f $csfile) { + $codespellfile = $csfile; + last; + } + } + } + + if (!defined($codespellfile)) { + warn "No codespell typos will be found - codespell dictionary not found\n"; + } elsif (open(my $spelling, '<', $codespellfile)) { while (<$spelling>) { my $line = $_;@@ -2963,13 +2976,21 @@ sub process { while ($rawline =~ /(?:^|[^a-z@])($misspellings)(?:\b|$|[^a-z@])/gi) { my $typo = $1; my $typo_fix = $spelling_fix{lc($typo)}; - $typo_fix = ucfirst($typo_fix) if ($typo =~ /^[A-Z]/); - $typo_fix = uc($typo_fix) if ($typo =~ /^[A-Z]+$/); + $typo_fix = $spelling_fix{$typo} if (!defined($typo_fix)); + $typo_fix = $spelling_fix{uc($typo)} if (!defined($typo_fix)); + $typo_fix = 'unknown typo fix' if (!defined($typo_fix)); + if ($typo =~ /^[A-Z]+$/) { + $typo_fix = uc($typo_fix); + } elsif ($typo =~ /^[A-Z]/) { + $typo_fix = ucfirst($typo_fix); + } + my $msg_level = \&WARN; $msg_level = \&CHK if ($file); if (&{$msg_level}("TYPO_SPELLING", "'$typo' may be misspelled - perhaps '$typo_fix'?\n" . $herecurr) && - $fix) { + $fix && + $typo_fix ne 'unknown typo fix') { $fixed[$fixlinenr] =~ s/(^|[^A-Za-z@])($typo)($|[^A-Za-z@])/$1$typo_fix$3/; } }
Joe Perches
maintainerQuoted 1 line · Maxim Uvarov
On Thu, 4 Jun 2020 at 03:39, Joe Perches <[email protected]> wrote:
Hi Maxim.
Quoted 10 lines
Quoted 8 lines
btw: My codespell dictionary file moved to
/usr/lib/python3/dist-packages/codespell_lib/data/dictionary.txtand I had to use --codespell --codespellfile=(above) so maybe there should be multiple lookups for this file like the array below.
Are there other standard codespell dictionary locations?
It might be better to support standard and non standard locations.
It already does with the --codespellfile=<location> opti.
Quoted 1 line
I think it's better to request from codespell where his dictionary is.
Maybe a good idea, but looking at the codespell git, for versions 1.17 on there are several standard dictionaries.
https://github.com/codespell-project/codespell/tree/v1.17.1/codespell_lib/data
Quoted 2 lines
I created ticket for this:
https://github.com/codespell-project/codespell/issues/1540
Even if codespell is updated, the script would have to deal with older versions that don't support requesting that option.
cheers, Joe
Quoted 13 lines · Joe Perches
Quoted 1 line · Maxim Uvarov
On Thu, 4 Jun 2020 at 03:39, Joe Perches <[email protected]> wrote:
Hi Maxim.
Quoted 7 lines
Quoted 7 lines
btw: My codespell dictionary file moved to
/usr/lib/python3/dist-packages/codespell_lib/data/dictionary.txtand I had to use --codespell --codespellfile=(above) so maybe there should be multiple lookups for this file like the array below.
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Are there other standard codespell dictionary locations?
It might be better to support standard and non standard locations.
It already does with the --codespellfile=<location> opti.
I mean if codespess is in your PATH then it will be good to query dictinary.txt. Even if you install codespell to your $HOME it still is possible to use it without manually specifying where is the file.
Quoted 6 lines
Quoted 1 line
I think it's better to request from codespell where his dictionary is.
Maybe a good idea, but looking at the codespell git, for versions 1.17 on there are several standard dictionaries.
https://github.com/codespell-project/codespell/tree/v1.17.1/codespell_lib/data
Quoted 6 lines
Quoted 2 lines
I created ticket for this:
https://github.com/codespell-project/codespell/issues/1540Even if codespell is updated, the script would have to deal with older versions that don't support requesting that option.
Yes, we have to support old versions.
Quoted 3 lines
cheers, Joe
Quoted 22 lines · Joe Perches
Quoted 12 lines · Kees Cook
My codespell dictionary has a lot of capitalized words. For example:
MSDOS->MS-DOS
Since checkpatch uses case-insensitive matching, I get an undefined variable warning and then empty suggestions for things like this:
Use of uninitialized value $typo_fix in concatenation (.) or string at
./scripts/checkpatch.plline 2958.WARNING: 'msdos' may be misspelled - perhaps ''?
+ structmsdos_dir_entry*de;Quoted 8 lines
This fixes the matcher to avoid the warning, but it's still a rather silly suggestion:
WARNING: 'msdos' may be misspelled - perhaps 'MS-DOS'?
+ structmsdos_dir_entry*de;So I'm not really sure what to do with this ... filter out bad suggestions instead?
Quoted 6 lines
Hey Kees.
Maybe this?
btw: My codespell dictionary file moved to
/usr/lib/python3/dist-packages/codespell_lib/data/dictionary.txt
Yeah, mine too. I think I may have added a symlink to my filesystem to work around this.
Quoted 3 lines
and I had to use --codespell --codespellfile=(above) so maybe there should be multiple lookups for this file like the array below.
That seems like a good idea.
Quoted 42 lines
Are there other standard codespell dictionary locations?
--- scripts/checkpatch.pl | 31 ++++++++++++++++++++++++++----- 1 file changed, 26 insertions(+), 5 deletions(-)scripts/checkpatch.pl +15
-2@@ -59,7 +59,7 @@ my $minimum_perl_version = 5.10.0; my $min_conf_desc_length = 4; my $spelling_file = "$D/spelling.txt"; my $codespell = 0; -my $codespellfile = "/usr/share/codespell/dictionary.txt"; +my $codespellfile; my $conststructsfile = "$D/const_structs.checkpatch"; my $typedefsfile = ""; my $color = "auto";@@ -716,7 +716,20 @@ if (open(my $spelling, '<', $spelling_file)) { } if ($codespell) { - if (open(my $spelling, '<', $codespellfile)) { + if (!defined($codespellfile)) { + my @csfiles = ("/usr/share/codespell/dictionary.txt", + "/usr/lib/python3/dist-packages/codespell_lib/data/dictionary.txt"); + foreach my $csfile (@csfiles) { + if (-f $csfile) { + $codespellfile = $csfile; + last; + } + } + } + + if (!defined($codespellfile)) { + warn "No codespell typos will be found - codespell dictionary not found\n"; + } elsif (open(my $spelling, '<', $codespellfile)) { while (<$spelling>) { my $line = $_;
scripts/checkpatch.pl sub process { · line 2980
Quoted 7 lines
scripts/checkpatch.pl +2
-2@@ -2963,13 +2976,21 @@ sub process { while ($rawline =~ /(?:^|[^a-z@])($misspellings)(?:\b|$|[^a-z@])/gi) { my $typo = $1; my $typo_fix = $spelling_fix{lc($typo)}; - $typo_fix = ucfirst($typo_fix) if ($typo =~ /^[A-Z]/); - $typo_fix = uc($typo_fix) if ($typo =~ /^[A-Z]+$/); + $typo_fix = $spelling_fix{$typo} if (!defined($typo_fix)); + $typo_fix = $spelling_fix{uc($typo)} if (!defined($typo_fix));
This won't catch stuff like:
Cambrige->Cambridge
because neither "cambrige" nor "CAMBRIGE" is in %spelling_fix. And the original text is lost due to the //i. :( I'm really not sure what to do with these things in codespell. Lower case everything? Throw away anything not all lower case?
Signature · 1 line
Kees Cook
Joe Perches
maintainerQuoted 90 lines · Kees Cook
Quoted 22 lines · Joe Perches
Quoted 12 lines · Kees Cook
My codespell dictionary has a lot of capitalized words. For example:
MSDOS->MS-DOS
Since checkpatch uses case-insensitive matching, I get an undefined variable warning and then empty suggestions for things like this:
Use of uninitialized value $typo_fix in concatenation (.) or string at
./scripts/checkpatch.plline 2958.WARNING: 'msdos' may be misspelled - perhaps ''?
+ structmsdos_dir_entry*de;Quoted 8 lines
This fixes the matcher to avoid the warning, but it's still a rather silly suggestion:
WARNING: 'msdos' may be misspelled - perhaps 'MS-DOS'?
+ structmsdos_dir_entry*de;So I'm not really sure what to do with this ... filter out bad suggestions instead?
Quoted 6 lines
Hey Kees.
Maybe this?
btw: My codespell dictionary file moved to
/usr/lib/python3/dist-packages/codespell_lib/data/dictionary.txtYeah, mine too. I think I may have added a symlink to my filesystem to work around this.
Quoted 3 lines
and I had to use --codespell --codespellfile=(above) so maybe there should be multiple lookups for this file like the array below.
That seems like a good idea.
Quoted 41 lines
Are there other standard codespell dictionary locations?
--- scripts/checkpatch.pl | 31 ++++++++++++++++++++++++++----- 1 file changed, 26 insertions(+), 5 deletions(-)scripts/checkpatch.pl +15
-2@@ -59,7 +59,7 @@ my $minimum_perl_version = 5.10.0; my $min_conf_desc_length = 4; my $spelling_file = "$D/spelling.txt"; my $codespell = 0; -my $codespellfile = "/usr/share/codespell/dictionary.txt"; +my $codespellfile; my $conststructsfile = "$D/const_structs.checkpatch"; my $typedefsfile = ""; my $color = "auto";@@ -716,7 +716,20 @@ if (open(my $spelling, '<', $spelling_file)) { } if ($codespell) { - if (open(my $spelling, '<', $codespellfile)) { + if (!defined($codespellfile)) { + my @csfiles = ("/usr/share/codespell/dictionary.txt", + "/usr/lib/python3/dist-packages/codespell_lib/data/dictionary.txt"); + foreach my $csfile (@csfiles) { + if (-f $csfile) { + $codespellfile = $csfile; + last; + } + } + } + + if (!defined($codespellfile)) { + warn "No codespell typos will be found - codespell dictionary not found\n"; + } elsif (open(my $spelling, '<', $codespellfile)) { while (<$spelling>) { my $line = $_;scripts/checkpatch.pl sub process { · line 2980
Quoted 7 lines
scripts/checkpatch.pl +2
-2@@ -2963,13 +2976,21 @@ sub process { while ($rawline =~ /(?:^|[^a-z@])($misspellings)(?:\b|$|[^a-z@])/gi) { my $typo = $1; my $typo_fix = $spelling_fix{lc($typo)}; - $typo_fix = ucfirst($typo_fix) if ($typo =~ /^[A-Z]/); - $typo_fix = uc($typo_fix) if ($typo =~ /^[A-Z]+$/); + $typo_fix = $spelling_fix{$typo} if (!defined($typo_fix)); + $typo_fix = $spelling_fix{uc($typo)} if (!defined($typo_fix));This won't catch stuff like:
Quoted 1 line
Cambrige->Cambridge
Huh? Did you test this?
$ git diff scripts/spelling.txt
scripts/spelling.txt +1 -0
@@ -251,6 +251,7 @@ callibration||calibration callled||called calucate||calculate calulate||calculate +Cambridg||Cambridge cancelation||cancellation cancle||cancel capabilites||capabilities
$ cat test_spell.c // SPDX-License-Identifier: GPL-2.0-only
/* This is a test.
* Cambridg
*/
$ ./scripts/checkpatch.pl --strict -f test_spell.c CHECK: 'Cambridg' may be misspelled - perhaps 'Cambridge'? #4: FILE: test_spell.c:4: + * Cambridg
total: 0 errors, 0 warnings, 1 checks, 5 lines checked
NOTE: For some of the reported defects, checkpatch may be able to
mechanically convert to the typical style using --fix or --fix-inplace.
test_spell.c has style problems, please review.
NOTE: If any of the errors are false positives, please report
them to the maintainer, see CHECKPATCH in MAINTAINERS.
Quoted 4 lines
because neither "cambrige" nor "CAMBRIGE" is in %spelling_fix. And the original text is lost due to the //i. :( I'm really not sure what to do with these things in codespell. Lower case everything? Throw away anything not all lower case?
Kees Cook
Reviewed-byQuoted 1 line · Joe Perches
Huh? Did you test this?
I didn't, no. I was going off my earlier discoveries about how the "msdos" thing got parsed weird. My apologies!
Quoted 4 lines
$ ./scripts/checkpatch.pl --strict -f test_spell.c CHECK: 'Cambridg' may be misspelled - perhaps 'Cambridge'? #4: FILE: test_spell.c:4: + * Cambridg
Thanks for sorting this (and me) out! :)
Signature · 1 line
Kees Cook