checkpatch: look for common misspellings
PATCH v2 1/1 checkpatch: look for common misspellings
Check for misspellings, based on Debian's lintian list. Several false positives were removed, and several additional words added that were common in the kernel:
backword backwords
invalide valide
recieves
singed unsinged
While going back and fixing existing spelling mistakes isn't a high priority, it'd be nice to try to catch them before they hit the tree.
In the 13830 commits between 3.15 and 3.16, the script would have noticed 560 spelling mistakes. The top 25 are shown here:
$ git log --pretty=oneline v3.15..v3.16 | wc -l
13830
$ git log --format='%H' v3.15..v3.16 | \
while read commit ; do \
echo "commit $commit" ; \
git log --format=email --stat -p -1 $commit | \
./scripts/checkpatch.pl --types=typo_spelling --no-summary - ; \
done | tee spell_v3.15..v3.16.txt | grep "may be misspelled" | \
awk '{print $2}' | tr A-Z a-z | sort | uniq -c | sort -rn
21 'seperate'
17 'endianess'
15 'sucess'
13 'noticable'
11 'occured'
11 'accomodate'
10 'interrup'
9 'prefered'
8 'unecessary'
8 'explicitely'
7 'supress'
7 'overriden'
7 'immediatly'
7 'funtion'
7 'defult'
7 'childs'
6 'succesful'
6 'splitted'
6 'specifc'
6 'reseting'
6 'recieve'
6 'changable'
5 'tmis'
5 'singed'
5 'preceeding'Thanks to Joe Perches for rewrites, suggestions, additional misspelling entries, and testing.
---
v2:
- Joe Perches made several improvements, including:
- relocated test to catch commit messages
- handle alternative capitalizations
- catch all mistakes in a line
- additional misspelling fix entries
--- scripts/checkpatch.pl | 44 ++- scripts/spelling.txt | 1042 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 1085 insertions(+), 1 deletion(-) create mode 100644 scripts/spelling.txt
scripts/checkpatch.pl +43 -1
@@ -9,7 +9,8 @@ use strict; use POSIX; my $P = $0; -$P =~ s@.*/@@g; +$P =~ s@(.*)/@@g; +my $D = $1; my $V = '0.32';
@@ -43,6 +44,7 @@ my $configuration_file = ".checkpatch.conf"; my $max_line_length = 80; my $ignore_perl_version = 0; my $minimum_perl_version = 5.10.0; +my $spelling_file = "$D/spelling.txt"; sub help { my ($exitcode) = @_;
@@ -429,6 +431,29 @@ our $allowed_asm_includes = qr{(?x: )}; # memory.h: ARM has a custom one +# Load common spelling mistakes and build regular expression list. +my $misspellings; +my @spelling_list; +my %spelling_fix; +open(my $spelling, '<', $spelling_file) + or die "$P: Can't open $spelling_file for reading: $!\n"; +while (<$spelling>) { + my $line = $_; + + $line =~ s/\s*\n?$//g; + $line =~ s/^\s*//g; + + next if ($line =~ m/^\s*#/); + next if ($line =~ m/^\s*$/); + + my ($suspect, $fix) = split(/\|\|/, $line); + + push(@spelling_list, $suspect); + $spelling_fix{$suspect} = $fix; +} +close($spelling); +$misspellings = join("|", @spelling_list); + sub build_types { my $mods = "(?x: \n" . join("|\n ", @modifierList) . "\n)"; my $all = "(?x: \n" . join("|\n ", @typeList) . "\n)";
@@ -2212,6 +2237,23 @@ sub process { "8-bit UTF-8 used in possible commit log\n" . $herecurr); } +# Check for various typo / spelling mistakes + if ($in_commit_log || $line =~ /^\+/) { + while ($rawline =~ /(?:^|[^a-z@])($misspellings)(?:$|[^a-z@])/gi) { + my $typo = $1; + my $typo_fix = $spelling_fix{lc($typo)}; + $typo_fix = ucfirst($typo_fix) if ($typo =~ /^[A-Z]/); + $typo_fix = uc($typo_fix) if ($typo =~ /^[A-Z]+$/); + my $msg_type = \&WARN; + $msg_type = \&CHK if ($file); + if (&{$msg_type}("TYPO_SPELLING", + "'$typo' may be misspelled - perhaps '$typo_fix'?\n" . $herecurr) && + $fix) { + $fixed[$fixlinenr] =~ s/(^|[^A-Za-z@])($typo)($|[^A-Za-z@])/$1$typo_fix$3/; + } + } + } + # ignore non-hunk lines and lines being removed next if (!$hunk_line || $line =~ /^-/);
scripts/spelling.txt new file +1042 -0
1,042 lines not on this page · raw message
Signature · 6 lines
1.9.1 -- Kees Cook Chrome OS Security
Quoted 11 lines · Kees Cook
Check for misspellings, based on Debian's lintian list. Several false positives were removed, and several additional words added that were common in the kernel:
backword backwords
invalide valide
recieves
singed unsingedWhile going back and fixing existing spelling mistakes isn't a high priority, it'd be nice to try to catch them before they hit the tree.
Seems sensible enough.
Hello Joe, Kees,
Sorry for late reply.
I was on holiday when the version 1 patch discussions were posted.
I am using codespell ( https://github.com/lucasdemarchi/codespell/ ). The codespell has its own typo dictionary.
The dictionary format is
typo->good (1 candidate)
typo->good1,good2, (multiple candidates) typo->good, comment (1 candidate with special remark)
Its similar to your typo||good format.
The license of the codespell is GPLv2 according to COPYING file in tar ball.
Compare number of typo samples in dictionary.
Your dictionary : 1033 codespell-1.4 : 4261
codespell-1.4 + my adding 5245
Your dictionary + codespell-1.4 + my adding - remove duplicate: 5742
Latest version of codespell is 1.7.
My dictionary is based on codespell-1.4. So I use the number as of 1.4.
I can provide my typo samples under GPLv2 license.
Masanari
Joe Perches
maintainerQuoted 1 line · Masanari Iida
Hello Joe, Kees,
Hello Masanari-san.
Quoted 2 lines
Sorry for late reply.
I was on holiday when the version 1 patch discussions were posted.
No worries, holidays are far more important than patches like this...
These patches are simple niceties, not fixes for bugs, so review and acceptance timing is not urgent.
Quoted 15 lines
I am using codespell ( https://github.com/lucasdemarchi/codespell/ ). The codespell has its own typo dictionary.
The dictionary format is
typo->good(1 candidate)
typo->good1,good2, (multiple candidates)typo->good, comment (1 candidate with special remark)Its similar to your typo||good format.
The license of the codespell is GPLv2 according to COPYING file in tar ball.
Compare number of typo samples in dictionary.
Your dictionary : 1033 codespell-1.4 : 4261
Quoted 7 lines
codespell-1.4 + my adding 5245
Your dictionary + codespell-1.4 + my adding - remove duplicate: 5742Latest version of codespell is 1.7.
My dictionary is based on codespell-1.4. So I use the number as of 1.4.I can provide my typo samples under GPLv2 license.
Thanks.
Any additions you have to the dictionary would be gladly welcomed.
Using a common format for the dictionary and any suggested corrections would be good too.
Maybe the dictionary and code should be changed to use the codespell format. It seems a bit more flexible than the lintian form.
I do not know if one project is more active than the other, but perhaps that should be the deciding factor. Or maybe just Kees' preference...
Merging all these together might not be a good solution though.
Right now, the checkpatch spelling code uses word boundaries that include an underscore.
checkpatch spelling tests are done on 4 segments of a #define like "PREFIX_PREFERED_SEG_ABC" finding the misspelling of PREFERED.
Some sifting of the dictionary is still necessary to eliminate some common prefixes to avoid too many false positives.
For example, "ths" was dropped because it's a prefix used by several modules even though it's a somewhat frequent typo.
Quoted 11 lines · On Mon, 8 Sep 2014 11:15:24 -0700 Kees Cook <[email protected]> wrote:
Check for misspellings, based on Debian's lintian list. Several false positives were removed, and several additional words added that were common in the kernel:
backword backwords
invalide valide
recieves
singed unsingedWhile going back and fixing existing spelling mistakes isn't a high priority, it'd be nice to try to catch them before they hit the tree.
I have a feeling this is going to be a rat hole and that scripts/spelling.txt will grow to consume the planet. Oh well, whatev.
Have a kernel joke:
scripts/spelling.txt renamed from scripts/spelling.txt~checkpatch-look-for-common-misspellings-fix +1 -0
@@ -553,6 +553,7 @@ jeffies||jiffies juse||just jus||just kown||known +kubys|linus langage||language langauage||language langauge||language
_
Joe Perches
maintainerQuoted 1 line · Andrew Morton
Have a kernel joke:
[]
scripts/spelling.txt jeffies||jiffies · line 553
Quoted 2 lines
scripts/spelling.txt renamed from scripts/spelling.txt~checkpatch-look-for-common-misspellings-fix +1
-0@@ -553,6 +553,7 @@ jeffies||jiffies +kubys|linus
Gimmu Smftre///
Quoted 11 lines · Andrew Morton
Quoted 9 lines · On Mon, 8 Sep 2014 11:15:24 -0700 Kees Cook <[email protected]> wrote:
Check for misspellings, based on Debian's lintian list. Several false positives were removed, and several additional words added that were common in the kernel:
backword backwords
invalide valide
recieves
singed unsinged
Quoted 5 lines
Quoted 2 lines
While going back and fixing existing spelling mistakes isn't a high priority, it'd be nice to try to catch them before they hit the tree.
I have a feeling this is going to be a rat hole and that
scripts/spelling.txtwill grow to consume the planet. Oh well, whatev.
What about making checkpatch use the codespell dictionay if codespell is installed?
Codespell is in Ubuntu 14.04LTS (but not in 12.04LTS).
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
Quoted 16 lines · Geert Uytterhoeven
Quoted 14 lines · Andrew Morton
Quoted 11 lines · On Mon, 8 Sep 2014 11:15:24 -0700 Kees Cook <[email protected]> wrote:
Check for misspellings, based on Debian's lintian list. Several false positives were removed, and several additional words added that were common in the kernel:
backword backwords
invalide valide
recieves
singed unsingedWhile going back and fixing existing spelling mistakes isn't a high priority, it'd be nice to try to catch them before they hit the tree.
Quoted 7 lines
Quoted 2 lines
I have a feeling this is going to be a rat hole and that
scripts/spelling.txtwill grow to consume the planet. Oh well, whatev.What about making checkpatch use the codespell dictionay if codespell is installed?
Codespell is in Ubuntu 14.04LTS (but not in 12.04LTS).
It's probably not a bad idea, but given the level of pruning that's been needed already to keep down the false positive rate, I'm nervous about a larger "general" corpus.
-Kees
Signature · 2 lines
Kees Cook Chrome OS Security
Joe Perches
maintainerQuoted 5 lines · Geert Uytterhoeven
Quoted 3 lines · Andrew Morton
Quoted 2 lines · On Mon, 8 Sep 2014 11:15:24 -0700 Kees Cook <[email protected]> wrote:
Check for misspellings, based on Debian's lintian list. Several false positives were removed, and several additional words added that were
[]
Quoted 7 lines
Quoted 2 lines
I have a feeling this is going to be a rat hole and that
scripts/spelling.txtwill grow to consume the planet. Oh well, whatev.What about making checkpatch use the codespell dictionay if codespell is installed?
Codespell is in Ubuntu 14.04LTS (but not in 12.04LTS).
I'm a little concerned about false positives if that's done, but it seems simple enough.
Maybe both of:
codespell: /usr/share/codespell/dictionary.txt
lintian: /usr/share/lintian/data/spelling/corrections
Talking about codespell, it detected 76 "informations" in 3.17-rc4. " grep -R informations * |wc -l" found 120 typos.
Test with "occured", codespell found 46, grep found 110. Test with "reseting" case, codespell found 21, grep found 26.
So I expect about half of the incoming typos will be detected by the tool, and be fixed.
Masanari
Joe Perches
maintainerQuoted 1 line · Masanari Iida
Test with "reseting" case, codespell found 21, grep found 26.
Hello Masanari.
How did codespell find any uses of reseting? What version of codespell are you using? (I tested with 1.7)
Looking at the git tree for codespell,
https://github.com/lucasdemarchi/codespell.git
the dictionary there doesn't have reseting.
If I add reseting->resetting to the dictionary, then codespell finds the same 31 uses that git grep -i does.
Quoted 5 lines · Joe Perches
Quoted 1 line · Masanari Iida
Test with "reseting" case, codespell found 21, grep found 26.
Hello Masanari.
Quoted 8 lines
How did codespell find any uses of reseting? What version of codespell are you using? (I tested with 1.7)
Looking at the git tree for codespell,
https://github.com/lucasdemarchi/codespell.git
the dictionary there doesn't have reseting.
Joe,
First of all, I use codespell 1.4 scripts with my original dictionary based on 1.4.
So I believe the "reseting" was added by me some times ago.
Quoted 4 lines
If I add
reseting->resettingto the dictionary, then codespell finds the same 31 uses that git grep -i does.
My codespell 1.4 works as case sensitive.
That's why we saw a little bit different result.
Masanari