Scripted update of the MAINTAINERS files
cover Scripted update of the MAINTAINERS files
MAINTAINERS is a very large file.
Update the parse-maintainer script to use specific filenames to allow scripting of the moving of various bits into separate files.
Add a bash script to move various bits into multiple files.
Joe Perches (4):
parse-maintainers: Add section pattern sorting
parse-maintainers: Use perl hash references and specific filenames
parse-maintainers: Move matching sections from MAINTAINERS scripts/move_maintainer_sections.bash
scripts/move_maintainer_sections.bash | 87 +++++++++++++++++++++++++++++++ scripts/parse-maintainers.pl | 97 ++++++++++++++++++++++++++--------- 2 files changed, 161 insertions(+), 23 deletions(-) create mode 100755 scripts/move_maintainer_sections.bash
Signature · 1 line
2.10.0.rc2.1.g053435c
PATCH 1/4 parse-maintainers: Add section pattern sorting
Section [A-Z]: patterns are not currently in any required sorting order. Add a specific sorting sequence to MAINTAINERS entries.
Sort F: and X: patterns in alphabetic order.
The preferred section ordering is:
SECTION HEADER
M: Maintainers
R: Reviewers
P: Named persons without email addresses
L: Mailing list addresses
S: Status of this section (Supported, Maintained, Orphan, etc...)
W: Any relevant URLs
T: Source code control type (git, quilt, etc)
Q: Patchwork patch acceptance queue site
B: Bug tracking URIs
C: Chat URIs
F: Files with wildcard patterns (alphabetic ordered)
X: Excluded files with wildcard patterns (alphabetic ordered)
N: Files with regex patterns
K: Keyword regexes in source code for maintainership identification
Miscellaneous perl neatening:
o Rename %map to %hash, map has a different meaning in perl
o Avoid using \& and local variables for function indirection
o Use return for a little c like clarity
o Use c-like function call style instead of &function
--- scripts/parse-maintainers.pl | 70 +++++++++++++++++++++++++++++++------------- 1 file changed, 49 insertions(+), 21 deletions(-)
scripts/parse-maintainers.pl +49 -21
@@ -2,9 +2,9 @@ use strict; -my %map; +my %hash; -# sort comparison function +# sort comparison functions sub by_category($$) { my ($a, $b) = @_;
@@ -15,20 +15,47 @@ sub by_category($$) { $a =~ s/THE REST/ZZZZZZ/g; $b =~ s/THE REST/ZZZZZZ/g; - $a cmp $b; + return $a cmp $b; +} + +sub by_pattern($$) { + my ($a, $b) = @_; + my $preferred_order = 'MRPLSWTQBCFXNK'; + + my $a1 = uc(substr($a, 0, 1)); + my $b1 = uc(substr($b, 0, 1)); + + my $a_index = index($preferred_order, $a1); + my $b_index = index($preferred_order, $b1); + + $a_index = 1000 if ($a_index == -1); + $b_index = 1000 if ($b_index == -1); + + if (($a1 =~ /^F$/ && $b1 =~ /^F$/) || + ($a1 =~ /^X$/ && $b1 =~ /^X$/)) { + return $a cmp $b; + } + + if ($a_index < $b_index) { + return -1; + } elsif ($a_index == $b_index) { + return 0; + } else { + return 1; + } } sub alpha_output { - my $key; - my $sort_method = \&by_category; - my $sep = ""; - - foreach $key (sort $sort_method keys %map) { - if ($key ne " ") { - print $sep . $key . "\n"; - $sep = "\n"; - } - print $map{$key}; + foreach my $key (sort by_category keys %hash) { + if ($key eq " ") { + chomp $hash{$key}; + print $hash{$key}; + } else { + print "\n" . $key . "\n"; + foreach my $pattern (sort by_pattern split('\n', $hash{$key})) { + print($pattern . "\n"); + } + } } }
@@ -42,7 +69,7 @@ sub trim { sub file_input { my $lastline = ""; my $case = " "; - $map{$case} = ""; + $hash{$case} = ""; while (<>) { my $line = $_;
@@ -51,27 +78,28 @@ sub file_input { if ($line =~ m/^([A-Z]):\s*(.*)/) { $line = $1 . ":\t" . trim($2) . "\n"; if ($lastline eq "") { - $map{$case} = $map{$case} . $line; + $hash{$case} = $hash{$case} . $line; next; } $case = trim($lastline); - exists $map{$case} and die "Header '$case' already exists"; - $map{$case} = $line; + exists $hash{$case} and die "Header '$case' already exists"; + $hash{$case} = $line; $lastline = ""; next; } if ($case eq " ") { - $map{$case} = $map{$case} . $lastline; + $hash{$case} = $hash{$case} . $lastline; $lastline = $line; next; } trim($lastline) eq "" or die ("Odd non-pattern line '$lastline' for '$case'"); $lastline = $line; } - $map{$case} = $map{$case} . $lastline; + $hash{$case} = $hash{$case} . $lastline; } -&file_input; -&alpha_output; +file_input(); +alpha_output(); + exit(0);
Signature · 1 line
2.10.0.rc2.1.g053435c
PATCH 2/4 parse-maintainers: Use perl hash references and specific filenames
In reply to Joe PerchesInstead of reading STDIN and writing STDOUT, use specific filenames of MAINTAINERS and MAINTAINERS.new.
Use hash references instead of global hash %hash so future modifications can read and write specific hashes to split up MAINTAINERS into multiple files using a script.
--- scripts/parse-maintainers.pl | 57 ++++++++++++++++++++++++++------------------ 1 file changed, 34 insertions(+), 23 deletions(-)
scripts/parse-maintainers.pl +34 -23
@@ -2,7 +2,7 @@ use strict; -my %hash; +my $P = $0; # sort comparison functions sub by_category($$) {
@@ -45,61 +45,72 @@ sub by_pattern($$) { } } +sub trim { + my $s = shift; + $s =~ s/\s+$//; + $s =~ s/^\s+//; + return $s; +} + sub alpha_output { - foreach my $key (sort by_category keys %hash) { + my ($hashref, $filename) = (@_); + + open(my $file, '>', "$filename") or die "$P: $filename: open failed - $!\n"; + foreach my $key (sort by_category keys %$hashref) { if ($key eq " ") { - chomp $hash{$key}; - print $hash{$key}; + chomp $$hashref{$key}; + print $file $$hashref{$key}; } else { - print "\n" . $key . "\n"; - foreach my $pattern (sort by_pattern split('\n', $hash{$key})) { - print($pattern . "\n"); + print $file "\n" . $key . "\n"; + foreach my $pattern (sort by_pattern split('\n', %$hashref{$key})) { + print $file ($pattern . "\n"); } } } -} - -sub trim { - my $s = shift; - $s =~ s/\s+$//; - $s =~ s/^\s+//; - return $s; + close($file); } sub file_input { + my ($hashref, $filename) = (@_); + my $lastline = ""; my $case = " "; - $hash{$case} = ""; + $$hashref{$case} = ""; + + open(my $file, '<', "$filename") or die "$P: $filename: open failed - $!\n"; - while (<>) { + while (<$file>) { my $line = $_; # Pattern line? if ($line =~ m/^([A-Z]):\s*(.*)/) { $line = $1 . ":\t" . trim($2) . "\n"; if ($lastline eq "") { - $hash{$case} = $hash{$case} . $line; + $$hashref{$case} = $$hashref{$case} . $line; next; } $case = trim($lastline); - exists $hash{$case} and die "Header '$case' already exists"; - $hash{$case} = $line; + exists $$hashref{$case} and die "Header '$case' already exists"; + $$hashref{$case} = $line; $lastline = ""; next; } if ($case eq " ") { - $hash{$case} = $hash{$case} . $lastline; + $$hashref{$case} = $$hashref{$case} . $lastline; $lastline = $line; next; } trim($lastline) eq "" or die ("Odd non-pattern line '$lastline' for '$case'"); $lastline = $line; } - $hash{$case} = $hash{$case} . $lastline; + $$hashref{$case} = $$hashref{$case} . $lastline; + close($file); } -file_input(); -alpha_output(); +my %hash; + +file_input(\%hash, "MAINTAINERS"); +alpha_output(\%hash, "MAINTAINERS.new"); exit(0);
Signature · 1 line
2.10.0.rc2.1.g053435c
PATCH 3/4 parse-maintainers: Move matching sections from MAINTAINERS
In reply to Joe PerchesAllow any number of command line arguments to match either the section header or the section contents and create new files.
Create MAINTAINERS.new and SECTION.new.
This allows scripting of the movement of various sections from MAINTAINERS.
--- scripts/parse-maintainers.pl | 12 ++++++++++++ 1 file changed, 12 insertions(+)
scripts/parse-maintainers.pl +12 -0
@@ -109,8 +109,20 @@ sub file_input { } my %hash; +my %new_hash; file_input(\%hash, "MAINTAINERS"); + +foreach my $type (@ARGV) { + foreach my $key (keys %hash) { + if ($key =~ /$type/ || $hash{$key} =~ /$type/) { + $new_hash{$key} = $hash{$key}; + delete $hash{$key}; + } + } +} + alpha_output(\%hash, "MAINTAINERS.new"); +alpha_output(\%new_hash, "SECTION.new"); exit(0);
Signature · 1 line
2.10.0.rc2.1.g053435c
PATCH 4/4 scripts/move_maintainer_sections.bash
In reply to Joe PerchesMove MAINTAINERS into a separate directory and reorder it. Separate various blocks of MAINTAINER sections into separate files.
--- scripts/move_maintainer_sections.bash | 87 +++++++++++++++++++++++++++++++++++ 1 file changed, 87 insertions(+) create mode 100755 scripts/move_maintainer_sections.bash
scripts/move_maintainer_sections.bash new file +87 -0
@@ -0,0 +1,87 @@ +# Select various bits out of MAINTAINERS and create new file and commit +function separate_and_commit () { + local filename=$1 + shift + local comment=$1 + shift + local selection=$@ + + cd MAINTAINERS + perl ../scripts/parse-maintainers.pl $selection + mv MAINTAINERS.new MAINTAINERS + mv SECTION.new $filename + cd .. + cat << EOF > commit.msg +MAINTAINERS: Move $comment sections to separate file + +Make it easier to handle merge conflicts. +EOF + git add MAINTAINERS/$filename + git commit -s -F commit.msg MAINTAINERS/MAINTAINERS MAINTAINERS/$filename + rm commit.msg +} + +# First move the MAINTAINERS file into a separate MAINTAINERS directory +git mv MAINTAINERS MAINTAINERS.old +mkdir MAINTAINERS +git mv MAINTAINERS.old MAINTAINERS/MAINTAINERS +cat << EOF > commit.msg +MAINTAINERS: Move to MAINTAINERS directory + +This allows breaking up the very large and difficult to merge +MAINTAINERS files into separate files +EOF +git commit -s -F commit.msg MAINTAINERS/MAINTAINERS +rm commit.msg + +# Sort the MAINTAINERS file appropriately + +cd MAINTAINERS +perl ../scripts/parse-maintainers.pl +mv MAINTAINERS.new MAINTAINERS +cd .. +cat << EOF > commit.msg +MAINTAINERS: Reorder and alphabetize file sections + +Move the sections patterns into a standardized order. +EOF +git commit -s -F commit.msg MAINTAINERS/MAINTAINERS +rm commit.msg + +# Move the kernel related sections + +separate_and_commit kernel kernel "\\nF:\\s*kernel/(?:.*/)*\\n" "\\nF:\\s*kernel/(?:.*/)*(?:.*\\.[ch]|.*\\*)?\\n" +separate_and_commit filesystems "filesystems" "\\nF:\\s*fs/" +separate_and_commit hypervisors "hypervisor subsystems and drivers" "\\nL:\\s*xen-devel" "\\nL:\\s*virtualization" "Hyper-V" + +# Move arch specific files + +for arch in $(find arch -maxdepth 1 -type d | \ + sed 's@^arch/@@' | \ + awk '{ if (NR > 1) { print; } }' | \ + sort) ; do + separate_and_commit arch_$arch arch/$arch "\\nF:\\s*arch/$arch/(?:.*/)*\\n" +done + +# Move various subsystems and driver sections + +separate_and_commit networking_core "networking core" "\\nF:\\s*net/" +separate_and_commit drivers_wireless "wireless drivers" "\\nF:\\s*drivers/net/wireless/" "\\bWIRELESS\\b" +separate_and_commit drivers_ethernet "ethernet drivers" "\\nF:\\s*drivers/net/ethernet/" +separate_and_commit drivers_scsi "scsi drivers" "\\nF:\\s*drivers/scsi/" +separate_and_commit drivers_usb "usb drivers" "\\nF:\\s*drivers/usb/" +separate_and_commit drivers_media "media drivers" "\\nF:\\s*drivers/media/" +separate_and_commit drivers_watchdog "watchdog drivers" "\\nF:\\s*drivers/watchdog/" +separate_and_commit sound "sound subsystems and drivers" "\\nF:\\s*sound/" +separate_and_commit hardware_monitoring "monitoring subsystem and drivers" "\\nL:\\s*linux-hwmon" "\\nF:\\s*drivers/hwmon/" +separate_and_commit acpi "ACPI subsystem and drivers" "\\nL:\\s*linux-acpi" "\\nF:\\s*drivers/acpi/" +separate_and_commit drivers_input "input drivers" "\\nL:\\s*linux-input" "\\nF:\\s*drivers/input/" +separate_and_commit drivers_video "video drivers" "\\nL:\\s*linux-input" "\\nF:\\s*drivers/video/" +separate_and_commit drivers_gpio "gpio drivers" "\\nL:\\s*linux-gpio" "\\nF:\\s*drivers/gpio/" +separate_and_commit drivers_serial "serial/tty drivers" "\\nL:\\s*linux-serial" "\\nF:\\s*drivers/tty/" +separate_and_commit drivers_gpu_drm "GPU/DRM drivers" "\\nF:\\s*drivers/gpu/" +separate_and_commit drivers_i2c "I2C drivers" "\\nL:\\s*linux-i2c" "\\nF:\\s*drivers/i2c/" +separate_and_commit drivers_staging "staging drivers" "\\nL:\\s*devel\@driverdev" "\\nF:\\s*drivers/staging/" +separate_and_commit power "power management and drivers" "\\nL:\\s*linux-pm" "\\nF:\\s*drivers/cpufreq/" "\\nF:\\s*drivers/cpuidle/" "\\nF:\\s*drivers/devfreq/" +separate_and_commit pci "PCI drivers" "\\nL:\\s*linux-pci" "\\nF:\\s*drivers/pci/" +separate_and_commit platform-driver-x86 "x86 platform drivers" "\\nL:\\s*platform-driver-x86" "\\nF:\\s*drivers/platform/x86/"
Signature · 1 line
2.10.0.rc2.1.g053435c
Re: [PATCH 4/4] scripts/move_maintainer_sections.bash
Quoted 2 lines · Joe Perches
Move
MAINTAINERSinto a separate directory and reorder it. Separate various blocks of MAINTAINER sections into separate files.
Hey Linus.
If/when you try this out, do please let me know what you think.
There are some oddities in the results like the "PCI SUBSYSTEM" block ending up in arch_x86, but for a first pass I think it's fairly reasonable.
Of course the move_maintainer_sections script isn't meant to end up in the tree, it was just the easiest way to send it.
Re: [PATCH 4/4] scripts/move_maintainer_sections.bash
Quoted 8 lines · Joe Perches
Quoted 2 lines · Joe Perches
Move
MAINTAINERSinto a separate directory and reorder it. Separate various blocks of MAINTAINER sections into separate files.Hey Linus.
If/when you try this out, do please let me know what you think.
Ok, I've applied the preparatory patches, but not run the script.
Or rather, I ran the script to see what happens, but I'm not going to push the end result.
Quoted 1 line
From a quick look at the end result, I note:
- the arch maintainer split is pointless. It ends up being just one
entry per architecture, and for x86_64 not even that (because the x86 pattern matched all of them)
- the two arch maintainer lists that end up being bigger is x86 and
arm, but the x86 one picked up a log of misleading ones (not just PCI:
EFI, various random other things too)
- they all end up having the empty line at the top because of how the
parse-maintainers.pl script works.
But *some* of it looks really nice.
The other thing I note is that the way the patches look, this is going to be a disaster to merge with any other work - and there really tends to be a lot of things touching MAINTAINERS.
I'll have to think about it.
But at least the infrastructure patches are applied,
Linus
Re: [PATCH 4/4] scripts/move_maintainer_sections.bash
Quoted 12 lines · Linus Torvalds
Quoted 8 lines · Joe Perches
Quoted 2 lines · Joe Perches
Move
MAINTAINERSinto a separate directory and reorder it. Separate various blocks of MAINTAINER sections into separate files.Hey Linus.
If/when you try this out, do please let me know what you think.
Ok, I've applied the preparatory patches, but not run the script.
Quoted 7 lines
Or rather, I ran the script to see what happens, but I'm not going to push the end result.
From a quick look at the end result, I note:
- the arch maintainer split is pointless. It ends up being just one
entry per architecture
I think that's just preliminary.
Many of the arches have a single maintainer and many of the drivers specific to that arch could/should be moved into the arch_<foo> file.
But that doesn't seem easily scriptable.
Quoted 2 lines
for
x86_64not even that (because the x86> pattern matched all of them)
What x86_64 section is that? There isn't an arch/x86_64 directory and only 1 combined entry for x86 and x86_64.
Quoted 3 lines
- the two arch maintainer lists that end up being bigger is x86 and
arm, but the x86 one picked up a log of misleading ones (not just PCI:
EFI, various random other things too)
Yeah, it's imperfect. Suggestions welcomed.
Quoted 2 lines
- they all end up having the empty line at the top because of how the
parse-maintainers.pl script works.
That's because I was a bit lazy about the output.
awk '{ if (NR > 1) { print; } }'would fix it.
Also a 00-README type file for the introductory section could be useful.
Quoted 5 lines
But *some* of it looks really nice.
The other thing I note is that the way the patches look, this is going to be a disaster to merge with any other work - and there really tends to be a lot of things touching
MAINTAINERS.
It is going to be messy.
Once done it should be easier though.
Even today there's a merge conflict (kinda) with -next and the alphabetic
reordering movement as one of the sections
is duplicated again. (SYNC
FILE FRAMEWORK)
Collecting the remainder MAINTAINERS patches after an -rc1 and applying them with a quilt like merge might help too if Andrew feels up to it.
Quoted 3 lines
I'll have to think about it.
But at least the infrastructure patches are applied,
Thanks.