manual: add package-list.txt generation support
PATCH v2 2/6 manual: add package-list.txt generation support
---
Change since v1:
* move the generation code in a script that is called in manual.mk (as suggested by Arnout)
* remove unneeded comment in the generated output (as suggested by Arnout)
--- docs/manual/manual.mk | 3 +++ support/scripts/gen-manual-pkg-list.sh | 19 +++++++++++++++++++ 2 files changed, 22 insertions(+) create mode 100755 support/scripts/gen-manual-pkg-list.sh
docs/manual/manual.mk +3 -0
@@ -24,6 +24,9 @@ $$(O)/docs/$(1)/$(1).$(4): docs/$(1)/$(1).txt $$($(call UPPERCASE,$(1))_SOURCES) -D $$(@D) $$< endef +$(TOPDIR)/docs/manual/package-list.txt: + $(TOPDIR)/support/scripts/gen-manual-pkg-list.sh > $@ + ################################################################################ # GENDOC -- generates the make targets needed to build asciidoc documentation. #
support/scripts/gen-manual-pkg-list.sh new file +19 -0
@@ -0,0 +1,19 @@ +#!/bin/sh +## +## gen-manual-pkg-list.sh +## +## Author(s): +## - Samuel MARTIN <[email protected]> +## +## Copyright (C) 2012 Samuel MARTIN +## + +# Generate the manual package-list.txt content (in asciidoc format) and print it +# to the standard output. + +printf "\n//\n// Autogenerated file\n//\n\n" +printf "[[package-list]]\n" +printf "Available packages\n" +printf -- "------------------\n\n" +grep -rlE --color=never '\((autotools|cmake|generic)-package\)' package | \ + sed -ne '/.*\/\(.*\).mk$/ s//* \1/p' | sort
Signature · 1 line
1.8.0.1
PATCH v2 3/6 manual: add deprecated-list.txt generation support
The whole deprecated-list.txt file is generated by the deprecated.py script by searching for symbols depending on BR2_DEPRECATED in the Buildroot code base.
---
Change since v1:
* rename the script deprecated.py -> gen-deprecated-list.py (to keep consistency)
--- docs/manual/manual.mk | 3 + support/scripts/gen-deprecated-list.py | 144 +++++++++++++++++++++++++++++++++ 2 files changed, 147 insertions(+) create mode 100755 support/scripts/gen-deprecated-list.py
docs/manual/manual.mk +3 -0
@@ -27,6 +27,9 @@ endef $(TOPDIR)/docs/manual/package-list.txt: $(TOPDIR)/support/scripts/gen-manual-pkg-list.sh > $@ +$(TOPDIR)/docs/manual/deprecated-list.txt: + python2 $(TOPDIR)/support/scripts/gen-deprecated-list.py > $@ + ################################################################################ # GENDOC -- generates the make targets needed to build asciidoc documentation. #
support/scripts/gen-deprecated-list.py new file +144 -0
144 lines not on this page · raw message
Signature · 1 line
1.8.0.1
PATCH v2 3/6 manual: add deprecated-list.txt generation support
Quoted 22 lines · Samuel Martin
The whole deprecated-list.txt file is generated by the
deprecated.pyscript by searching for symbols depending onBR2_DEPRECATEDin the Buildroot code base.Signed-off-by Samuel Martin<[email protected]>---
Change since v1:
* rename the script
deprecated.py->gen-deprecated-list.py(to keep consistency)--- docs/manual/manual.mk | 3 + support/scripts/gen-deprecated-list.py | 144 +++++++++++++++++++++++++++++++++ 2 files changed, 147 insertions(+)create mode 100755
support/scripts/gen-deprecated-list.pydocs/manual/manual.mk +0
-0
docs/manual/manual.mk endef · line 31
Quoted 5 lines
docs/manual/manual.mk +2
-0@@ -27,6 +27,9 @@ endef $(TOPDIR)/docs/manual/package-list.txt: $(TOPDIR)/support/scripts/gen-manual-pkg-list.sh> $@ +$(TOPDIR)/docs/manual/deprecated-list.txt: + python2 $(TOPDIR)/support/scripts/gen-deprecated-list.py> $@
We don't have python2 in dependencies.sh, and anyway some things
don't currently work if python is python3. So I'd just call the script without the python2. Anyway, does it not work with python3?
Quoted 29 lines
docs/manual/manual.mk +1
-0+ ################################################################################ # GENDOC -- generates the make targets needed to build asciidoc documentation. #support/scripts/gen-deprecated-list.py new file +19
-0@@ -0,0 +1,144 @@ +#!/usr/bin/env python +## +## deprecated-packages.py +## +## Author(s): +## - Samuel MARTIN<[email protected]> +## +## Copyright (C) 2012 Samuel MARTIN +## + +# Python 2.7 script searching for kconfig symbols depending on 'BR2_DEPRECATED' +# and generating (printing to the standard output) the manual file in asciidoc. + +import os +import re +import sys + + +ASCIIDOC_HEADER = """\
support/scripts/gen-deprecated-list.py line 29
Quoted 10 lines
support/scripts/gen-deprecated-list.py new file +10
-0+// +// Autogenerated file +// + +[[deprecated]] +Deprecated list +--------------- + +The following stuff are marked as _deprecated_ in Buildroot due to +their status either too old or unmaintained.
Better:
The following packages are marked as _deprecated_ because they are either too old or unmaintained in buildroot.
support/scripts/gen-deprecated-list.py line 38
Quoted 9 lines
support/scripts/gen-deprecated-list.py new file +9
-0+ +// Please check and sort by grepping the source running: +// +// $ git grep -EB4 'depends on BR2_DEPRECATED' +// +// and: +// +// $ git grep -EB4 'depends on BR2_DEPRECATED' | \\ +// grep -Eo '(:|-).*?(config|comment) BR2_.*'
This is redundant now. Instead, mention the make target
after 'Autogenerated'.
Quoted 17 lines
support/scripts/gen-deprecated-list.py new file +17
-0+""" + +NOT_SEARCHED = (".git", "board", "configs", "docs", "output", "support") + +# Relative path to category data map +# key: relative path from buildroot topdir to the search location +# value: (rank_in_menuconfig, category_name, recursive_search) +# rank_in_menuconfig is used for ordering the diplaying +# category_name is used in the diplaying +# recursive_search when searching for deprecated stuff +CAT_DIR2DATA = { + 'arch' : (0, "target architecture", True), + '.' : (1, "build options", False), + 'toolchain' : (2, "toolchain", True), + 'system' : (3, "system configuration", True), + 'package' : (4, "package selection", True), + 'fs' : (5, "filesystem images", True),
support/scripts/gen-deprecated-list.py line 65
Quoted 10 lines
support/scripts/gen-deprecated-list.py new file +10
-0+ 'boot' : (6, "bootloaders", True), + 'linux' : (7, "kernel", True), + } + +DEPR_SYMBOL = "BR2_DEPRECATED" + +_REGEX = r"config BR2_(.*?)\n" + \ + "((.*?(?!config)(prompt|bool|string|int) \"(.*?)\".*?|[^\n]+)\n)*" + \ + "(.*?(?!config )" + DEPR_SYMBOL + ".*?)\n" + \ + "((.*?(?!config)(prompt|bool|string|int) \"(.*?)\".*?|[^\n]+)\n)*"
This regex warrants some documentation :-) perhaps with re.VERBOSE
Why is the !config needed in the prompt lines?
Is there a reason to use the non-greedy .*? (except between the quotes)?
I would also name the "interesting" parts with (?P<...>) and use finditer()
and m.group() to retrieve them. But that's a matter of taste, I guess.
support/scripts/gen-deprecated-list.py line 73
Quoted 8 lines
support/scripts/gen-deprecated-list.py new file +8
-0+ +REGEX = re.compile(_REGEX, flags=re.MULTILINE) + + +def _get_buildroot_topdir(): + topdir = os.path.join(os.path.dirname(__file__), "..", "..") + topdir = os.path.abspath(topdir) + return topdir
I would just use the current directory as topdir. All the rest of
buildroot anyway assumes it's being run from the topdir.
Quoted 19 lines
support/scripts/gen-deprecated-list.py new file +19
-0+ +def get_dir_list(): + root = _get_buildroot_topdir() + dirs = ['.'] + for dir_ in os.listdir(root): + if dir_ in NOT_SEARCHED: + continue + dir__ = os.path.join(root, dir_) + if not os.path.isdir(dir__): + continue + dirs += [dir_] + return dirs + +def find_deprecated(root, recursive): + deprecated = list() + for root_, _, files_ in os.walk(root): + if not recursive and root_ != root: + break + for file_ in files_:
support/scripts/gen-deprecated-list.py line 102
Quoted 10 lines
support/scripts/gen-deprecated-list.py new file +10
-0+ if not file_.startswith("Config.in"): + continue + with open(os.path.join(root_, file_), "r") as f: + content = f.read() + if not DEPR_SYMBOL in content: + continue + found = REGEX.findall(content) + if found: + deprecated += found + return deprecated
It's quite a big change, but: why not use the source statements
in Config.in to do the recursion? The regex it easy enough:
r'^\s*source "(?<path>.*?)"$' This would be more future-safe:
when new directories are added, they will be handled correctly.
It would also be good if you could add a warning when
len(found) != len(re.findall(DEPR_SYMBOL, content))
support/scripts/gen-deprecated-list.py line 111
Quoted 9 lines
support/scripts/gen-deprecated-list.py new file +9
-0+ + +class Category(): + + def __init__(self, directory): + self.path = os.path.join(_get_buildroot_topdir(), directory) + rank, name, rec = CAT_DIR2DATA.get(directory, (99, directory, True)) + self.name = name + self.rank = rank
You can directly do
self.rank, self.name, rec = CAT_DIR2DATA.get(directory, (99, directory, True))
Quoted 5 lines
support/scripts/gen-deprecated-list.py new file +5
-0+ self.depr_items = find_deprecated(self.path, rec) + + def __str__(self): + items_ = list() + for item in self.depr_items:
support/scripts/gen-deprecated-list.py line 126
Quoted 10 lines
support/scripts/gen-deprecated-list.py new file +10
-0+ name_ = item[0].lower().replace("_", " ") + name_ = re.sub("^package ", "", name_) + vers = re.sub(".*?(version )?([0-9].*)", r'\2', name_) + if vers: + vers = re.sub(" ", ".", vers) + name_ = re.sub("(version )?([0-9].*)", vers, name_) + symbol = item[4] + if not symbol: + symbol = item[9] + items_ += ["** %-25s +[%s]+" % (name_, symbol)]
I don't think the name_ part adds much value. In most cases, symbol
is either the same or more descriptive. There are a few exceptions, but these could be fixed by changing the Config.in. These need to be
modified: gcc target, squashfs3 rootfs, uboot 11.06 and 11.09.
BTW, symbol is not really symbol but rather 'description' or something
similar.
support/scripts/gen-deprecated-list.py line 137
Quoted 11 lines
support/scripts/gen-deprecated-list.py new file +11
-0+ items_.sort() + output = "\n* %s:\n\n" % self.name.capitalize() + output += "\n".join(items_) + output += "\n" + return output + +def main(): + categories = [Category(directory) for directory in get_dir_list()] + categories = [category for category in categories if category.depr_items] + categories.sort(cmp=lambda x, y: x.rank - y.rank if x.rank != y.rank \ + else cmp(x.name, y.name))
categories.sort(key=lambda x: (x.rank, x.name))
support/scripts/gen-deprecated-list.py line 141
Quoted 4 lines
support/scripts/gen-deprecated-list.py new file +4
-0+ output = ASCIIDOC_HEADER + for category in categories: + output += str(category) + print output
Don't concat the output, just print directly.
print ASCIIDOC_HEADER
for category in categories:
print category
Regards,
Arnout
Quoted 3 lines
support/scripts/gen-deprecated-list.py new file +3
-0+ +if __name__ == "__main__": + main()
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286540 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
PATCH v2 6/6 manual: add generated *-list.txt
In reply to Samuel Martin* update package-list.txt (formerly named pkg-list.txt)
* update deprecated-list.txt
* update appendix.txt (which include these 2 generated files)
---
Change since v1:
* regenerate *-list.txt files
---
docs/manual/appendix.txt | 13 ++---
docs/manual/deprecated-list.txt | 68 ++++++++++++++++----------
docs/manual/{pkg-list.txt => package-list.txt} | 10 ++++
3 files changed, 55 insertions(+), 36 deletions(-)
rename docs/manual/{pkg-list.txt => package-list.txt} (99%)docs/manual/appendix.txt +3 -10
@@ -5,15 +5,8 @@ Appendix include::makedev-syntax.txt[] -[[package-list]] -Available packages ------------------- -// docs/manaual/pkg-list.txt is generated using the following command: -// $ git grep -E '\((autotools|cmake|generic)-package\)' package/ | \ -// cut -d':' -f1 | grep '\.mk$' | \ -// sed -e 's;.*\?/\(.*\?\).mk;* \1;' | \ -// sort > docs/manual/pkg-list.txt - -include::pkg-list.txt[] +// autogenerated +include::package-list.txt[] +// autogenerated include::deprecated-list.txt[]
docs/manual/deprecated-list.txt +42 -26
86 lines not on this page · raw message
docs/manual/package-list.txt renamed from docs/manual/pkg-list.txt +10 -0
@@ -1,3 +1,12 @@ + +// +// Autogenerated file +// + +[[package-list]] +Available packages +------------------ + * acl * acpid * alsa-lib
@@ -346,6 +355,7 @@ * luaexpat * luafilesystem * luajit +* lua-msgpack-native * luasocket * lvm2 * lzma
Signature · 1 line
1.8.0.1
PATCH v2 6/6 manual: add generated *-list.txt
Quoted 21 lines · Samuel Martin
docs/manual/package-list.txt renamed from docs/manual/pkg-list.txt +9
-0@@ -1,3 +1,12 @@ + +// +// Autogenerated file +// + +[[package-list]] +Available packages +------------------ + * acl * acpid * alsa-lib
docs/manual/package-list.txt line 361
Quoted 7 lines
docs/manual/package-list.txt renamed from docs/manual/pkg-list.txt +1
-0@@ -346,6 +355,7 @@ * luaexpat * luafilesystem * luajit +* lua-msgpack-native * luasocket * lvm2 * lzma
I love this! Now you can easily see what has changed between two versions.
Regards,
Arnout
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286540 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
PATCH v2 2/6 manual: add package-list.txt generation support
In reply to Samuel MartinQuoted 2 lines · Samuel Martin
Signed-off-by Samuel Martin<[email protected]>
(untested)
But since I'm pedantic, I still have some comments :-)
[snip]
Quoted 6 lines
support/scripts/gen-manual-pkg-list.sh new file +0
-0
support/scripts/gen-manual-pkg-list.sh line 8
Quoted 8 lines
support/scripts/gen-manual-pkg-list.sh new file +8
-0@@ -0,0 +1,19 @@ +#!/bin/sh +## +## gen-manual-pkg-list.sh +## +## Author(s): +## - Samuel MARTIN<[email protected]> +## +## Copyright (C) 2012 Samuel MARTIN
If you add a copyright statement, ideally you should also refer to the GPL:
This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
(The fact that you put the Copyright statement there makes it look as if it's proprietary. Adding the GPL reference disambiguates it. Of course, it is implied by the global buildroot license even if you leave out the GPL reference.)
support/scripts/gen-manual-pkg-list.sh line 17
Quoted 9 lines
support/scripts/gen-manual-pkg-list.sh new file +9
-0+## + +# Generate the manual package-list.txt content (in asciidoc format) and print it +# to the standard output. + +printf "\n//\n// Autogenerated file\n//\n\n" +printf "[[package-list]]\n" +printf "Available packages\n" +printf -- "------------------\n\n"
Since this is now part of a script, you can just cat a here document.
Much more readable.
support/scripts/gen-manual-pkg-list.sh line 18
Quoted 1 line
support/scripts/gen-manual-pkg-list.sh new file +1
-0+grep -rlE --color=never '\((autotools|cmake|generic)-package\)' package | \
We globally unexport GREP_OPTIONS, so the --color=never is redundant.
In addition, it would barf on non-gnu-grep.
Regards,
Arnout
Quoted 1 line
support/scripts/gen-manual-pkg-list.sh new file +1
-0+ sed -ne '/.*\/\(.*\).mk$/ s//* \1/p' | sort
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286540 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F