Pull request for branch for-master/doc

Signed-off-by Samuel Martin<[email protected]>

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Samuel" == Samuel Martin <[email protected]> writes:

Signed-off-by Samuel Martin <[email protected]>

---
  docs/manual/manual.mk | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

docs/manual/manual.mk +1 -1

index d664603..aa20534 100644
@@ -18,7 +18,7 @@ $(1): $(1)-$(3)
  $(1)-$(3): $$(O)/docs/$(1)/$(1).$(4)
 
  $$(O)/docs/$(1)/$(1).$(4): docs/$(1)/$(1).txt $$($(call UPPERCASE,$(1))_SOURCES)
-	@echo "Generating $(5) $(1)..."
+	@$(call MESSAGE,"Generating $(5) $(1)...")
  	$(Q)mkdir -p $$(@D)
  	$(Q)a2x $(6) -f $(2) -d book -L -r $(TOPDIR)/docs/images \
  	  -D $$(@D) $$<

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Samuel" == Samuel Martin <[email protected]> writes:

Signed-off-by Samuel Martin <[email protected]>

---
  docs/manual/faq-troubleshooting.txt | 21 +++++++++++++++++++++
  1 file changed, 21 insertions(+)

docs/manual/faq-troubleshooting.txt +6 -0

index f91a8ef..8b0882a 100644
@@ -126,3 +126,24 @@ directory as the new root, will most likely fail.
  If you want to run the target filesystem inside a chroot, or as an NFS
  root, then use the tarball image generated in +images/+ and extract it
  as root.
+
+[[faq-relocatable-toolchains]]
+Are Buildroot toolchains relocatable?
+-------------------------------------
+
+Because Buildroot uses several toolchain backends, there is one anwser

docs/manual/faq-troubleshooting.txt +5 -0

+
+Using the internal Buildroot toolchain backend
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+These toolchains are currently *not relocatable*.

Signed-off-by Samuel Martin <[email protected]>

---
 docs/manual/faq-troubleshooting.txt | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

docs/manual/faq-troubleshooting.txt +6 -0

index f91a8ef..8b0882a 100644
@@ -126,3 +126,24 @@ directory as the new root, will most likely fail.
 If you want to run the target filesystem inside a chroot, or as an NFS
 root, then use the tarball image generated in +images/+ and extract it
 as root.
+
+[[faq-relocatable-toolchains]]
+Are Buildroot toolchains relocatable?
+-------------------------------------
+
+Because Buildroot uses several toolchain backends, there is one anwser

Dear Samuel Martin,

Quoted 20 lines · Samuel Martin

Signed-off-by Samuel Martin <[email protected]>

---
 docs/manual/faq-troubleshooting.txt | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

docs/manual/faq-troubleshooting.txt +6 -0

index f91a8ef..8b0882a 100644
@@ -126,3 +126,24 @@ directory as the new root, will most likely fail.
 If you want to run the target filesystem inside a chroot, or as an NFS
 root, then use the tarball image generated in +images/+ and extract it
 as root.
+
+[[faq-relocatable-toolchains]]
+Are Buildroot toolchains relocatable?
+-------------------------------------
+
+Because Buildroot uses several toolchain backends, there is one anwser

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 144

Quoted 10 lines

docs/manual/faq-troubleshooting.txt +10 -0

+per backend:
+
+Using the external or crosstool-NG toolchain backend
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+These toolchains are mostly likely relocatable; the _Buildroot compiler
+wrapper_ is relocatable.
+
+However, some tools built and installed into the +HOST_DIR+ may not be
+relocatable, with some absolute pathes hard-coded inside.

I find this text rather confusing. What does "mostly likely" means for the reader? The reader surely wants to know if the toolchain is relocatable or not.

Therefore, I think this text should be changed to clarify what is meant by "toolchain". Is it just the bare toolchain (gcc, binutils, libc, kernel headers), or is it the complete toolchain that Buildroot produces, with all the additional libraries and headers?

Also, you mention the "Buildroot compiler wrapper", but nowhere in the documentation we explain what it is, and I don't think it should be explained in the documentation, except in a section dedicated to "Deep Buildroot internals", but certainly not to answer a FAQ question about relocatable toolchains.

I would rather think that a section discussing how to use the "SDK" produced by Buildroot, and how it can be shared with application developers, could mention that the SDK is currently not relocatable, but that we are working on fixing this.

Signed-off-by Samuel Martin <[email protected]>

---
  docs/manual/common-usage.txt        |  1 +
  docs/manual/faq-troubleshooting.txt | 34 ++++++++++++++++++++++++++++++++++
  2 files changed, 35 insertions(+)

docs/manual/common-usage.txt +0 -0

index c7f511a..b36965f 100644

docs/manual/faq-troubleshooting.txt +0 -0

index 8b0882a..f367275 100644

Signed-off-by Samuel Martin <[email protected]>

---
 docs/manual/common-usage.txt        |  1 +
 docs/manual/faq-troubleshooting.txt | 34 ++++++++++++++++++++++++++++++++++
 2 files changed, 35 insertions(+)

docs/manual/common-usage.txt +1 -0

index c7f511a..b36965f 100644
@@ -20,6 +20,7 @@ all sources that you previously selected in the configurator
 You can now disconnect or copy the content of your +dl+
 directory to the build-host.
 
+[[daily-oot-build]]
 Building out-of-tree
 ~~~~~~~~~~~~~~~~~~~~
 

docs/manual/faq-troubleshooting.txt +24 -0

index 8b0882a..f367275 100644
@@ -147,3 +147,37 @@ Using the internal Buildroot toolchain backend
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 These toolchains are currently *not relocatable*.
+
+However, it is possible to build a toolchain using the internal Buildroot
+backend in another location (see xref:daily-oot-build[]), then use it as an
+_external_ one to prevent from rebuilding it after issuing +make clean+
+(see xref:faq-avoid-rebuild-toolchain[]).
+
+[[faq-avoid-rebuild-toolchain]]
+How to avoid always rebuilding the toolchain?
+---------------------------------------------
+
+When using the internal Buildroot toolchain backend or the
+http://crosstool-ng.org[crosstool-NG] one, runnong +make clean+ not only
+remove the built binaries, but also the whole toolchain.
+
+It is possible to prevent this by, first, building the toolchain in a
+place different from the one  where the images will be built.
+
+To do so:
+
+1. Build the toolchain somewhere:
++
+----------------------------
+make toolchain O=/path/to/toolchain
+----------------------------

Dear Samuel Martin,

Quoted 60 lines · Samuel Martin

Signed-off-by Samuel Martin <[email protected]>

---
 docs/manual/common-usage.txt        |  1 +
 docs/manual/faq-troubleshooting.txt | 34 ++++++++++++++++++++++++++++++++++
 2 files changed, 35 insertions(+)

docs/manual/common-usage.txt +1 -0

index c7f511a..b36965f 100644
@@ -20,6 +20,7 @@ all sources that you previously selected in the configurator
 You can now disconnect or copy the content of your +dl+
 directory to the build-host.
 
+[[daily-oot-build]]
 Building out-of-tree
 ~~~~~~~~~~~~~~~~~~~~
 

docs/manual/faq-troubleshooting.txt +33 -0

index 8b0882a..f367275 100644
@@ -147,3 +147,37 @@ Using the internal Buildroot toolchain backend
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 These toolchains are currently *not relocatable*.
+
+However, it is possible to build a toolchain using the internal Buildroot
+backend in another location (see xref:daily-oot-build[]), then use it as an
+_external_ one to prevent from rebuilding it after issuing +make clean+
+(see xref:faq-avoid-rebuild-toolchain[]).
+
+[[faq-avoid-rebuild-toolchain]]
+How to avoid always rebuilding the toolchain?
+---------------------------------------------
+
+When using the internal Buildroot toolchain backend or the
+http://crosstool-ng.org[crosstool-NG] one, runnong +make clean+ not only
+remove the built binaries, but also the whole toolchain.
+
+It is possible to prevent this by, first, building the toolchain in a
+place different from the one  where the images will be built.
+
+To do so:
+
+1. Build the toolchain somewhere:
++
+----------------------------
+make toolchain O=/path/to/toolchain
+----------------------------
++
+1. Use this pre-built toolchain as an external one in a new and clean
+   build (see xref:external-toolchain-backend[]) with a different output
+   directory than the pre-built toolchain one.
+
+This way, you will not have to rebuild the toolchain after invoking
++make clean+.
+
+However, the toolchain should be regenerated after any change of the

docs/manual/faq-troubleshooting.txt Using the internal Buildroot toolchain backend · line 183

Quoted 1 line

docs/manual/faq-troubleshooting.txt +1 -0

++Target Architecture+ (or its _variant_), or done in the +Toolchain+ menu.

I think this text is misleading, because it suggests that the only way to avoid rebuilding the toolchain over and over again is to build a toolchain with Buildroot, and then use it as an external toolchain in Buildroot. There are other options that are possible:
* Use Crosstool-NG outside of Buildroot to generate a toolchain
* Use a pre-built toolchain

I have the feeling that this new text doesn't articulate very well with the text that was added in 401d323698266468e322471530ee1997835fc335.

We have to be careful to not add a set of random facts in the manual one after the other, without keeping a consistency and ordering on the different parts of the manual.

Signed-off-by Samuel Martin <[email protected]>

---
  docs/manual/patch-policy.txt | 6 +++---
  1 file changed, 3 insertions(+), 3 deletions(-)

docs/manual/patch-policy.txt +3 -3

index 78ac5b6..468a954 100644
@@ -34,7 +34,7 @@ Most patches are provided within Buildroot, in the package
  directory; these typically aim to fix cross-compilation, libc support,
  or other such issues.
 
-These patch files should be named +<packagename>-*.patch+.
+These patch files should be named +<packagename>-<description>.patch+.
 
  A +series+ file, as used by +quilt+, may also be added in the
  package directory. In that case, the +series+ file defines the patch

@@ -57,8 +57,8 @@ How patches are applied
    applied according to the +series+ file;
  +
  * Otherwise, patch files matching `<packagename>-*.patch` or
-  `<packagename>-*.patch.<arch>` (where +<arch>+ is the architecture
-  name) are applied following the +ls+ command order.
+  `<packagename>-<description>.patch.<arch>` (where +<arch>+ is the
+  architecture name) are applied following the +ls+ command order.
 
  . Run the +<packagename>_POST_PATCH_HOOKS+ commands if defined.
 

Signed-off-by Samuel Martin <[email protected]>

---
 docs/manual/patch-policy.txt | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

docs/manual/patch-policy.txt +1 -1

index 78ac5b6..468a954 100644
@@ -34,7 +34,7 @@ Most patches are provided within Buildroot, in the package
 directory; these typically aim to fix cross-compilation, libc support,
 or other such issues.
 
-These patch files should be named +<packagename>-*.patch+.
+These patch files should be named +<packagename>-<description>.patch+.
 
 A +series+ file, as used by +quilt+, may also be added in the
 package directory. In that case, the +series+ file defines the patch

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Samuel" == Samuel Martin <[email protected]> writes:

* update post-build scripts mechanism that now support several scripts
* fix formatting

Signed-off-by Samuel Martin <[email protected]>

---
  docs/manual/customize-rootfs.txt | 30 ++++++++++++++++--------------
  1 file changed, 16 insertions(+), 14 deletions(-)

docs/manual/customize-rootfs.txt +0 -0

index 4a4d620..95dfc16 100644
@@ -25,18 +25,18 @@ there are a few ways to customize the resulting target filesystem.
    directories, +.empty+ files and files ending with +~+ are excluded.
    _Among these first 3 methods, this one should be preferred_.
 

docs/manual/customize-rootfs.txt +4 -4

    found in the +System configuration+ menu. The destination root
-  filesystem folder is given as the first argument to this script,
-  and this script can then be used to remove or modify any file in your
+  filesystem folder is given as the first argument to these scripts,
+  and these scripts can then be used to remove or modify any file in your
    target filesystem. You should, however, use this feature with care.
    Whenever you find that a certain package generates wrong or unneeded
-  files, you should fix that package rather than work around it with a
-  post-build cleanup script.
+  files, you should fix that package rather than work around it with some
+  post-build cleanup scripts.
    You may also use these variables in your post-build script:
      - +BUILDROOT_CONFIG+: the path to the Buildroot .config file
      - +HOST_DIR+, +STAGING_DIR+, +TARGET_DIR+: see

@@ -55,23 +55,25 @@ there are a few ways to customize the resulting target filesystem.
    installation. Note that this method is *not recommended*, as it
    duplicates the entire skeleton, which prevents from taking advantage

docs/manual/customize-rootfs.txt +3 -2

  extract your root filesystem tarball in a location exported by your
  NFS server, or to create a special firmware image that bundles your
  root filesystem and kernel image, or any other custom action), you can
  specify a space-separated list of scripts in the
-+BR2_ROOTFS_POST_IMAGE_SCRIPT+ configuration option.
++BR2_ROOTFS_POST_IMAGE_SCRIPT+ configuration option. This option can be
+found in the +System configuration+ menu as well.
 
  Each of those scripts will be called with the path to the +images+
  output directory as first and unique argument, and will be executed
  with the main Buildroot source directory as the current
  directory. Those scripts will be executed as the user that executes
  Buildroot, which should normally not be the root user. Therefore, any
-action requiring root permissions in one of these post-image script
+action requiring root permissions in one of these _post-image scripts_
  will require special handling (usage of fakeroot or sudo), which is
  left to the script developer.
 

* add the gen-manual-lists.py script generating the target and host package tables, and the deprecated stuff list as well. These tables and lists are generated parsing the Config.in files;
* add a make target to update these lists in the manual;
* update the these appendices of the manual.

Signed-off-by Samuel Martin <[email protected]>

---
  docs/manual/appendix.txt            |  13 +-
  docs/manual/deprecated-list.txt     |  81 ++-
  docs/manual/manual.mk               |   7 +
  docs/manual/package-list.txt        | 948 ++++++++++++++++++++++++++++++++++++
  docs/manual/pkg-list.txt            | 870 ---------------------------------

  support/scripts/gen-manual-lists.py | 237 +++++++++
  6 files changed, 1233 insertions(+), 923 deletions(-)

create mode 100644 docs/manual/package-list.txt delete mode 100644 docs/manual/pkg-list.txt create mode 100755 support/scripts/gen-manual-lists.py

docs/manual/appendix.txt +3 -10

index ef34169..37548f7 100644
@@ -6,15 +6,8 @@ Appendix
 
  include::makedev-syntax.txt[]
 
-[[package-list]]
-Available packages
-------------------
-// docs/manaual/pkg-list.txt is generated using the following command:
-// $ git grep -E '\((autotools|cmake|generic)-package\)' package/ |  \
-//     cut -d':' -f1 | grep '\.mk$' | \
-//     sed -e 's;.*\?/\(.*\?\).mk;* \1;' | \
-//     sort > docs/manual/pkg-list.txt
-
-include::pkg-list.txt[]
+// autogenerated
+include::package-list.txt[]
 
+// autogenerated
  include::deprecated-list.txt[]

docs/manual/deprecated-list.txt +0 -0

index 6dc87a4..3468219 100644

docs/manual/manual.mk +0 -0

index aa20534..dd07b4e 100644

docs/manual/manual.mk +0 -0

  ################################################################################
  # GENDOC -- generates the make targets needed to build asciidoc documentation.
  #

docs/manual/package-list.txt new file +3 -0

new file mode 100644
index 0000000..79d4ec6
@@ -0,0 +1,948 @@
+//
+// Automatically generated list for Buildroot manual.
+// Buildroot 2013.02-00794-g199d6d6-dirty

docs/manual/package-list.txt new file +9 -0

+| acl                            | . -> System tools
+| acpid                          | . -> Hardware handling
+| alsa-lib                       | . -> Libraries -> Audio/Sound
+| alsa-utils                     | . -> Audio and video applications
+| alsamixergui                   | . -> Graphic libraries and applications (graphic/text)
+| applewmproto                   | . -> Graphic libraries and applications (graphic/text) -> X11R7 X protocols
+| appres                         | . -> Graphic libraries and applications (graphic/text) -> X11R7 Applications
+| apr                            | . -> Libraries -> Other
+| apr-util                       | . -> Libraries -> Other

support/scripts/gen-manual-lists.py new file +25 -0

new file mode 100755
index 0000000..f2a2dc4
@@ -0,0 +1,237 @@
+#!/usr/bin/env python2
+##
+## gen-manual-lists.py
+##
+## This script generates the following Buildroot manual appendices:
+##  - the package tables (one for the target, the other for host tools);
+##  - the deprecated items.
+##
+## Author(s):
+##  - Samuel Martin <[email protected]>
+##
+## Copyright (C) 2013 Samuel Martin
+##
+## This program is free software; you can redistribute it and/or modify
+## it under the terms of the GNU General Public License as published by
+## the Free Software Foundation; either version 2 of the License, or
+## (at your option) any later version.
+##
+## This program is distributed in the hope that it will be useful,
+## but WITHOUT ANY WARRANTY; without even the implied warranty of
+## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+## GNU General Public License for more details.
+##
+## You should have received a copy of the GNU General Public License
+## along with this program; if not, write to the Free Software

support/scripts/gen-manual-lists.py new file +10 -0

+
+import os
+import re
+import sys
+import datetime
+from argparse import ArgumentParser
+
+try:
+    import kconfiglib
+except ImportError:

support/scripts/gen-manual-lists.py new file +4 -0

+    if sort:
+        lines.sort(key=lambda x:x.lower())
+    if root:
+        loc_label = _get_parents(None, root) + [root.get_title(), "..."]

support/scripts/gen-manual-lists.py new file +11 -0

+        self.package_dir = os.path.join(self.base_dir, self.package_dirname)
+        self.config = kconfiglib.Config(os.path.join(self.base_dir, self.root_config))
+        self.package_symbols = []
+        self._fill_package_symbols()
+        self._deprecated = self.config.get_symbol(self.deprecated_symbol)
+        self.br_version_full = os.environ.get("BR2_VERSION_FULL")
+        self.generation_date = str(datetime.datetime.utcnow())
+
+    def _get_package_symbol(self, package_name):
+        pkg_symbol = re.sub("[-+.]", "_", package_name)
+        pkg_symbol = pkg_symbol.upper()

support/scripts/gen-manual-lists.py new file +13 -0

+
+    def _get_symbol_label(self, item):
+        label = item.prompts[0][0]
+        if self._is_deprecated(item):
+            label += " *(deprecated)*"
+        return label
+
+    def _format_list(self, root, items, section_anchor, section_title):
+        table = format_asciidoc_table(root, items, get_label=self._get_symbol_label)
+        content = _LIST_FORMAT.format(
+            br_version_full=self.br_version_full,
+            gen_date=self.generation_date,
+            section_anchor=section_anchor,

support/scripts/gen-manual-lists.py new file +42 -0

+        items = get_symbol_subset(root_item.get_items(), filter=self._is_package)
+        content = self._format_list(root_item, items, "package-list", root_title)
+        content += "\n\n"
+        root_title = "Host utilities"
+        root_item = [menu for menu in self.config.get_menus()
+            if menu.get_title().lower() == root_title.lower()][0]
+        items = get_symbol_subset(root_item.get_items(), filter=self._is_package)
+        content += self._format_list(root_item, items, "host-package-list", root_title)
+        if not output:
+            print(content)
+            return
+        with open(output, 'w') as fout:
+            fout.write(content)
+
+    def print_deprecated_list(self, output=None):
+        items = get_symbol_subset(self.config.get_top_level_items(),
+            filter=self._is_deprecated)
+        content = self._format_list(None, items, "deprecated", "Deprecated stuff")
+        if not output:
+            print(content)
+            return
+        with open(output, 'w') as fout:
+            fout.write(content)
+
+if __name__ == '__main__':
+    parser = ArgumentParser()
+    parser.add_argument("--package-file", dest="package_file",
+        help="output package list file")
+    parser.add_argument("--deprecated-file", dest="deprecated_file",
+        help="output deprecated list file")
+    parser.add_argument("-n", "--dry-run", dest="dry_run", action='store_true',
+        help="output both package list and deprecated stuff into stdout")
+    args = parser.parse_args()
+    buildroot = Buildroot()
+    if args.dry_run:
+        buildroot.print_package_list()
+        buildroot.print_deprecated_list()
+        sys.exit()
+    if args.package_file:
+        buildroot.print_package_list(output=args.package_file)
+    if args.deprecated_file:
+        buildroot.print_deprecated_list(output=args.deprecated_file)

Hi Samuel,

Quoted 6 lines · Samuel Martin

* add the gen-manual-lists.py script generating the target and host package tables, and the deprecated stuff list as well. These tables and lists are generated parsing the Config.in files;
* add a make target to update these lists in the manual;
* update the these appendices of the manual.

The first two should be a separate patch from the third one. Although,
admittedly, it was pretty convenient to review all of them together.

Even though I have a lot of comments, this patch should be included in
-rc2, otherwise the package list in the manual is inconsistent.

Quoted 10 lines

Signed-off-by Samuel Martin <[email protected]>

---
  docs/manual/appendix.txt            |  13 +-
  docs/manual/deprecated-list.txt     |  81 ++-
  docs/manual/manual.mk               |   7 +
  docs/manual/package-list.txt        | 948
++++++++++++++++++++++++++++++++++++
  docs/manual/pkg-list.txt            | 870

---------------------------------

Any particular reason to rename it?

Quoted 42 lines

  support/scripts/gen-manual-lists.py | 237 +++++++++
  6 files changed, 1233 insertions(+), 923 deletions(-)

create mode 100644 docs/manual/package-list.txt delete mode 100644 docs/manual/pkg-list.txt create mode 100755 support/scripts/gen-manual-lists.py

docs/manual/appendix.txt +3 -10

index ef34169..37548f7 100644
@@ -6,15 +6,8 @@ Appendix
 
  include::makedev-syntax.txt[]
 
-[[package-list]]
-Available packages
-------------------
-// docs/manaual/pkg-list.txt is generated using the following command:
-// $ git grep -E '\((autotools|cmake|generic)-package\)' package/ |  \
-//     cut -d':' -f1 | grep '\.mk$' | \
-//     sed -e 's;.*\?/\(.*\?\).mk;* \1;' | \
-//     sort > docs/manual/pkg-list.txt
-
-include::pkg-list.txt[]
+// autogenerated
+include::package-list.txt[]
 
+// autogenerated
  include::deprecated-list.txt[]

docs/manual/deprecated-list.txt +0 -5

index 6dc87a4..3468219 100644
@@ -1,46 +1,41 @@
-// -*- mode:doc -*- ;
-
-[[deprecated]]
-Deprecated list
----------------

docs/manual/deprecated-list.txt line 8

Quoted 3 lines

docs/manual/deprecated-list.txt +0 -3

-
-The following stuff are marked as _deprecated_ in Buildroot due to
-their status either too old or unmaintained.

It's a pity that this part is gone. I would by the way move this to
appendix.txt: that's cleaner than putting part of the manual text in the python script. Same for the text in package-list.txt.

[snip]

Quoted 9 lines

docs/manual/manual.mk +0 -0

index aa20534..dd07b4e 100644
@@ -24,6 +24,13 @@ $$(O)/docs/$(1)/$(1).$(4): docs/$(1)/$(1).txt $$($(call
 UPPERCASE,$(1))_SOURCES)
          -D $$(@D) $$<
  endef
 

Quoted 8 lines

docs/manual/manual.mk +6 -0

+manual-update-lists:
+       @$(call MESSAGE,"Updating the manual lists...")
+       $(Q)BR2_DEFCONFIG="" srctree=$(TOPDIR) \
+               $(TOPDIR)/support/scripts/gen-manual-lists.py \
+               --package-file $(TOPDIR)/docs/manual/package-list.txt \
+               --deprecated-file

$(TOPDIR)/docs/manual/deprecated-list.txt
+

This new goal should be added to .PHONY.

Quoted 22 lines

################################################################################
# GENDOC -- generates the make targets needed to build asciidoc
documentation.
#

docs/manual/package-list.txt new file +9 -0

new file mode 100644
index 0000000..79d4ec6
@@ -0,0 +1,948 @@
+//
+// Automatically generated list for Buildroot manual.
+// Buildroot 2013.02-00794-g199d6d6-dirty
+// Generation date: 2013-02-11 22:24:44.739087
+//
+
+[[package-list]]
+Package Selection for the target
+--------------------------------

docs/manual/package-list.txt line 13

Quoted 5 lines

docs/manual/package-list.txt new file +4 -0

+
+[width="90%",cols="^1,4",options="header"]
+|===================================================
+| Packages                       | Package Selection for the target ->
...

Nice! Maybe even better if this stands out a bit from the rest, perhaps:

<| Packages | Package Selection for the target -> ...

Quoted 11 lines

+| acl                            | . -> System tools
+| acpid                          | . -> Hardware handling
+| alsa-lib                       | . -> Libraries -> Audio/Sound
+| alsa-utils                     | . -> Audio and video applications
+| alsamixergui                   | . -> Graphic libraries and

applications (graphic/text)
+| applewmproto | . -> Graphic libraries and
applications (graphic/text) -> X11R7 X protocols
+| appres | . -> Graphic libraries and
applications (graphic/text) -> X11R7 Applications

Quoted 12 lines

+| apr                            | . -> Libraries -> Other
+| apr-util                       | . -> Libraries -> Other
+| argp-standalone                | . -> Libraries -> Other
+| argus                          | . -> Networking applications
+| arptables                      | . -> Networking applications
+| at                             | . -> Shell and utilities
+| atk                            | . -> Libraries -> Graphics
+| attr                           | . -> System tools
+| audiofile                      | . -> Libraries -> Audio/Sound
+| aumix                          | . -> Audio and video applications
+| autoconf *(deprecated)*        | . -> Development tools
+| automake *(deprecated)*        | . -> Development tools

Is it intentional to keep the deprecated ones in this list?

[snip]

Quoted 40 lines

support/scripts/gen-manual-lists.py new file +26 -0

new file mode 100755
index 0000000..f2a2dc4
@@ -0,0 +1,237 @@
+#!/usr/bin/env python2
+##
+## gen-manual-lists.py
+##
+## This script generates the following Buildroot manual appendices:
+##  - the package tables (one for the target, the other for host tools);
+##  - the deprecated items.
+##
+## Author(s):
+##  - Samuel Martin <[email protected]>
+##
+## Copyright (C) 2013 Samuel Martin
+##
+## This program is free software; you can redistribute it and/or modify
+## it under the terms of the GNU General Public License as published by
+## the Free Software Foundation; either version 2 of the License, or
+## (at your option) any later version.
+##
+## This program is distributed in the hope that it will be useful,
+## but WITHOUT ANY WARRANTY; without even the implied warranty of
+## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+## GNU General Public License for more details.
+##
+## You should have received a copy of the GNU General Public License
+## along with this program; if not, write to the Free Software
+## Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307

USA

support/scripts/gen-manual-lists.py new file +6 -0

+##
+
+## Note about python2.
+##
+## This script can currently only be run using python2 interpreter due to
+## its kconfiglib dependency (which is not yet python3 friendly).

support/scripts/gen-manual-lists.py line 35

Quoted 3 lines

support/scripts/gen-manual-lists.py new file +3 -0

+
+from __future__ import print_function
+from __future__ import unicode_literals

These future things only exist in recent pythons, and we don't do any
checks for a specific python version... That said, this script isn't supposed to be run by a normal user so I guess it doesn't really matter.

Quoted 19 lines

support/scripts/gen-manual-lists.py new file +17 -0

+
+import os
+import re
+import sys
+import datetime
+from argparse import ArgumentParser
+
+try:
+    import kconfiglib
+except ImportError:
+    message = """
+Could not find the module 'kconfiglib' in the PYTHONPATH:
+"""
+    message += "\n".join(["  {0}".format(path) for path in sys.path])
+    message += """
+
+Make sure the Kconfiglib directory is in the PYTHONPATH, then relaunch

support/scripts/gen-manual-lists.py line 55

Quoted 4 lines

the script.

support/scripts/gen-manual-lists.py new file +3 -0

+
+You can get kconfiglib from:
+  https://github.com/ulfalizer/Kconfiglib

Can we not just dump a copy of kconfiglib in the scripts directory? Or is
it more than one file?

support/scripts/gen-manual-lists.py line 63

Quoted 10 lines

support/scripts/gen-manual-lists.py new file +8 -0

+
+
+"""
+    sys.stderr.write(message)
+    raise
+
+
+_LIST_FORMAT = """\

This symbol would be more appropriate as an attribute of the Buildroot
class.

support/scripts/gen-manual-lists.py line 72

Quoted 9 lines

support/scripts/gen-manual-lists.py new file +9 -0

+//
+// Automatically generated list for Buildroot manual.
+// Buildroot {br_version_full}
+// Generation date: {gen_date}
+//
+
+[[{section_anchor}]]
+{section_title}
+{section_level}

As I briefly mentioned before: I think it's better to just generate the
table with the script, and keep the surrounding text and title in the manually written asciidoc files. That means you'll need three files, but there's no issue with that, is there.

Quoted 9 lines

support/scripts/gen-manual-lists.py new file +6 -0

+
+{table}
+"""
+
+
+def format_asciidoc_table(root, items, get_label, filter=lambda x: True,

sort=True):

I would prefer using different names than standard python functions. So
filterfunc instead of filter. sort is OK because it only exists as an attribute of list objects.

support/scripts/gen-manual-lists.py line 85

Quoted 9 lines

support/scripts/gen-manual-lists.py new file +7 -0

+    def _format_entry(label, parents):
+        return "| {0:<30} | {1}\n".format(label, " -> ".join(parents))
+    def _get_parents(root, item):
+        parent = item.get_parent()
+        parents = []
+        while parent and parent != root:
+            if parent.is_menu():

I guess the parent can be either a menu or a choice. Seems a good idea to
me to keep choices in the path as well.

support/scripts/gen-manual-lists.py line 89

Quoted 6 lines

support/scripts/gen-manual-lists.py new file +4 -0

+                parents += [parent.get_title()]
+            parent = parent.get_parent()
+        parents.reverse()
+        return parents

An empty line here would be nice.

support/scripts/gen-manual-lists.py line 98

Quoted 9 lines

support/scripts/gen-manual-lists.py new file +9 -0

+    lines = []
+    for item in items:
+        if not filter(item):
+            continue
+        if not item.is_symbol() or not item.prompts:
+            continue
+        loc = [] if root is None else ["."]
+        loc += _get_parents(root, item)
+        lines += [_format_entry(get_label(item), loc)]

lines.append(...) would be more appropriate.

Quoted 4 lines

support/scripts/gen-manual-lists.py new file +4 -0

+    if sort:
+        lines.sort(key=lambda x:x.lower())
+    if root:
+        loc_label = _get_parents(None, root) + [root.get_title(), "..."]

support/scripts/gen-manual-lists.py line 112

Quoted 10 lines

support/scripts/gen-manual-lists.py new file +10 -0

+    else:
+        loc_label = ["Location"]
+    lines.insert(0, _format_entry("Packages", loc_label))
+    table = "[width=\"90%\",cols=\"^1,4\",options=\"header\"]\n"
+    table += "|===================================================\n"
+    table += "".join(lines)
+    table += "|===================================================\n"
+    return table
+
+def get_symbol_subset(items, filter):

Same remark about filter here.

support/scripts/gen-manual-lists.py line 120

Quoted 8 lines

support/scripts/gen-manual-lists.py new file +8 -0

+    subset = []
+    for item in items:
+        if item.is_symbol():
+            if not item.prompts:
+                continue
+            if not filter(item):
+                continue
+            subset += [item]

subset.append(item)

support/scripts/gen-manual-lists.py line 123

Quoted 3 lines

support/scripts/gen-manual-lists.py new file +3 -0

+        elif item.is_menu() or item.is_choice():
+            subset += get_symbol_subset(item.get_items(), filter)
+    return subset

This function is an ideal candidate for an iterator. But iterators are a
bit tricky to understand so I don't mind keeping it as it is.

support/scripts/gen-manual-lists.py line 133

Quoted 12 lines

support/scripts/gen-manual-lists.py new file +10 -0

+
+
+class Buildroot:
+    root_config = "Config.in"
+    package_dirname = "package"
+    package_prefixes = ["BR2_PACKAGE_", "BR2_PACKAGE_HOST_"]
+    deprecated_symbol = "BR2_DEPRECATED"
+
+    def __init__(self):
+        self.base_dir = os.environ.get("srctree")

Why not use the TOPDIR variable?

Quoted 24 lines

+ self.package_dir = os.path.join(self.base_dir,
self.package_dirname)
+ self.config = kconfiglib.Config(os.path.join(self.base_dir,
self.root_config))

support/scripts/gen-manual-lists.py new file +10 -0

+        self.package_symbols = []
+        self._fill_package_symbols()
+        self._deprecated = self.config.get_symbol(self.deprecated_symbol)
+        self.br_version_full = os.environ.get("BR2_VERSION_FULL")
+        self.generation_date = str(datetime.datetime.utcnow())
+
+    def _get_package_symbol(self, package_name):
+        pkg_symbol = re.sub("[-+.]", "_", package_name)
+        pkg_symbol = pkg_symbol.upper()
+        pkg_symbols = [prefix + pkg_symbol for prefix in

self.package_prefixes]

support/scripts/gen-manual-lists.py new file +7 -0

+        return pkg_symbols
+
+    def _fill_package_symbols(self):
+        for root, _, files in os.walk(self.package_dir):
+            pkg_name = os.path.basename(root)
+            for file_ in files:
+                if not file_.endswith(".mk") or not pkg_name ==

Quoted 4 lines

file_[:-3]:

support/scripts/gen-manual-lists.py new file +2 -0

+                    continue
+                self.package_symbols +=

self._get_package_symbol(pkg_name)

I don't really like this. Wouldn't it be possible to construct the package
name from the location of the Config.in where it is defined?

And check for it directly in _is_package instead of pre-constructing the list?

Quoted 4 lines

support/scripts/gen-manual-lists.py new file +2 -0

+
+    def _is_deprecated(self, symbol):

support/scripts/gen-manual-lists.py line 166

Quoted 10 lines

support/scripts/gen-manual-lists.py new file +10 -0

+        return self._deprecated in symbol.get_referenced_symbols()
+
+    def _is_from_package(self, symbol):
+        if hasattr(symbol, 'get_def_locations'):
+            def_loc = symbol.get_def_locations()[0][0]
+        elif hasattr(symbol, 'get_location'):
+            def_loc = symbol.get_location()[0]
+        else:
+            return False
+        return self.package_dir in def_loc

This function isn't used, is it?

Quoted 9 lines

support/scripts/gen-manual-lists.py new file +5 -0

+
+    def _is_package(self, symbol):
+        # return symbol.get_name().startswith(self.package_prefixes[0])
+        return symbol.get_name() in self.package_symbols
+        pkg_name =

os.path.basename(os.path.dirname(symbol.get_def_locations()[0][0]))
+ return symbol.get_name() in self._get_package_symbol(pkg_name)

The final commit shouldn't contain three different versions :-)

Quoted 22 lines

support/scripts/gen-manual-lists.py new file +9 -0

+
+    def _get_symbol_label(self, item):
+        label = item.prompts[0][0]
+        if self._is_deprecated(item):
+            label += " *(deprecated)*"
+        return label
+
+    def _format_list(self, root, items, section_anchor, section_title):
+        table = format_asciidoc_table(root, items,

get_label=self._get_symbol_label)

support/scripts/gen-manual-lists.py new file +10 -0

+        content = _LIST_FORMAT.format(
+            br_version_full=self.br_version_full,
+            gen_date=self.generation_date,
+            section_anchor=section_anchor,
+            section_title=section_title,
+            section_level=re.sub(".", "-", section_title),
+            table=table
+            )
+        return content
+

support/scripts/gen-manual-lists.py line 195

Quoted 4 lines

support/scripts/gen-manual-lists.py new file +4 -0

+    def print_package_list(self, output=None):
+        root_title = "Package Selection for the target"
+        root_item = [menu for menu in self.config.get_menus()
+            if menu.get_title().lower() == root_title.lower()][0]

This could get better error handling, and the list construct makes it less
readable than a simple for loop.