manual: remove lists of packages (branch yem/manual-no-package-list)

Hello All!

This series removes the list of packages from the manual

Those are huges, take up more than half of the manual and do not have much value in the manual.

The only list tht kinda makes sense is the list iof virtual packages, their symbols and their providers, to help in quickly identifying them (for both users and packagers).

We replace the biggish and complex script that generates those lists with a much simpler one that is targeted at only virtual packages.

The new script does not parse the Kconfig structure and uses a heuristic to detect virtual packages and their providers: just grep the source tree.

However, that turned up two discrepancies in how we define two virtual packages, cryptodev and jpeg; their providers did not properly declare themselves as providers.

We fix that, so the heuristic works for those two and their providers as well.

Changes v1 -> v2:
- keep the list of virtual packages
- fix cryptodev and jpeg providers

Regards,
Yann E. MORIN.

The following changes since commit 15cf25114a4918bab0bf7a5d5cd198f864c02929

agentpp: bump to 4.0.6 (2016-09-02 23:05:35 +0200)

are available in the git repository at:

https://gitlab.com/ymorin/buildroot.git

for you to fetch changes up to 206a0e9bf7a1e99fa176fe1345d245881f14b858

manual: get rid of the lists of packages (2016-09-04 00:38:22 +0200)

----------------------------------------------------------------
Yann E. MORIN (4):
package/cryptodev: properly declare providers package/openssl: depend on the virtual cryptodev package package/jpeg: properly declare providers
manual: get rid of the lists of packages

 docs/manual/appendix.txt                   |   32 +-
 docs/manual/manual.mk                      |   21 +-
 package/cryptodev-linux/cryptodev-linux.mk |    2 +
 package/jpeg-turbo/jpeg-turbo.mk           |    2 +
 package/libjpeg/libjpeg.mk                 |    1 +
 package/ocf-linux/ocf-linux.mk             |    2 +
 package/openssl/openssl.mk                 |    9 +-
 support/scripts/gen-manual-lists           |   62 +
 support/scripts/gen-manual-lists.py        |  513 ----
 support/scripts/kconfiglib.py              | 3541 ----------------------------
 support/scripts/readme.kconfiglib          |   30 -
 11 files changed, 76 insertions(+), 4139 deletions(-)
 create mode 100755 support/scripts/gen-manual-lists
 delete mode 100644 support/scripts/gen-manual-lists.py
 delete mode 100644 support/scripts/kconfiglib.py
 delete mode 100644 support/scripts/readme.kconfiglib

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

cryptodev-linux and ocf-linux are providers of the cryptodev virtual package, so they must declare themselves as such.

We do not have any problem for now, as the only consumer (openssl) of both cryptodev-linux and ocf-linux did not depend on the virtual package but on each provider. The other consumer is gnutls, but has only a dependency on cryptodev-linux.

However, openssl can be turned into a proper consumer of the virtual package, which would trigger the bug.

For the sake of correctness, make both cryptodev-linux and ocf-linux declare themselves as providers for the cryptodev virtual package.

---
 package/cryptodev-linux/cryptodev-linux.mk | 2 ++
 package/ocf-linux/ocf-linux.mk             | 2 ++
 2 files changed, 4 insertions(+)

index c3f5355..d23b647 100644
@@ -10,6 +10,8 @@ CRYPTODEV_LINUX_INSTALL_STAGING = YES
 CRYPTODEV_LINUX_LICENSE = GPLv2+
 CRYPTODEV_LINUX_LICENSE_FILES = COPYING
 
+CRYPTODEV_LINUX_PROVIDES = cryptodev
+
 define CRYPTODEV_LINUX_MODULE_GEN_VERSION_H
 	$(MAKE) -C $(@D) version.h
 endef

index 35968d5..01854fa 100644
@@ -8,6 +8,8 @@ OCF_LINUX_VERSION = 20120127
 OCF_LINUX_SITE = http://downloads.sourceforge.net/project/ocf-linux/ocf-linux/$(OCF_LINUX_VERSION)
 OCF_LINUX_INSTALL_STAGING = YES
 
+OCF_LINUX_PROVIDES = cryptodev
+
 OCF_LINUX_MODULE_SUBDIRS = ocf
 OCF_LINUX_MODULE_MAKE_OPTS = \
 	CONFIG_OCF_OCF=m \

2.7.4

Instead of depending on each provider of cryptodev, make openssl depend on the virtual package.

This is easy because in both cases the openssl build systems needs the same configuration options.

---
 package/openssl/openssl.mk | 9 ++-------
 1 file changed, 2 insertions(+), 7 deletions(-)

index b926fe3..efe2b49 100644
@@ -28,14 +28,9 @@ ifeq ($(BR2_USE_MMU),)
 OPENSSL_CFLAGS += -DHAVE_FORK=0
 endif
 
-ifeq ($(BR2_PACKAGE_CRYPTODEV_LINUX),y)
+ifeq ($(BR2_PACKAGE_HAS_CRYPTODEV),y)
 OPENSSL_CFLAGS += -DHAVE_CRYPTODEV -DUSE_CRYPTODEV_DIGESTS
-OPENSSL_DEPENDENCIES += cryptodev-linux
-endif
-
-ifeq ($(BR2_PACKAGE_OCF_LINUX),y)
-OPENSSL_CFLAGS += -DHAVE_CRYPTODEV -DUSE_CRYPTODEV_DIGESTS
-OPENSSL_DEPENDENCIES += ocf-linux
+OPENSSL_DEPENDENCIES += cryptodev
 endif
 
 # Some architectures are optimized in OpenSSL

2.7.4

libjpeg and jpeg-turo are providers of the jpeg virtual package, so they must declare themselves as such.

We do not have any problem, because the dependency is computed from the Kconfig-defined option BR2_PACKAGE_PROVIDES_JPEG, and we do not check the Makefile-defined <FOO>_PROVIDES variable.

For the sake of correctness, make both libjpeg and jpeg-turo declare themselves as providers for the jpeg virtual package.

---
 package/jpeg-turbo/jpeg-turbo.mk | 2 ++
 package/libjpeg/libjpeg.mk       | 1 +
 2 files changed, 3 insertions(+)

index 047e925..454b648 100644
@@ -10,6 +10,8 @@ JPEG_TURBO_SITE = http://downloads.sourceforge.net/project/libjpeg-turbo/$(JPEG_
 JPEG_TURBO_LICENSE = jpeg-license (BSD-3c-like)
 JPEG_TURBO_LICENSE_FILES = README README-turbo.txt
 JPEG_TURBO_INSTALL_STAGING = YES
+JPEG_TURBO_PROVIDES = jpeg
+
 JPEG_TURBO_CONF_OPTS = --with-jpeg8
 
 ifeq ($(BR2_X86_CPU_HAS_MMX)$(BR2_ARM_CPU_HAS_NEON),y)

index 14aba61..b8f4bac 100644
@@ -10,6 +10,7 @@ LIBJPEG_SOURCE = jpegsrc.v$(LIBJPEG_VERSION).tar.gz
 LIBJPEG_LICENSE = jpeg-license (BSD-3c-like)
 LIBJPEG_LICENSE_FILES = README
 LIBJPEG_INSTALL_STAGING = YES
+LIBJPEG_PROVIDES = jpeg
 
 define LIBJPEG_REMOVE_USELESS_TOOLS
 	rm -f $(addprefix $(TARGET_DIR)/usr/bin/,cjpeg djpeg jpegtran rdjpgcom wrjpgcom)

2.7.4

We currently have four lists of packages in the manual:
- the non-virtual target packages,
- the virtual target packages,
- the host packages,
- the deprecated features.

Those list take more than half of the manual. They do not serve much purpose except to show off. The only one that is interesting is the list of virtual packages, the associated symbols and their providers.

After the recent discussion on the list [0], remove them, except the list of virtual packages.

Replace our biggish and complex generating script (and its companion library kconfiglib) with a much simpler script.

The drawback is that, when it is a sub-option of a provider that does provide the feature, we loose that information in the new list. We believe this is an acceptable drawback.

[0] http://lists.busybox.net/pipermail/buildroot/2016-September/171199.html

---
 docs/manual/appendix.txt            |   32 +-
 docs/manual/manual.mk               |   21 +-
 support/scripts/gen-manual-lists    |   62 +
 support/scripts/gen-manual-lists.py |  513 -----
 support/scripts/kconfiglib.py       | 3541 -----------------------------------
 support/scripts/readme.kconfiglib   |   30 -
 6 files changed, 67 insertions(+), 4132 deletions(-)
 create mode 100755 support/scripts/gen-manual-lists
 delete mode 100644 support/scripts/gen-manual-lists.py
 delete mode 100644 support/scripts/kconfiglib.py
 delete mode 100644 support/scripts/readme.kconfiglib

index 87a20bd..0c7b165 100644
@@ -4,39 +4,11 @@
 include::makedev-syntax.txt[]
 include::makeusers-syntax.txt[]
 
-
-// Automatically generated lists:
-
-[[package-list]]
-== List of target packages available in Buildroot
-
-include::package-list.txt[]
-
 [[virtual-package-list]]
 == List of virtual packages
 
 These are the virtual packages known to +Buildroot+, with the
-corresponding symbols and providers.
+corresponding symbols and providers (some may need sub-options
+to be enabled).
 
 include::virtual-package-list.txt[]
-
-[[host-package-list]]
-== List of host utilities available in Buildroot
-
-The following packages are all available in the menu +Host utilities+.
-
-include::host-package-list.txt[]
-
-[[deprecated-list]]
-== Deprecated features
-
-The following features are marked as _deprecated_ in Buildroot due to
-them being either too old or unmaintained. They will be removed at
-some point, so stop using them.
-Each deprecated symbol in kconfig depends on a symbol
-+BR2_DEPRECATED_SINCE_xxxx_xx+, which provides an indication of when
-the feature can be removed: features will not be removed within the
-year following deprecation. For example, a symbol depending on
-+BR2_DEPRECATED_SINCE_2013_05+ can be removed from 2014.05 onwards.
-
-include::deprecated-list.txt[]

index b26f912..4ee28f1 100644
@@ -7,28 +7,13 @@
 MANUAL_SOURCES = $(sort $(wildcard docs/manual/*.txt) $(wildcard docs/images/*))
 MANUAL_RESOURCES = $(TOPDIR)/docs/images
 
-# Ensure the kconfig snippet is generated (for MANUAL_GEN_LISTS, below):
-MANUAL_DEPENDENCIES += prepare-kconfig
-
 # Our manual needs to generate lists
 define MANUAL_GEN_LISTS
 	$(Q)$(call MESSAGE,"Updating the manual lists...")
-	$(Q)$(COMMON_CONFIG_ENV) \
-		BR2_DEFCONFIG="" \
-		TOPDIR=$(TOPDIR) \
-		O=$(@D) \
-		python -B $(TOPDIR)/support/scripts/gen-manual-lists.py
+	$(Q)support/scripts/gen-manual-lists \
+		-d $(TOPDIR)/package \
+		-o $(@D)/virtual-package-list.txt
 endef
 MANUAL_POST_RSYNC_HOOKS += MANUAL_GEN_LISTS
 
-# Our list-generating script requires argparse
-define MANUAL_CHECK_LISTS_DEPS
-	$(Q)if ! python -c "import argparse" >/dev/null 2>&1 ; then \
-		echo "You need python with argparse on your host to generate" \
-			"the list of packages in the manual"; \
-		exit 1; \
-	fi
-endef
-MANUAL_CHECK_DEPENDENCIES_HOOKS += MANUAL_CHECK_LISTS_DEPS
-
 $(eval $(call asciidoc-document))

deleted file mode 100644
index 4b653b8..0000000

2.7.4

Part of the diff is not shown here. View the raw message.

Hello,

I would like to challenge the fact that the list of virtual packages is more useful than the rest of the list.

virtual packages are purely internal things, you only need to know about virtual packages when you're adding new packages in Buildroot. And when you're adding new packages in Buildroot, you have far enough experience to do a "git grep virtual-package" and find which ones are available.

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

Thomas, All,

On 2016-09-04 15:20 +0200, Thomas Petazzoni spake thusly:

Of course, I do not agree! ;-)

But do I care enough to argue? Certainly not. I tend to be less interested in making things easy/easier for the users nowadays... :-/

I'll send an updated patch that removes all the lists altogether.

Regards,
Yann E. MORIN.

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

Hello,

Well, let's hear what the others have to say about this before taking a decision.

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

Hello,

Nobody really gave any opinion. I had some quick discussion with Peter in private on IRC, and he seemed to be in favor of completely dropping the list of packages from the manual.

So: patch applied. Thanks a lot!

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

Hi,

Yes, indeed - Sorry, I didn't apply the patch yet, I got caught up in real life stuff (tm).

Bye, Peter Korsgaard

Hello,

Those three patches applied to master. Thanks!

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com