Skip to contentSkip to conversation

documentation audit script

PATCH Applied Scott Garman 2 participants 5 messages Oct 25, 2011 – Nov 2, 2011

lore
Share & tools
lore
#0 lore raw

cover documentation audit script

Hello,

I've been using this script to perform audits of which packages are producing documentation, and feel it's long overdue for this to be added to the git tree. So here it is.

The following changes since commit f586aaa8d00361a9597a546d665077c75cf4d520:

libxml-parser-perl, libxml-simple-perl, expat, sgmlspl-native, git: bump PR to rebuild after perl upgrade (2011-10-25 08:36:01 +0100)

are available in the git repository at:
git://git.yoctoproject.org/poky-contrib sgarman/docaudit-script
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=sgarman/docaudit-script

Scott Garman (1):
documentation-audit.sh: script for auditing documentation build
status

 scripts/contrib/documentation-audit.sh |   92 ++++++++++++++++++++++++++++++++
 1 files changed, 92 insertions(+), 0 deletions(-)
 create mode 100755 scripts/contrib/documentation-audit.sh
Signature · 1 line
1.7.5.4
#1 lore raw

PATCH 1/1 documentation-audit.sh: script for auditing documentation build status

This script is used to enumerate which recipes are building documentation. It does this by checking that a -doc package gets generated and contains files.

The script works by building each recipe using the output from bitbake -s.

It will generate several report files, listing which recipes include documentation, which are missing documentation, and which did not successfully build at all.

---
 scripts/contrib/documentation-audit.sh |   92 ++++++++++++++++++++++++++++++++
 1 files changed, 92 insertions(+), 0 deletions(-)
 create mode 100755 scripts/contrib/documentation-audit.sh
scripts/contrib/documentation-audit.sh new file +92 -0
new file mode 100755
index 0000000..5070fee
@@ -0,0 +1,92 @@
+#!/bin/bash
+#
+# Perform an audit of which packages provide documentation and which
+# are missing -doc packages.
+#
+# Setup requirements: be sure to be building for MACHINE=qemux86. Run
+# this script after source'ing the build environment script, so you're
+# running it from build/ directory.
+#
+# Maintainer: Scott Garman <[email protected]>
+
+REPORT_DOC_SIMPLE="documentation_exists.txt"
+REPORT_DOC_DETAIL="documentation_exists_detail.txt"
+REPORT_MISSING_SIMPLE="documentation_missing.txt"
+REPORT_MISSING_DETAIL="documentation_missing_detail.txt"
+REPORT_BUILD_ERRORS="build_errors.txt"
+
+rm -rf $REPORT_DOC_SIMPLE $REPORT_DOC_DETAIL $REPORT_MISSING_SIMPLE $REPORT_MISSING_DETAIL
+
+BITBAKE=`which bitbake`
+if [ -z "$BITBAKE" ]; then
+	echo "Error: bitbake command not found."
+	echo "Did you forget to source the build environment script?"
+	exit 1
+fi
+
+echo "REMINDER: you need to build for MACHINE=qemux86 or you won't get useful results"
+echo "REMINDER: you need to have COMMERCIAL_LICENSE = \"\" in local.conf or you'll get false positives"
+
+for pkg in `bitbake -s | awk '{ print \$1 }'`; do
+	if [[ "$pkg" == "Loading" || "$pkg" == "Loaded" ||
+          "$pkg" == "Parsing" || "$pkg" == "Package" ||
+          "$pkg" == "NOTE:"   || "$pkg" == "WARNING:" ||
+          "$pkg" == "done."   || "$pkg" == "============" ]]
+	then
+		# Skip initial bitbake output
+		continue
+	fi
+	if [[ "$pkg" =~ -native$ || "$pkg" =~ -nativesdk$ ||
+          "$pkg" =~ -cross-canadian ]]; then
+		# Skip native/nativesdk/cross-canadian recipes
+		continue
+	fi
+	if [[ "$pkg" =~ ^meta- || "$pkg" =~ ^task- || "$pkg" =~ -image ]]; then
+		# Skip meta, task and image recipes
+		continue
+	fi
+	if [[ "$pkg" =~ ^glibc- || "$pkg" =~ ^libiconv$ ||
+          "$pkg" =~ -toolchain$ || "$pkg" =~ ^package-index$ ||
+          "$pkg" =~ ^linux- || "$pkg" =~ ^adt-installer$ ||
+          "$pkg" =~ ^eds-tools$ || "$pkg" =~ ^external-python-tarball$ ||
+          "$pkg" =~ ^qt4-embedded$ || "$pkg" =~ ^qt-mobility ]]; then
+		# Skip glibc, libiconv, -toolchain, and other recipes known
+		# to cause build conflicts or trigger false positives.
+		continue
+	fi	
+
+	echo "Building package $pkg..."
+	bitbake $pkg > /dev/null
+	if [ $? -ne 0 ]; then
+		echo "There was an error building package $pkg" >> "$REPORT_MISSING_DETAIL"
+		echo "$pkg" >> $REPORT_BUILD_ERRORS
+
+		# Do not skip the remaining tests, as sometimes the
+		# exit status is 1 due to QA errors, and we can still
+		# perform the -doc checks.
+	fi
+
+	echo "$pkg built successfully, checking for a documentation package..."
+	WORKDIR=`bitbake -e $pkg | grep ^WORKDIR | awk -F '=' '{ print \$2 }' | awk -F '"' '{ print \$2 }'`
+	FIND_DOC_PKG=`find $WORKDIR/packages-split/*-doc -maxdepth 0 -type d`
+	if [ -z "$FIND_DOC_PKG" ]; then
+		# No -doc package was generated:
+		echo "No -doc package: $pkg" >> "$REPORT_MISSING_DETAIL"
+		echo "$pkg" >> $REPORT_MISSING_SIMPLE
+		continue
+	fi
+
+	FIND_DOC_FILES=`find $FIND_DOC_PKG -type f`
+	if [ -z "$FIND_DOC_FILES" ]; then
+		# No files shipped with the -doc package:
+		echo "No files shipped with the -doc package: $pkg" >> "$REPORT_MISSING_DETAIL"
+		echo "$pkg" >> $REPORT_MISSING_SIMPLE
+		continue
+	fi
+
+	echo "Documentation shipped with $pkg:" >> "$REPORT_DOC_DETAIL"
+	echo "$FIND_DOC_FILES" >> "$REPORT_DOC_DETAIL"
+	echo ""	>> "$REPORT_DOC_DETAIL"
+
+	echo "$pkg" >> "$REPORT_DOC_SIMPLE"
+done
Signature · 1 line
1.7.5.4
#2 lore raw
In reply to Scott Garman
Quoted 14 lines · Scott Garman

Hello,

I've been using this script to perform audits of which packages are producing documentation, and feel it's long overdue for this to be added to the git tree. So here it is.

The following changes since commit f586aaa8d00361a9597a546d665077c75cf4d520:

libxml-parser-perl, libxml-simple-perl, expat, sgmlspl-native, git: bump PR to rebuild after perl upgrade (2011-10-25 08:36:01 +0100)

are available in the git repository at:
git://git.yoctoproject.org/poky-contrib sgarman/docaudit-script
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=sgarman/docaudit-script

Quoted 8 lines

Scott Garman (1):
documentation-audit.sh: script for auditing documentation build
status

  scripts/contrib/documentation-audit.sh |   92 ++++++++++++++++++++++++++++++++
  1 files changed, 92 insertions(+), 0 deletions(-)

create mode 100755 scripts/contrib/documentation-audit.sh

Can you please run this and generate a report.

Thanks
Sau!

#3 lore raw
Quoted 22 lines · Saul Wold
Quoted 21 lines · Scott Garman

Hello,

I've been using this script to perform audits of which packages are producing documentation, and feel it's long overdue for this to be added to the git tree. So here it is.

The following changes since commit
f586aaa8d00361a9597a546d665077c75cf4d520:

libxml-parser-perl, libxml-simple-perl, expat, sgmlspl-native, git:
bump PR to rebuild after perl upgrade (2011-10-25 08:36:01 +0100)

are available in the git repository at:
git://git.yoctoproject.org/poky-contrib sgarman/docaudit-script
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=sgarman/docaudit-script

Scott Garman (1):
documentation-audit.sh: script for auditing documentation build status

Quoted 7 lines
Quoted 5 lines

scripts/contrib/documentation-audit.sh | 92
++++++++++++++++++++++++++++++++

1 files changed, 92 insertions(+), 0 deletions(-)

create mode 100755 scripts/contrib/documentation-audit.sh

Can you please run this and generate a report.

I did this and sent the report to the yocto ML on 10/26.

Scott

Signature · 3 lines
Scott Garman
Embedded Linux Engineer - Yocto Project
Intel Open Source Technology Center
Applied
#4 lore raw
In reply to Scott Garman
Quoted 14 lines · Scott Garman

Hello,

I've been using this script to perform audits of which packages are producing documentation, and feel it's long overdue for this to be added to the git tree. So here it is.

The following changes since commit f586aaa8d00361a9597a546d665077c75cf4d520:

libxml-parser-perl, libxml-simple-perl, expat, sgmlspl-native, git: bump PR to rebuild after perl upgrade (2011-10-25 08:36:01 +0100)

are available in the git repository at:
git://git.yoctoproject.org/poky-contrib sgarman/docaudit-script
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=sgarman/docaudit-script

Quoted 8 lines

Scott Garman (1):
documentation-audit.sh: script for auditing documentation build
status

  scripts/contrib/documentation-audit.sh |   92 ++++++++++++++++++++++++++++++++
  1 files changed, 92 insertions(+), 0 deletions(-)

create mode 100755 scripts/contrib/documentation-audit.sh

Merged into OE-Core

Thanks
Sau!