Skip to contentSkip to conversation

Make source code browsable locally

PATCH Superseded Tobias Hagelborn 1 participant 3 messages Mar 24, 2017

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover Make source code browsable locally

In our organization, there is a demand to be able download source code for searching.
With the use of sstate-cache this has been challenge. Also, the source is not gathered but rather spread in a larger directory structure together with build output. There are probably others that also want to be able to do the same.

This addition does:
* Force download of all source via the srctree_all task
* Symlink the source together in a separate directory structure for easier search without the risk of matching with build output.

This is achived with a new class srctree.bbclass that implements the needed tasks and a devtool command srctree which offers a convenience command to set up source for a recipe and it's dependencies.

There is also a way of organising the package source code in your own way.
This is done by implementing a custom dir generation function and assign to SRCTREE_PN_PATH in a custom class and set SRCTREE_EXTENSION in a configuration file.

Tobias Hagelborn (2):
srctree.bbclass: Download sourcecode and make it searchable
devtool: srctree: New command for setting up browsable source-code

 meta/classes/srctree.bbclass   | 92 ++++++++++++++++++++++++++++++++++++++++++
 scripts/lib/devtool/srctree.py | 87 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 179 insertions(+)
 create mode 100644 meta/classes/srctree.bbclass
 create mode 100644 scripts/lib/devtool/srctree.py
Signature · 1 line
2.1.4
#1 lore raw

PATCH 1/2 srctree.bbclass: Download sourcecode and make it searchable

This class is used to create a symlink to the source in ${S} in a separate directory structure. ${SECTION} is used, if available, for naming the path.

Intended for 'devtool srctree <recipe>' to set up a tree with the source for all specified recipes and their dependencies to be able to do a more convenient local search in source code.

Example:

nfs-utils (console/network) maps to ->
console/
network/
nfs-utils

---
 meta/classes/srctree.bbclass | 92 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 92 insertions(+)
 create mode 100644 meta/classes/srctree.bbclass
meta/classes/srctree.bbclass new file +92 -0
new file mode 100644
index 0000000..53df3c6
@@ -0,0 +1,92 @@
+# Create symlinks to source (${S}) in a separate directory structure.
+#
+# Copyright (C) 2016-2017 Axis Communications AB
+# Author: Tobias Hagelborn <[email protected]>
+#
+# Released under the MIT license (see COPYING.MIT for the terms)
+#
+# This class is used to create a symlink to the source in ${S} in a
+# separate directory structure. ${SECTION} is used, if available, for
+# naming the path.
+#
+# Intended for 'devtool srctree <recipe>' to set up a tree with the
+# source for all specified recipes and their dependencies to be able
+# to do a more convenient local search in source code.
+#
+# Example:
+#
+# nfs-utils (console/network) maps to ->
+#   console/
+#     network/
+#       nfs-utils
+
+# Use the following to extend the srctree class with custom functions
+SRCTREE_EXTENSION ?= ""
+
+inherit ${SRCTREE_EXTENSION}
+
+# Default PN sub-path function
+def srctree_get_path(d):
+    section = d.getVar("SECTION", True)
+    if section:
+        return section.split()[0]
+    else:
+        d.getVar("BPN", True)
+
+# Configurable browsable source tree root
+SRCTREE_DIR ?= "${TOPDIR}/src"
+
+# Path within ${SRCTREE_DIR} for the current PN. Can be overridden in
+# a custom class via ${SRCTREE_EXTENSION} or set to a fixed value.
+# Example: SRCTREE_PN_PATH = "${PN}"
+SRCTREE_PN_PATH ?= "${@srctree_get_path(d)}"
+SYM_S = "${SRCTREE_DIR}/${SRCTREE_PN_PATH}"
+
+create_srctree_readme() {
+	readme="${SRCTREE_DIR}/README"
+
+	[ -e "$readme" ] || cat >"$readme" <<EOF
+This directory contains a directory tree for source code browsing.
+
+IMPORTANT:
+
+ * The source tree is intended for CODE BROWSING ONLY.
+ * The tree contains symbolic links directly to the source as used by BitBake.
+ * Any CHANGES made to the source code MAY BE REMOVED by BitBake AT ANY TIME.
+ * BitBake will NOT automatically rebuild packages due to code changes made
+   directly in the source tree (which you should not do in the first place).
+ * The source tree will NOT reflect changes made to recipes or, e.g., by
+   running "devtool modify -x <recipe>" until "devtool srctree" is re-executed.
+EOF
+}
+
+do_srctree[nostamp] = "1"
+do_srctree[dirs] = "${SRCTREE_DIR}"
+addtask srctree after do_patch
+do_srctree() {
+	if [ "${S}" = "${WORKDIR}" ]; then
+		[ "${SRCTREE_VERBOSE}" != 1 ] ||
+			bbwarn "Not setting up a link for '${SRCTREE_PN_PATH}' (since $""{S} = $""{WORKDIR})"
+		return
+	fi
+
+	# Ignore all cases where S is not properly set up for any reason
+	if [ ! -d "${S}" ]; then
+		[ "${SRCTREE_VERBOSE}" != 1 ] ||
+			bbwarn "No source available for ${PN}";
+		return
+	fi
+
+	mkdir -p $(dirname "${SYM_S}")
+	ln -sfn ${S} ${SYM_S}
+
+	create_srctree_readme
+}
+
+addtask srctree_all after do_srctree
+do_srctree_all[recrdeptask] = "do_srctree_all do_srctree"
+do_srctree_all[recideptask] = "do_${BB_DEFAULT_TASK}"
+do_srctree_all[doc] = "Setup source tree recursivly"
+do_srctree_all() {
+	:
+}
Signature · 1 line
2.1.4
#2 lore raw

PATCH 2/2 devtool: srctree: New command for setting up browsable source-code

In reply to Tobias Hagelborn

Setup a directory tree for source code according to section name. (if known).

This tree is intended for code browsing ONLY.
If source is moved, for instance with devtool modify, this command should be rerun to be up-to-date with that move.

---
 scripts/lib/devtool/srctree.py | 87 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 87 insertions(+)
 create mode 100644 scripts/lib/devtool/srctree.py
scripts/lib/devtool/srctree.py new file +87 -0
new file mode 100644
index 0000000..dbf1608
@@ -0,0 +1,87 @@
+# Development tool - srctree command plugin
+#
+# Copyright (C) 2016-2017 Axis Communications AB
+# Author: Tobias Hagelborn <[email protected]>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+
+"""Devtool srctree plugin"""
+
+import os
+import logging
+import tempfile
+import bb
+
+from devtool import exec_build_env_command
+
+logger = logging.getLogger('devtool')
+
+
+def _create_conf_file(values, conf_file=None):
+    """ Create bitbake config for use with srctree devtool command """
+    if not conf_file:
+        _, conf_file = tempfile.mkstemp(suffix='.conf')
+    elif not os.path.exists(os.path.dirname(conf_file)):
+        logger.debug("Creating folder %s", os.path.dirname(conf_file))
+        bb.utils.mkdirhier(os.path.dirname(conf_file))
+    if values:
+        with open(conf_file, 'w') as f:
+            for key, value in values.items():
+                f.write('%s = "%s"\n' % (key, value))
+    return conf_file
+
+
+#pylint: disable=unused-argument
+def srctree(args, config, basepath, workspace):
+    """Entry point for the devtool 'srctree' subcommand"""
+
+    values = {}
+    if args.destination:
+        values['SRCTREE_DIR'] = os.path.abspath(args.destination)
+        logger.info('Setting up srctree in: %s', values['SRCTREE_DIR'])
+    if args.verbose:
+        values['SRCTREE_VERBOSE'] = "1"
+    conf_file = _create_conf_file(values)
+
+    with open(conf_file, 'a') as f:
+        f.write('INHERIT += "srctree"\n')
+
+    workdir = basepath
+    exec_build_env_command(
+        config.init_path, workdir,
+        'bitbake -k -R %s -c srctree_all %s' % (conf_file, ' '.join(args.recipename)),
+        watch=True)
+    os.unlink(conf_file)
+
+
+def register_commands(subparsers, context):
+    """Register devtool subcommands from this plugin"""
+    parser_search = subparsers.add_parser(
+        'srctree',
+        help='Setup a directory tree for source code browsing',
+        description='Setup a directory tree for source code according to section name (if known). '
+                    'This tree is intended for code browsing ONLY. '
+                    'If source is moved, for instance with devtool modify, '
+                    'this command should be rerun to be up-to-date with that move. '
+                    'Please read the README file that is created together with '
+                    'the source tree for more information.')
+    parser_search.add_argument('recipename', nargs='+',
+                               help='Recipe to use. Multiple recipes are accepted. '
+                               'Example: core-image-minimal')
+    parser_search.add_argument('-D', '--destination',
+                               help='Destination root folder. Default: ${TOPDIR}/src.')
+    parser_search.add_argument('-v', '--verbose', action="store_true",
+                               help='Be more verbose.')
+    parser_search.set_defaults(func=srctree)
Signature · 1 line
2.1.4