RFC: new 'yocto-layer' tool for creating generic Yocto layers

From: Tom Zanussi <[email protected]>

Implementation of the 'yocto-layer' command-line tool, for creating generic layers and listing their input properties.

Signed-off-by Tom Zanussi <[email protected]>

---
 scripts/yocto-layer | 142 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 142 insertions(+)
 create mode 100755 scripts/yocto-layer

scripts/yocto-layer new file +142 -0

new file mode 100755
index 0000000..f759275
@@ -0,0 +1,142 @@
+#!/usr/bin/env python
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#
+# Copyright (c) 2012, Intel Corporation.
+# All rights reserved.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+# DESCRIPTION
+# 'yocto-layer' is the Yocto Tool that helps users create a new Yocto
+# layer.  Invoking it without any arguments will display help screens
+# for the 'yocto-layer' command and list the available 'yocto-layer'
+# subcommands.  Invoking a subcommand without any arguments will
+# likewise display help screens for the specified subcommand.  Please
+# use that interface for detailed help.
+#
+# AUTHORS
+# Tom Zanussi <tom.zanussi (at] intel.com>
+#
+
+__version__ = "0.1.0"
+
+import os
+import sys
+import optparse
+import logging
+
+scripts_path = os.path.abspath(os.path.dirname(os.path.abspath(sys.argv[0])))
+lib_path = scripts_path + '/lib'
+sys.path = sys.path + [lib_path]
+
+from bsp.help import *
+from bsp.engine import *
+
+
+def yocto_layer_create_subcommand(args, usage_str):
+    """
+    Command-line handling for layer creation.  The real work is done by
+    bsp.engine.yocto_layer_create()
+    """
+    parser = optparse.OptionParser(usage = usage_str)
+
+    parser.add_option("-o", "--outdir", dest = "outdir", action = "store",
+                      help = "name of layer dir to create")
+    parser.add_option("-i", "--infile", dest = "properties_file", action = "store",
+                      help = "name of file containing the values for layer input properties as a JSON file")
+    parser.add_option("-c", "--codedump", dest = "codedump", action = "store_true",
+                      default = False, help = "dump the generated code to layergen.out")
+    (options, args) = parser.parse_args(args)
+
+    if len(args) != 1:
+        logging.error("Wrong number of arguments, exiting\n")
+        parser.print_help()
+        sys.exit(1)
+
+    layer_name = args[0]
+
+    if options.outdir:
+        layer_output_dir = options.outdir
+    else:
+        layer_output_dir = "meta-" + layer_name
+
+    yocto_layer_create(layer_name, scripts_path, layer_output_dir, options.codedump, options.properties_file)
+
+
+def yocto_layer_list_subcommand(args, usage_str):
+    """
+    Command-line handling for listing available layer properties and
+    values.  The real work is done by bsp.engine.yocto_layer_list()
+    """
+    parser = optparse.OptionParser(usage = usage_str)
+
+    parser.add_option("-o", "--outfile", action = "store", dest = "properties_file",
+                      help = "dump the possible values for layer properties to a JSON file")
+
+    (options, args) = parser.parse_args(args)
+
+    if not yocto_layer_list(args, scripts_path, options.properties_file):
+        logging.error("Bad list arguments, exiting\n")
+        parser.print_help()
+        sys.exit(1)
+
+
+subcommands = {
+    "create": [yocto_layer_create_subcommand,
+               yocto_layer_create_usage,
+               yocto_layer_create_help],
+    "list":   [yocto_layer_list_subcommand,
+               yocto_layer_list_usage,
+               yocto_layer_list_help],
+}
+
+
+def start_logging(loglevel):
+    logging.basicConfig(filname = 'yocto-layer.log', filemode = 'w', level=loglevel)
+
+
+def main():
+    parser = optparse.OptionParser(version = "yocto-layer version %s" % __version__,
+                                   usage = yocto_layer_usage)
+
+    parser.disable_interspersed_args()
+    parser.add_option("-D", "--debug", dest = "debug", action = "store_true",
+                      default = False, help = "output debug information")
+
+    (options, args) = parser.parse_args()
+
+    loglevel = logging.INFO
+    if options.debug:
+        loglevel = logging.DEBUG
+    start_logging(loglevel)
+
+    if len(args):
+        if args[0] == "help":
+            if len(args) == 1:
+                parser.print_help()
+                sys.exit(1)
+
+    invoke_subcommand(args, parser, yocto_layer_help_usage, subcommands)
+
+
+if __name__ == "__main__":
+    try:
+        ret = main()
+    except Exception:
+        ret = 1
+        import traceback
+        traceback.print_exc(5)
+    sys.exit(ret)
+

From: Tom Zanussi <[email protected]>

Add a 'layer' target containing all the data that will be used to generate a generic yocto layer.

Signed-off-by Tom Zanussi <[email protected]>

---

Quoted 7 lines · On 12/17/2012 09:51 AM, [email protected] wrote:

From: Tom Zanussi <[email protected]>

Add a 'layer' target containing all the data that will be used to generate a generic yocto layer.

Signed-off-by Tom Zanussi <[email protected]>

---

Great stuff Tom, only one nit below:

scripts/lib/bsp/substrate/target/arch/layer/{{ if create_example_recipe == \"y\": }} recipes-example/example/{{=example_recipe_name}}-0.1/helloworld.c line 5

Quoted 10 lines

new file mode 100644
index 0000000..90ce90e

scripts/lib/bsp/substrate/target/arch/layer/{{ if create_example_recipe == \"y\": }} recipes-example/example/{{=example_recipe_name}}-0.1/helloworld.c new file +5 -0

@@ -0,0 +1,6 @@
+#include <stdio.h>
+
+int main(int argc, char **argv)
+{
+    printf("Hello World!\n");

If you specify a return type of int, you should be returning one:

	return 0;

:-)

scripts/lib/bsp/substrate/target/arch/layer/{{ if create_example_recipe == \"y\": }} recipes-example/example/{{=example_recipe_name}}-0.1/helloworld.c line 6

Quoted 1 line

scripts/lib/bsp/substrate/target/arch/layer/{{ if create_example_recipe == \"y\": }} recipes-example/example/{{=example_recipe_name}}-0.1/helloworld.c new file +1 -0

+}

Looks good otherwise.

Quoted 25 lines · Darren Hart

Quoted 7 lines · On 12/17/2012 09:51 AM, [email protected] wrote:

From: Tom Zanussi <[email protected]>

Add a 'layer' target containing all the data that will be used to generate a generic yocto layer.

Signed-off-by Tom Zanussi <[email protected]>

---

Great stuff Tom, only one nit below:

scripts/lib/bsp/substrate/target/arch/layer/{{ if create_example_recipe == \"y\": }} recipes-example/example/{{=example_recipe_name}}-0.1/helloworld.c line 5

Quoted 10 lines

new file mode 100644
index 0000000..90ce90e

scripts/lib/bsp/substrate/target/arch/layer/{{ if create_example_recipe == \"y\": }} recipes-example/example/{{=example_recipe_name}}-0.1/helloworld.c new file +5 -0

@@ -0,0 +1,6 @@
+#include <stdio.h>
+
+int main(int argc, char **argv)
+{
+    printf("Hello World!\n");

If you specify a return type of int, you should be returning one:

Quoted 4 lines

	return 0;

:-)

Hmm, I don't think that's actually necessary for main() - if you don't specify a return value for main(), it should already implicitly return 0 IIRC...

Tom

Quoted 4 lines

scripts/lib/bsp/substrate/target/arch/layer/{{ if create_example_recipe == \"y\": }} recipes-example/example/{{=example_recipe_name}}-0.1/helloworld.c line 6

Quoted 1 line

scripts/lib/bsp/substrate/target/arch/layer/{{ if create_example_recipe == \"y\": }} recipes-example/example/{{=example_recipe_name}}-0.1/helloworld.c new file +1 -0

+}

Looks good otherwise.

Quoted 31 lines · Tom Zanussi

Quoted 29 lines · Darren Hart

Quoted 7 lines · On 12/17/2012 09:51 AM, [email protected] wrote:

From: Tom Zanussi <[email protected]>

Add a 'layer' target containing all the data that will be used to generate a generic yocto layer.

Signed-off-by Tom Zanussi <[email protected]>

---

Great stuff Tom, only one nit below:

scripts/lib/bsp/substrate/target/arch/layer/{{ if create_example_recipe == \"y\": }} recipes-example/example/{{=example_recipe_name}}-0.1/helloworld.c line 5

Quoted 10 lines

new file mode 100644
index 0000000..90ce90e

scripts/lib/bsp/substrate/target/arch/layer/{{ if create_example_recipe == \"y\": }} recipes-example/example/{{=example_recipe_name}}-0.1/helloworld.c new file +5 -0

@@ -0,0 +1,6 @@
+#include <stdio.h>
+
+int main(int argc, char **argv)
+{
+    printf("Hello World!\n");

If you specify a return type of int, you should be returning one:

	return 0;

:-)

Quoted 3 lines

Hmm, I don't think that's actually necessary for main() - if you don't specify a return value for main(), it should already implicitly return 0 IIRC...

Not true - that would be compiler specific and should not be relied on. Always best to be explicit (plus it will eliminate a GCC warning!)

Quoted 7 lines

Tom

Quoted 4 lines

scripts/lib/bsp/substrate/target/arch/layer/{{ if create_example_recipe == \"y\": }} recipes-example/example/{{=example_recipe_name}}-0.1/helloworld.c line 6

Quoted 1 line

scripts/lib/bsp/substrate/target/arch/layer/{{ if create_example_recipe == \"y\": }} recipes-example/example/{{=example_recipe_name}}-0.1/helloworld.c new file +1 -0

+}

Looks good otherwise.

Quoted 25 lines · Darren Hart

Quoted 7 lines · On 12/17/2012 09:51 AM, [email protected] wrote:

From: Tom Zanussi <[email protected]>

Add a 'layer' target containing all the data that will be used to generate a generic yocto layer.

Signed-off-by Tom Zanussi <[email protected]>

---

Great stuff Tom, only one nit below:

scripts/lib/bsp/substrate/target/arch/layer/{{ if create_example_recipe == \"y\": }} recipes-example/example/{{=example_recipe_name}}-0.1/helloworld.c line 5

Quoted 10 lines

new file mode 100644
index 0000000..90ce90e

scripts/lib/bsp/substrate/target/arch/layer/{{ if create_example_recipe == \"y\": }} recipes-example/example/{{=example_recipe_name}}-0.1/helloworld.c new file +5 -0

@@ -0,0 +1,6 @@
+#include <stdio.h>
+
+int main(int argc, char **argv)
+{
+    printf("Hello World!\n");

If you specify a return type of int, you should be returning one:

Quoted 4 lines

	return 0;

:-)

Hmm, I don't think that's actually necessary for main() - if you don't specify a return value for main(), it should already implicitly return 0 IIRC...

From: Tom Zanussi <[email protected]>

This does a bit of refactoring of the bsp-generation code to make it generically reusable for generating non-bsp layers.

The first user remains the existing yocto-bsp tool; these changes allow a second user, the new yocto-layer tool, to use the same code.

From: Tom Zanussi <[email protected]>

Add a new yocto_layer_create() function that will be used to generate a generic yocto layer (for the new 'yocto-layer' command).

Signed-off-by Tom Zanussi <[email protected]>

---
 scripts/lib/bsp/engine.py | 56 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 56 insertions(+)

scripts/lib/bsp/engine.py +56 -0

index 8985544..d406e79 100644
@@ -1352,6 +1352,62 @@ def expand_targets(context, bsp_output_dir):
     return target_files
 
 
+def yocto_layer_create(layer_name, scripts_path, layer_output_dir, codedump, properties_file):
+    """
+    Create yocto layer
+
+    layer_name - user-defined layer name
+    scripts_path - absolute path to yocto /scripts dir
+    bsp_output_dir - dirname to create for BSP
+    codedump - dump generated code to bspgen.out
+    properties_file - use values from here if nonempty i.e no prompting
+
+    arch - the arch for a generic layer is 'generic-layer', defined in
+           scripts/lib/bsp/substrate/target/generic-layer
+    """
+    if os.path.exists(bsp_output_dir):
+        print "\nBSP output dir already exists, exiting. (%s)" % bsp_output_dir
+        sys.exit(1)
+
+    properties = None
+
+    if properties_file:
+        try:
+            infile = open(properties_file, "r")
+        except IOError:
+            print "Couldn't open properties file %s for reading, exiting" % properties_file
+            sys.exit(1)
+
+        properties = json.load(infile)
+
+    os.mkdir(bsp_output_dir)
+
+    context = create_context(machine, arch, scripts_path)
+    target_files = expand_targets(context, bsp_output_dir)
+
+    input_lines = gather_inputlines(target_files)
+
+    program_lines = []
+
+    gen_program_header_lines(program_lines)
+
+    gen_initial_property_vals(input_lines, program_lines)
+
+    if properties:
+        gen_supplied_property_vals(properties, program_lines)
+
+    gen_program_machine_lines(machine, program_lines)
+
+    if not properties:
+        gen_program_input_lines(input_lines, program_lines, context)
+
+    gen_program_lines(target_files, program_lines)
+
+    run_program_lines(program_lines, codedump)
+
+    print "New %s BSP created in %s" % (arch, bsp_output_dir)
+
+
 def yocto_bsp_create(machine, arch, scripts_path, bsp_output_dir, codedump, properties_file):
     """
     Create bsp

Quoted 4 lines · On 12/17/2012 12:51 PM, [email protected] wrote:

From: Tom Zanussi <[email protected]>

Add a new yocto_layer_create() function that will be used to generate a generic yocto layer (for the new 'yocto-layer' command).

How is a "yocto layer" different from an OpenEmbedded layer? If you insist on "yocto layter", wouldn't "Yocto Project layer" be more accurate.

One of our goals for next year should be to clean up our terminology to reduce confusion in the user community.

Philip

Quoted 74 lines

Signed-off-by Tom Zanussi <[email protected]>

---
 scripts/lib/bsp/engine.py | 56 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 56 insertions(+)

scripts/lib/bsp/engine.py +56 -0

index 8985544..d406e79 100644
@@ -1352,6 +1352,62 @@ def expand_targets(context, bsp_output_dir):
     return target_files
 
 
+def yocto_layer_create(layer_name, scripts_path, layer_output_dir, codedump, properties_file):
+    """
+    Create yocto layer
+
+    layer_name - user-defined layer name
+    scripts_path - absolute path to yocto /scripts dir
+    bsp_output_dir - dirname to create for BSP
+    codedump - dump generated code to bspgen.out
+    properties_file - use values from here if nonempty i.e no prompting
+
+    arch - the arch for a generic layer is 'generic-layer', defined in
+           scripts/lib/bsp/substrate/target/generic-layer
+    """
+    if os.path.exists(bsp_output_dir):
+        print "\nBSP output dir already exists, exiting. (%s)" % bsp_output_dir
+        sys.exit(1)
+
+    properties = None
+
+    if properties_file:
+        try:
+            infile = open(properties_file, "r")
+        except IOError:
+            print "Couldn't open properties file %s for reading, exiting" % properties_file
+            sys.exit(1)
+
+        properties = json.load(infile)
+
+    os.mkdir(bsp_output_dir)
+
+    context = create_context(machine, arch, scripts_path)
+    target_files = expand_targets(context, bsp_output_dir)
+
+    input_lines = gather_inputlines(target_files)
+
+    program_lines = []
+
+    gen_program_header_lines(program_lines)
+
+    gen_initial_property_vals(input_lines, program_lines)
+
+    if properties:
+        gen_supplied_property_vals(properties, program_lines)
+
+    gen_program_machine_lines(machine, program_lines)
+
+    if not properties:
+        gen_program_input_lines(input_lines, program_lines, context)
+
+    gen_program_lines(target_files, program_lines)
+
+    run_program_lines(program_lines, codedump)
+
+    print "New %s BSP created in %s" % (arch, bsp_output_dir)
+
+
 def yocto_bsp_create(machine, arch, scripts_path, bsp_output_dir, codedump, properties_file):
     """
     Create bsp

Quoted 9 lines · Philip Balister

Quoted 4 lines · On 12/17/2012 12:51 PM, [email protected] wrote:

From: Tom Zanussi <[email protected]>

Add a new yocto_layer_create() function that will be used to generate a generic yocto layer (for the new 'yocto-layer' command).

How is a "yocto layer" different from an OpenEmbedded layer? If you insist on "yocto layter", wouldn't "Yocto Project layer" be more accurate.

Technically it isn't any different, but it's based on the template engine code and templates that make up the 'Yocto BSP Tools', which for various reasons (they create Yocto-compliant BSP layers, something not of interest to OpenEmbedded proper, and it probably doesn't make sense to clutter oe-core with a bunch of 'template data', etc) live in poky and not oe-core.

As such, the new tool is named 'yocto-layer' to match the other existing Yocto BSP tools, 'yocto-bsp' and 'yocto-kernel'.

Tom

Quoted 79 lines

One of our goals for next year should be to clean up our terminology to reduce confusion in the user community.

Philip

Quoted 74 lines

Signed-off-by Tom Zanussi <[email protected]>

---
 scripts/lib/bsp/engine.py | 56 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 56 insertions(+)

scripts/lib/bsp/engine.py +56 -0

index 8985544..d406e79 100644
@@ -1352,6 +1352,62 @@ def expand_targets(context, bsp_output_dir):
     return target_files
 
 
+def yocto_layer_create(layer_name, scripts_path, layer_output_dir, codedump, properties_file):
+    """
+    Create yocto layer
+
+    layer_name - user-defined layer name
+    scripts_path - absolute path to yocto /scripts dir
+    bsp_output_dir - dirname to create for BSP
+    codedump - dump generated code to bspgen.out
+    properties_file - use values from here if nonempty i.e no prompting
+
+    arch - the arch for a generic layer is 'generic-layer', defined in
+           scripts/lib/bsp/substrate/target/generic-layer
+    """
+    if os.path.exists(bsp_output_dir):
+        print "\nBSP output dir already exists, exiting. (%s)" % bsp_output_dir
+        sys.exit(1)
+
+    properties = None
+
+    if properties_file:
+        try:
+            infile = open(properties_file, "r")
+        except IOError:
+            print "Couldn't open properties file %s for reading, exiting" % properties_file
+            sys.exit(1)
+
+        properties = json.load(infile)
+
+    os.mkdir(bsp_output_dir)
+
+    context = create_context(machine, arch, scripts_path)
+    target_files = expand_targets(context, bsp_output_dir)
+
+    input_lines = gather_inputlines(target_files)
+
+    program_lines = []
+
+    gen_program_header_lines(program_lines)
+
+    gen_initial_property_vals(input_lines, program_lines)
+
+    if properties:
+        gen_supplied_property_vals(properties, program_lines)
+
+    gen_program_machine_lines(machine, program_lines)
+
+    if not properties:
+        gen_program_input_lines(input_lines, program_lines, context)
+
+    gen_program_lines(target_files, program_lines)
+
+    run_program_lines(program_lines, codedump)
+
+    print "New %s BSP created in %s" % (arch, bsp_output_dir)
+
+
 def yocto_bsp_create(machine, arch, scripts_path, bsp_output_dir, codedump, properties_file):
     """
     Create bsp

Quoted 11 lines · Tom Zanussi

Quoted 9 lines · Philip Balister

Quoted 4 lines · On 12/17/2012 12:51 PM, [email protected] wrote:

From: Tom Zanussi <[email protected]>

Add a new yocto_layer_create() function that will be used to generate a generic yocto layer (for the new 'yocto-layer' command).

How is a "yocto layer" different from an OpenEmbedded layer? If you insist on "yocto layter", wouldn't "Yocto Project layer" be more accurate.

Quoted 6 lines

Technically it isn't any different, but it's based on the template engine code and templates that make up the 'Yocto BSP Tools', which for various reasons (they create Yocto-compliant BSP layers, something not of interest to OpenEmbedded proper, and it probably doesn't make sense to clutter oe-core with a bunch of 'template data', etc) live in poky and not oe-core.

Actually, BSP's should work standalone with oe-core, so it is of interest.

Quoted 3 lines

As such, the new tool is named 'yocto-layer' to match the other existing Yocto BSP tools, 'yocto-bsp' and 'yocto-kernel'.

I will now get pedantic. <rday mode on>

The meta-yocto layer functions as a distro layer. So referring to things as yocto this and yocto that, suggests they only work with the meta-yocto layer.

So it doesn't make sense to me to put general purpose tools in a distro layer.

<rday mode off>

Quoted 87 lines

Tom

Quoted 79 lines

One of our goals for next year should be to clean up our terminology to reduce confusion in the user community.

Philip

Quoted 74 lines

Signed-off-by Tom Zanussi <[email protected]>

---
 scripts/lib/bsp/engine.py | 56 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 56 insertions(+)

scripts/lib/bsp/engine.py +56 -0

index 8985544..d406e79 100644
@@ -1352,6 +1352,62 @@ def expand_targets(context, bsp_output_dir):
     return target_files
 
 
+def yocto_layer_create(layer_name, scripts_path, layer_output_dir, codedump, properties_file):
+    """
+    Create yocto layer
+
+    layer_name - user-defined layer name
+    scripts_path - absolute path to yocto /scripts dir
+    bsp_output_dir - dirname to create for BSP
+    codedump - dump generated code to bspgen.out
+    properties_file - use values from here if nonempty i.e no prompting
+
+    arch - the arch for a generic layer is 'generic-layer', defined in
+           scripts/lib/bsp/substrate/target/generic-layer
+    """
+    if os.path.exists(bsp_output_dir):
+        print "\nBSP output dir already exists, exiting. (%s)" % bsp_output_dir
+        sys.exit(1)
+
+    properties = None
+
+    if properties_file:
+        try:
+            infile = open(properties_file, "r")
+        except IOError:
+            print "Couldn't open properties file %s for reading, exiting" % properties_file
+            sys.exit(1)
+
+        properties = json.load(infile)
+
+    os.mkdir(bsp_output_dir)
+
+    context = create_context(machine, arch, scripts_path)
+    target_files = expand_targets(context, bsp_output_dir)
+
+    input_lines = gather_inputlines(target_files)
+
+    program_lines = []
+
+    gen_program_header_lines(program_lines)
+
+    gen_initial_property_vals(input_lines, program_lines)
+
+    if properties:
+        gen_supplied_property_vals(properties, program_lines)
+
+    gen_program_machine_lines(machine, program_lines)
+
+    if not properties:
+        gen_program_input_lines(input_lines, program_lines, context)
+
+    gen_program_lines(target_files, program_lines)
+
+    run_program_lines(program_lines, codedump)
+
+    print "New %s BSP created in %s" % (arch, bsp_output_dir)
+
+
 def yocto_bsp_create(machine, arch, scripts_path, bsp_output_dir, codedump, properties_file):
     """
     Create bsp

Quoted 27 lines · Philip Balister

Quoted 11 lines · Tom Zanussi

Quoted 9 lines · Philip Balister

Quoted 4 lines · On 12/17/2012 12:51 PM, [email protected] wrote:

From: Tom Zanussi <[email protected]>

Add a new yocto_layer_create() function that will be used to generate a generic yocto layer (for the new 'yocto-layer' command).

How is a "yocto layer" different from an OpenEmbedded layer? If you insist on "yocto layter", wouldn't "Yocto Project layer" be more accurate.

Quoted 6 lines

Technically it isn't any different, but it's based on the template engine code and templates that make up the 'Yocto BSP Tools', which for various reasons (they create Yocto-compliant BSP layers, something not of interest to OpenEmbedded proper, and it probably doesn't make sense to clutter oe-core with a bunch of 'template data', etc) live in poky and not oe-core.

Actually, BSP's should work standalone with oe-core, so it is of interest.

Quoted 3 lines

As such, the new tool is named 'yocto-layer' to match the other existing Yocto BSP tools, 'yocto-bsp' and 'yocto-kernel'.

I will now get pedantic. <rday mode on>

Quoted 9 lines

The meta-yocto layer functions as a distro layer. So referring to things as yocto this and yocto that, suggests they only work with the meta-yocto layer.

So it doesn't make sense to me to put general purpose tools in a distro layer.

<rday mode off>

Right, none of the yocto- tools has anything to do with the meta-yocto layer, so if that's how it's taken, then, yeah, it's misleading.

It wouldn't bother me to do a global search and replace of 'yocto' with 'oe' for the tools - I guess that would imply it would all be moving into oe-core, though. Is that what you're suggesting?

Tom

Quoted 87 lines

Quoted 87 lines

Tom

Quoted 79 lines

One of our goals for next year should be to clean up our terminology to reduce confusion in the user community.

Philip

Quoted 74 lines

Signed-off-by Tom Zanussi <[email protected]>

---
 scripts/lib/bsp/engine.py | 56 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 56 insertions(+)

scripts/lib/bsp/engine.py +56 -0

index 8985544..d406e79 100644
@@ -1352,6 +1352,62 @@ def expand_targets(context, bsp_output_dir):
     return target_files
 
 
+def yocto_layer_create(layer_name, scripts_path, layer_output_dir, codedump, properties_file):
+    """
+    Create yocto layer
+
+    layer_name - user-defined layer name
+    scripts_path - absolute path to yocto /scripts dir
+    bsp_output_dir - dirname to create for BSP
+    codedump - dump generated code to bspgen.out
+    properties_file - use values from here if nonempty i.e no prompting
+
+    arch - the arch for a generic layer is 'generic-layer', defined in
+           scripts/lib/bsp/substrate/target/generic-layer
+    """
+    if os.path.exists(bsp_output_dir):
+        print "\nBSP output dir already exists, exiting. (%s)" % bsp_output_dir
+        sys.exit(1)
+
+    properties = None
+
+    if properties_file:
+        try:
+            infile = open(properties_file, "r")
+        except IOError:
+            print "Couldn't open properties file %s for reading, exiting" % properties_file
+            sys.exit(1)
+
+        properties = json.load(infile)
+
+    os.mkdir(bsp_output_dir)
+
+    context = create_context(machine, arch, scripts_path)
+    target_files = expand_targets(context, bsp_output_dir)
+
+    input_lines = gather_inputlines(target_files)
+
+    program_lines = []
+
+    gen_program_header_lines(program_lines)
+
+    gen_initial_property_vals(input_lines, program_lines)
+
+    if properties:
+        gen_supplied_property_vals(properties, program_lines)
+
+    gen_program_machine_lines(machine, program_lines)
+
+    if not properties:
+        gen_program_input_lines(input_lines, program_lines, context)
+
+    gen_program_lines(target_files, program_lines)
+
+    run_program_lines(program_lines, codedump)
+
+    print "New %s BSP created in %s" % (arch, bsp_output_dir)
+
+
 def yocto_bsp_create(machine, arch, scripts_path, bsp_output_dir, codedump, properties_file):
     """
     Create bsp

From: Tom Zanussi <[email protected]>

Since I've been doing kind of similar work lately for the 'custom kernel' support for yocto-bsp and have gotten several requests lately (mainly from Darren in support of the new kernel documentation) for something like this, I decided to just go ahead and try to quickly implement a general-purpose layer-generation tool based on the BSP-generation code used in the yocto-bsp tool.

There's actually an enhancement request bug for this already in the Yocto bugzilla, but it doesn't contain many details:

Bug 3094 - Add a layer generation tool:

https://bugzilla.yoctoproject.org/show_bug.cgi?id=3094

So anyway, this is my initial take on what that bug means. It's an RFC for a couple reasons, first that I'd like to get comments on whether this is actually what's needed for a 'layer generation tool', and second, because I still need to do some cleanup of the refactored code, but don't want to waste much time on it if it's not really what's needed.

Below are a couple examples of how it's used - you really have to try it yourself to see what's generated - I don't want to post tarballs or such on the list and it's simple to generate and look at the layers.

The first case is just a very simple layer with a layer.conf and README - basically the simplest layer you can create, and which exists mainly because even that is easy to get wrong. As with the yocto-bsp tool, the script queries the user for a couple items, here we just take the defaults, which are a priority of 6 for the layer and no other components such as example recipes:

[trz@empanada build]$ yocto-layer create simplestlayer
Please enter the layer priority you'd like to use for the layer: [default: 6] Would you like to have an example recipe created? (y/n) [default: n] Would you like to have an example bbappend file created? (y/n) [default: n]

New layer created in meta-simplestlayer.

Don't forget to add it to your BBLAYERS (for details see meta-simplestlayer\README).

[trz@empanada build]$ find .
.
./meta-simplestlayer
./meta-simplestlayer/conf
./meta-simplestlayer/conf/layer.conf
./meta-simplestlayer/README
./meta-simplestlayer/COPYING.MIT

In the second case, we tell the tool that we do want an example .bb and and an example .bbappend. We're queried for the recipe name that we want our recipe to have, and for the .bbappend, the name of the base recipe and its version. Below you can see the files it generates - please look at the files themselves to see the contents. For the recipe example, it generates a recipe based on the example in the Yocto manual, and my own helloworld.c code (untested so far which is also why this is an RFC). For the .bbappend example, it just creates an empty .patch file with some instructions on what to do to modify the parent recipe with a patch:

[trz@empanada build]$ yocto-layer create mylayer
Please enter the layer priority you'd like to use for the layer: [default: 6]
Would you like to have an example recipe created? (y/n) [default: n] y
Please enter the name you'd like to use for your example recipe: [default: example] flork
Would you like to have an example bbappend file created? (y/n) [default: n] y
Please enter the name you'd like to use for your bbappend file: [default: example] chork
Please enter the version number you'd like to use for your bbappend file (this should match the recipe you're appending to): [default: 0.1] 0.22.3

New layer created in meta-mylayer.

Don't forget to add it to your BBLAYERS (for details see meta-mylayer\README).

[trz@empanada build]$ find .
.
./meta-mylayer
./meta-mylayer/recipes-example
./meta-mylayer/recipes-example/example ./meta-mylayer/recipes-example/example/flork-0.1 ./meta-mylayer/recipes-example/example/flork-0.1/helloworld.c ./meta-mylayer/recipes-example/example/flork-0.1/example.patch ./meta-mylayer/recipes-example/example/flork_0.1.bb ./meta-mylayer/conf
./meta-mylayer/conf/layer.conf
./meta-mylayer/recipes-example-bbappend ./meta-mylayer/recipes-example-bbappend/example-bbappend ./meta-mylayer/recipes-example-bbappend/example-bbappend/chork-0.22.3 ./meta-mylayer/recipes-example-bbappend/example-bbappend/chork-0.22.3/example.patch ./meta-mylayer/recipes-example-bbappend/example-bbappend/chork_0.22.3.bbappend ./meta-mylayer/README
./meta-mylayer/COPYING.MIT

Please try it and look at the generated code - any comments or suggestions are welcome.

Thanks,

Tom

The following changes since commit da598d4f0fcf6faa62055084a51d70d735399d33:

eglibc: fix checksums of fetched patches (2012-12-13 18:02:21 +0000)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib.git tzanussi/yocto-layer-rfc
http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=tzanussi/yocto-layer-rfc

Tom Zanussi (5):
scripts/lib/bsp/engine.py: add yocto_layer_create()
yocto-layer: new script
yocto-layer: add help/usage
yocto-layer: add 'layer' template data scripts/lib/bsp/engine.py: refactor bsp-creation code

 scripts/lib/bsp/engine.py                          |  86 +++++++-
 scripts/lib/bsp/help.py                            | 228 +++++++++++++++++++++
 .../bsp/substrate/target/arch/layer/COPYING.MIT    |  17 ++
 scripts/lib/bsp/substrate/target/arch/layer/README |  64 ++++++
 .../substrate/target/arch/layer/conf/layer.conf    |  10 +
 .../target/arch/layer/layer-questions.noinstall    |  14 ++
 .../example.patch"                                 |  12 ++
 ..._name}}_{{=example_bbappend_version}}.bbappend" |   8 +
 .../{{=example_recipe_name}}-0.1/example.patch"    |  12 ++
 .../{{=example_recipe_name}}-0.1/helloworld.c"     |   6 +
 .../example/{{=example_recipe_name}}_0.1.bb"       |  23 +++
 scripts/yocto-layer                                | 142 +++++++++++++
 12 files changed, 612 insertions(+), 10 deletions(-)
 create mode 100644 scripts/lib/bsp/substrate/target/arch/layer/COPYING.MIT
 create mode 100644 scripts/lib/bsp/substrate/target/arch/layer/README
 create mode 100644 scripts/lib/bsp/substrate/target/arch/layer/conf/layer.conf
 create mode 100644 scripts/lib/bsp/substrate/target/arch/layer/layer-questions.noinstall
 create mode 100644 "scripts/lib/bsp/substrate/target/arch/layer/{{ if create_example_bbappend == \"y\": }} recipes-example-bbappend/example-bbappend/{{=example_bbappend_name}}-{{=example_bbappend_version}}/example.patch"
 create mode 100644 "scripts/lib/bsp/substrate/target/arch/layer/{{ if create_example_bbappend == \"y\": }} recipes-example-bbappend/example-bbappend/{{=example_bbappend_name}}_{{=example_bbappend_version}}.bbappend"
 create mode 100644 "scripts/lib/bsp/substrate/target/arch/layer/{{ if create_example_recipe == \"y\": }} recipes-example/example/{{=example_recipe_name}}-0.1/example.patch"
 create mode 100644 "scripts/lib/bsp/substrate/target/arch/layer/{{ if create_example_recipe == \"y\": }} recipes-example/example/{{=example_recipe_name}}-0.1/helloworld.c"
 create mode 100644 "scripts/lib/bsp/substrate/target/arch/layer/{{ if create_example_recipe == \"y\": }} recipes-example/example/{{=example_recipe_name}}_0.1.bb"
 create mode 100755 scripts/yocto-layer

Quoted 33 lines · On 12/17/2012 09:51 AM, [email protected] wrote:

From: Tom Zanussi <[email protected]>

Since I've been doing kind of similar work lately for the 'custom kernel' support for yocto-bsp and have gotten several requests lately (mainly from Darren in support of the new kernel documentation) for something like this, I decided to just go ahead and try to quickly implement a general-purpose layer-generation tool based on the BSP-generation code used in the yocto-bsp tool.

There's actually an enhancement request bug for this already in the Yocto bugzilla, but it doesn't contain many details:

Bug 3094 - Add a layer generation tool:

https://bugzilla.yoctoproject.org/show_bug.cgi?id=3094

So anyway, this is my initial take on what that bug means. It's an RFC for a couple reasons, first that I'd like to get comments on whether this is actually what's needed for a 'layer generation tool', and second, because I still need to do some cleanup of the refactored code, but don't want to waste much time on it if it's not really what's needed.

Below are a couple examples of how it's used - you really have to try it yourself to see what's generated - I don't want to post tarballs or such on the list and it's simple to generate and look at the layers.

The first case is just a very simple layer with a layer.conf and README - basically the simplest layer you can create, and which exists mainly because even that is easy to get wrong. As with the yocto-bsp tool, the script queries the user for a couple items, here we just take the defaults, which are a priority of 6 for the layer and no other components such as example recipes:

Quoted 4 lines

[trz@empanada build]$ yocto-layer create simplestlayer
Please enter the layer priority you'd like to use for the layer: [default: 6] Would you like to have an example recipe created? (y/n) [default: n] Would you like to have an example bbappend file created? (y/n) [default: n]

Can these options be specified on the cli? It would be nice for documentation to not have to list all the questions to describe how to accomplish a specific task such as this. e.g.

[trz@empanada build]$ yocto-layer create simplestlayer -p 6 --no-recipe --no-bbappend

Or possibly having defaults to those questions accepted by default unless [-i|--interactive] is specified?

empanada :-) Mmmmmm

Quoted 5 lines

New layer created in meta-simplestlayer.

Don't forget to add it to your BBLAYERS (for details see meta-simplestlayer\README).

Quoted 8 lines

[trz@empanada build]$ find .
.
./meta-simplestlayer
./meta-simplestlayer/conf
./meta-simplestlayer/conf/layer.conf
./meta-simplestlayer/README
./meta-simplestlayer/COPYING.MIT

It might make sense to create a recipes/ directory as well, just to support convention.

Quoted 12 lines

In the second case, we tell the tool that we do want an example .bb and and an example .bbappend. We're queried for the recipe name that we want our recipe to have, and for the .bbappend, the name of the base recipe and its version. Below you can see the files it generates - please look at the files themselves to see the contents. For the recipe example, it generates a recipe based on the example in the Yocto manual, and my own helloworld.c code (untested so far which is also why this is an RFC). For the .bbappend example, it just creates an empty .patch file with some instructions on what to do to modify the parent recipe with a patch:

Quoted 7 lines

[trz@empanada build]$ yocto-layer create mylayer
Please enter the layer priority you'd like to use for the layer: [default: 6]
Would you like to have an example recipe created? (y/n) [default: n] y
Please enter the name you'd like to use for your example recipe: [default: example] flork
Would you like to have an example bbappend file created? (y/n) [default: n] y
Please enter the name you'd like to use for your bbappend file: [default: example] chork
Please enter the version number you'd like to use for your bbappend file (this should match the recipe you're appending to): [default: 0.1] 0.22.3

Wow, that's quite a bit more than I was asking for... I can imagine that some folks my find that useful.

Quoted 24 lines

New layer created in meta-mylayer.

Don't forget to add it to your BBLAYERS (for details see meta-mylayer\README).

[trz@empanada build]$ find .
.
./meta-mylayer
./meta-mylayer/recipes-example
./meta-mylayer/recipes-example/example ./meta-mylayer/recipes-example/example/flork-0.1 ./meta-mylayer/recipes-example/example/flork-0.1/helloworld.c ./meta-mylayer/recipes-example/example/flork-0.1/example.patch ./meta-mylayer/recipes-example/example/flork_0.1.bb ./meta-mylayer/conf
./meta-mylayer/conf/layer.conf
./meta-mylayer/recipes-example-bbappend ./meta-mylayer/recipes-example-bbappend/example-bbappend ./meta-mylayer/recipes-example-bbappend/example-bbappend/chork-0.22.3 ./meta-mylayer/recipes-example-bbappend/example-bbappend/chork-0.22.3/example.patch ./meta-mylayer/recipes-example-bbappend/example-bbappend/chork_0.22.3.bbappend ./meta-mylayer/README
./meta-mylayer/COPYING.MIT

Quoted 2 lines

Please try it and look at the generated code - any comments or suggestions are welcome.

In concept, looks pretty good to me, and a natural refactoring of yocto-bsp. I'll review the patches independently.

Thanks,

Darren

Quoted 44 lines

Thanks,

Tom

The following changes since commit da598d4f0fcf6faa62055084a51d70d735399d33:

eglibc: fix checksums of fetched patches (2012-12-13 18:02:21 +0000)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib.git tzanussi/yocto-layer-rfc
http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=tzanussi/yocto-layer-rfc

Tom Zanussi (5):
scripts/lib/bsp/engine.py: add yocto_layer_create()
yocto-layer: new script
yocto-layer: add help/usage
yocto-layer: add 'layer' template data scripts/lib/bsp/engine.py: refactor bsp-creation code

 scripts/lib/bsp/engine.py                          |  86 +++++++-
 scripts/lib/bsp/help.py                            | 228 +++++++++++++++++++++
 .../bsp/substrate/target/arch/layer/COPYING.MIT    |  17 ++
 scripts/lib/bsp/substrate/target/arch/layer/README |  64 ++++++
 .../substrate/target/arch/layer/conf/layer.conf    |  10 +
 .../target/arch/layer/layer-questions.noinstall    |  14 ++
 .../example.patch"                                 |  12 ++
 ..._name}}_{{=example_bbappend_version}}.bbappend" |   8 +
 .../{{=example_recipe_name}}-0.1/example.patch"    |  12 ++
 .../{{=example_recipe_name}}-0.1/helloworld.c"     |   6 +
 .../example/{{=example_recipe_name}}_0.1.bb"       |  23 +++
 scripts/yocto-layer                                | 142 +++++++++++++
 12 files changed, 612 insertions(+), 10 deletions(-)
 create mode 100644 scripts/lib/bsp/substrate/target/arch/layer/COPYING.MIT
 create mode 100644 scripts/lib/bsp/substrate/target/arch/layer/README
 create mode 100644 scripts/lib/bsp/substrate/target/arch/layer/conf/layer.conf
 create mode 100644 scripts/lib/bsp/substrate/target/arch/layer/layer-questions.noinstall
 create mode 100644 "scripts/lib/bsp/substrate/target/arch/layer/{{ if create_example_bbappend == \"y\": }} recipes-example-bbappend/example-bbappend/{{=example_bbappend_name}}-{{=example_bbappend_version}}/example.patch"
 create mode 100644 "scripts/lib/bsp/substrate/target/arch/layer/{{ if create_example_bbappend == \"y\": }} recipes-example-bbappend/example-bbappend/{{=example_bbappend_name}}_{{=example_bbappend_version}}.bbappend"
 create mode 100644 "scripts/lib/bsp/substrate/target/arch/layer/{{ if create_example_recipe == \"y\": }} recipes-example/example/{{=example_recipe_name}}-0.1/example.patch"
 create mode 100644 "scripts/lib/bsp/substrate/target/arch/layer/{{ if create_example_recipe == \"y\": }} recipes-example/example/{{=example_recipe_name}}-0.1/helloworld.c"
 create mode 100644 "scripts/lib/bsp/substrate/target/arch/layer/{{ if create_example_recipe == \"y\": }} recipes-example/example/{{=example_recipe_name}}_0.1.bb"
 create mode 100755 scripts/yocto-layer

Quoted 3 lines · Darren Hart

Quoted 2 lines · On 12/17/2012 09:51 AM, [email protected] wrote:

From: Tom Zanussi <[email protected]>

I'm thinking that most users probably won't use these tools very often, so providing a complete command-line equivalent encompassing all the options would be more trouble than it's worth. It might make sense though to do it for the case of the 'simplest layer' which would mean just adding a 'priority' option in addition to the layer name already specified on the command line.

Quoted 3 lines

It might make sense to create a recipes/ directory as well, just to support convention.

Not sure what you mean - if you say 'y' to either the recipe or the bbappend, it will create a recipes-example/ dir to contain them. Do you mean something else?