Variable typing & reworked imports handling

PATCH Applied Chris Larson 3 participants 8 messages May 18, 2011 – May 20, 2011

lore

Share & tools

lore

Chris Larson

author

2011-05-18 21:05 UTC #0 lore raw

cover Variable typing & reworked imports handling

From: Chris Larson <[email protected]>

Pull URL: git://git.openembedded.org/openembedded-core-contrib
Branch: kergoth/oe-sync-base
Browse: http://git.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=kergoth/oe-sync-base

Thanks,
Chris Larson <[email protected]>
---

Chris Larson (5):
Implement variable typing (sync from OE) oe.data: expand the flags
base.bbclass: switch to current OE's imports handling Shift oe import logic out of the event handler base.bbclass: use oe.data for OE_IMPORTS

 meta/classes/base.bbclass      |   46 +++++++++--------
 meta/classes/typecheck.bbclass |   12 +++++
 meta/lib/oe/data.py            |   18 +++++++
 meta/lib/oe/maketype.py        |  100 ++++++++++++++++++++++++++++++++++++++
 meta/lib/oe/test_types.py      |   62 ++++++++++++++++++++++++
 meta/lib/oe/types.py           |  104 ++++++++++++++++++++++++++++++++++++++++
 6 files changed, 320 insertions(+), 22 deletions(-)
 create mode 100644 meta/classes/typecheck.bbclass
 create mode 100644 meta/lib/oe/data.py
 create mode 100644 meta/lib/oe/maketype.py
 create mode 100644 meta/lib/oe/test_types.py
 create mode 100644 meta/lib/oe/types.py

Chris Larson

author

2011-05-18 21:05 UTC #1 lore raw

PATCH 1/5 Implement variable typing (sync from OE)

From: Chris Larson <[email protected]>

This implementation consists of two components:

- Type creation python modules, whose job it is to construct objects of the defined type for a given variable in the metadata
- typecheck.bbclass, which iterates over all configuration variables with a type defined and uses oe.types to check the validity of the values

This gives us a few benefits:

- Automatic sanity checking of all configuration variables with a defined type
- Avoid duplicating the "how do I make use of the value of this variable" logic between its users. For variables like PATH, this is simply a split(), for boolean variables, the duplication can result in confusing, or even mismatched semantics (is this 0/1, empty/nonempty, what?)
- Make it easier to create a configuration UI, as the type information could be used to provide a better interface than a text edit box (e.g checkbox for 'boolean', dropdown for 'choice')

This functionality is entirely opt-in right now. To enable the configuration variable type checking, simply INHERIT += "typecheck". Example of a failing type check:

BAZ = "foo"
BAZ[type] = "boolean"

$ bitbake -p
FATAL: BAZ: Invalid boolean value 'foo'
$

Examples of leveraging oe.types in a python snippet:

PACKAGES[type] = "list"

python () {
import oe.data
for pkg in oe.data.typed_value("PACKAGES", d):
bb.note("package: %s" % pkg)
}

LIBTOOL_HAS_SYSROOT = "yes"
LIBTOOL_HAS_SYSROOT[type] = "boolean"

python () {
    import oe.data
    assert(oe.data.typed_value("LIBTOOL_HAS_SYSROOT", d) == True)
}

Signed-off-by Chris Larson <[email protected]>

---
 meta/classes/base.bbclass      |    1 +
 meta/classes/typecheck.bbclass |   12 +++++
 meta/lib/oe/data.py            |   13 +++++
 meta/lib/oe/maketype.py        |  100 ++++++++++++++++++++++++++++++++++++++
 meta/lib/oe/test_types.py      |   62 ++++++++++++++++++++++++
 meta/lib/oe/types.py           |  104 ++++++++++++++++++++++++++++++++++++++++
 6 files changed, 292 insertions(+), 0 deletions(-)
 create mode 100644 meta/classes/typecheck.bbclass
 create mode 100644 meta/lib/oe/data.py
 create mode 100644 meta/lib/oe/maketype.py
 create mode 100644 meta/lib/oe/test_types.py
 create mode 100644 meta/lib/oe/types.py

meta/classes/base.bbclass +1 -0

index 23095ec..b552de2 100644
@@ -28,6 +28,7 @@ python sys_path_eh () {
 
         import oe.path
         import oe.utils
+        import oe.data
         inject("bb", bb)
         inject("sys", sys)
         inject("time", time)

meta/classes/typecheck.bbclass new file +12 -0

new file mode 100644
index 0000000..646cd4e
@@ -0,0 +1,12 @@
+# Check types of bitbake configuration variables
+#
+# See oe.types for details.
+
+python check_types() {
+    import oe.types
+    if isinstance(e, bb.event.ConfigParsed):
+        for key in e.data.keys():
+            if e.data.getVarFlag(key, "type"):
+                oe.types.value(key, e.data)
+}
+addhandler check_types

meta/lib/oe/data.py new file +13 -0

new file mode 100644
index 0000000..8b7c3cd
@@ -0,0 +1,13 @@
+import oe.maketype
+import bb.msg
+
+def typed_value(key, d):
+    """Construct a value for the specified metadata variable, using its flags
+    to determine the type and parameters for construction."""
+    var_type = d.getVarFlag(key, 'type')
+    flags = d.getVarFlags(key)
+
+    try:
+        return oe.maketype.create(d.getVar(key, True) or '', var_type, **flags)
+    except (TypeError, ValueError), exc:
+        bb.msg.fatal(bb.msg.domain.Data, "%s: %s" % (key, str(exc)))

meta/lib/oe/maketype.py new file +100 -0

100 lines not on this page · raw message

meta/lib/oe/test_types.py new file +62 -0

62 lines not on this page · raw message

meta/lib/oe/types.py new file +104 -0

104 lines not on this page · raw message

Signature · 1 line

1.7.1

Chris Larson

author

2011-05-18 21:05 UTC #2 lore raw

PATCH 2/5 oe.data: expand the flags

In reply to Chris Larson

From: Chris Larson <[email protected]>

Signed-off-by Chris Larson <[email protected]>

---
 meta/lib/oe/data.py |    5 +++++
 1 files changed, 5 insertions(+), 0 deletions(-)

meta/lib/oe/data.py +5 -0

index 8b7c3cd..4b4d03e 100644
@@ -6,6 +6,11 @@ def typed_value(key, d):
     to determine the type and parameters for construction."""
     var_type = d.getVarFlag(key, 'type')
     flags = d.getVarFlags(key)
+    if flags is not None:
+        flags = dict((flag, bb.data.expand(value, d))
+                     for flag, value in flags.iteritems())
+    else:
+        flags = {}
 
     try:
         return oe.maketype.create(d.getVar(key, True) or '', var_type, **flags)

Signature · 1 line

1.7.1

Chris Larson

author

2011-05-18 21:05 UTC #3 lore raw

PATCH 3/5 base.bbclass: switch to current OE's imports handling

In reply to Chris Larson

From: Chris Larson <[email protected]>

The current mechanism makes it easier for classes to add new oe modules to be automatically imported, and thereby made available to python snippets (${@}).

Signed-off-by Chris Larson <[email protected]>

---
 meta/classes/base.bbclass |   22 +++++++++-------------
 1 files changed, 9 insertions(+), 13 deletions(-)

meta/classes/base.bbclass +9 ~~-13~~

index b552de2..51f5a14 100644
@@ -10,32 +10,28 @@ inherit metadata_scm
 inherit buildstats
 inherit logging
 
-python sys_path_eh () {
+OE_IMPORTS += "os sys time oe.path oe.utils oe.data"
+
+python oe_import () {
     if isinstance(e, bb.event.ConfigParsed):
-        import sys
-        import os
-        import time
+        import os, sys
 
         bbpath = e.data.getVar("BBPATH", True).split(":")
         sys.path[0:0] = [os.path.join(dir, "lib") for dir in bbpath]
 
         def inject(name, value):
-            """Make a python object accessible from everywhere for the metadata"""
+            """Make a python object accessible from the metadata"""
             if hasattr(bb.utils, "_context"):
                 bb.utils._context[name] = value
             else:
                 __builtins__[name] = value
 
-        import oe.path
-        import oe.utils
-        import oe.data
-        inject("bb", bb)
-        inject("sys", sys)
-        inject("time", time)
-        inject("oe", oe)
+        for toimport in e.data.getVar("OE_IMPORTS", True).split():
+            imported = __import__(toimport)
+            inject(toimport.split(".", 1)[0], imported)
 }
 
-addhandler sys_path_eh
+addhandler oe_import
 
 die() {
 	bbfatal "$*"

Signature · 1 line

1.7.1

Chris Larson

author

2011-05-18 21:05 UTC #4 lore raw

PATCH 4/5 Shift oe import logic out of the event handler

In reply to Chris Larson

From: Chris Larson <[email protected]>

This can be useful if we need the imports from another config parsed event handler, and can't rely upon the base one running before that one.

Signed-off-by Chris Larson <[email protected]>

---
 meta/classes/base.bbclass |   33 ++++++++++++++++++---------------
 1 files changed, 18 insertions(+), 15 deletions(-)

meta/classes/base.bbclass +18 ~~-15~~

index 51f5a14..391545c 100644
@@ -12,26 +12,29 @@ inherit logging
 
 OE_IMPORTS += "os sys time oe.path oe.utils oe.data"
 
-python oe_import () {
-    if isinstance(e, bb.event.ConfigParsed):
-        import os, sys
+def oe_import(d):
+    import os, sys
+
+    bbpath = d.getVar("BBPATH", True).split(":")
+    sys.path[0:0] = [os.path.join(dir, "lib") for dir in bbpath]
 
-        bbpath = e.data.getVar("BBPATH", True).split(":")
-        sys.path[0:0] = [os.path.join(dir, "lib") for dir in bbpath]
+    def inject(name, value):
+        """Make a python object accessible from the metadata"""
+        if hasattr(bb.utils, "_context"):
+            bb.utils._context[name] = value
+        else:
+            __builtins__[name] = value
 
-        def inject(name, value):
-            """Make a python object accessible from the metadata"""
-            if hasattr(bb.utils, "_context"):
-                bb.utils._context[name] = value
-            else:
-                __builtins__[name] = value
+    for toimport in d.getVar("OE_IMPORTS", True).split():
+        imported = __import__(toimport)
+        inject(toimport.split(".", 1)[0], imported)
 
-        for toimport in e.data.getVar("OE_IMPORTS", True).split():
-            imported = __import__(toimport)
-            inject(toimport.split(".", 1)[0], imported)
+python oe_import_eh () {
+    if isinstance(e, bb.event.ConfigParsed):
+	oe_import(e.data)
 }
 
-addhandler oe_import
+addhandler oe_import_eh
 
 die() {
 	bbfatal "$*"

Signature · 1 line

1.7.1

Chris Larson

author

2011-05-18 21:05 UTC #5 lore raw

PATCH 5/5 base.bbclass: use oe.data for OE_IMPORTS

In reply to Chris Larson

From: Chris Larson <[email protected]>

Signed-off-by Chris Larson <[email protected]>

---
 meta/classes/base.bbclass |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)

meta/classes/base.bbclass +3 -1

index 391545c..7950bc3 100644
@@ -11,6 +11,7 @@ inherit buildstats
 inherit logging
 
 OE_IMPORTS += "os sys time oe.path oe.utils oe.data"
+OE_IMPORTS[type] = "list"
 
 def oe_import(d):
     import os, sys

@@ -25,7 +26,8 @@ def oe_import(d):
         else:
             __builtins__[name] = value
 
-    for toimport in d.getVar("OE_IMPORTS", True).split():
+    import oe.data
+    for toimport in oe.data.typed_value("OE_IMPORTS", d):
         imported = __import__(toimport)
         inject(toimport.split(".", 1)[0], imported)

Signature · 1 line

1.7.1

Joshua Lock

Acked-by on the v1 series

2011-05-20 00:21 UTC #6 lore raw

In reply to Chris Larson

Quoted 8 lines · Chris Larson

From: Chris Larson <[email protected]>
Pull URL: git://git.openembedded.org/openembedded-core-contrib
Branch: kergoth/oe-sync-base
Browse: http://git.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=kergoth/oe-sync-base
Thanks,
Chris Larson <[email protected]>

I like this, I was thinking about typing in BB about the time you first showed me this so am pleased to see it submitted.

I think this will be especially useful for the oft discussed configuration editor UI. Thanks!

Acked-by Joshua Lock <[email protected]>

Quoted 28 lines

---
Chris Larson (5):
Implement variable typing (sync from OE) oe.data: expand the flags
base.bbclass: switch to current OE's imports handling Shift oe import logic out of the event handler base.bbclass: use oe.data for OE_IMPORTS
 meta/classes/base.bbclass      |   46 +++++++++--------
 meta/classes/typecheck.bbclass |   12 +++++
 meta/lib/oe/data.py            |   18 +++++++
 meta/lib/oe/maketype.py        |  100 ++++++++++++++++++++++++++++++++++++++
 meta/lib/oe/test_types.py      |   62 ++++++++++++++++++++++++
 meta/lib/oe/types.py           |  104 ++++++++++++++++++++++++++++++++++++++++
 6 files changed, 320 insertions(+), 22 deletions(-)
 create mode 100644 meta/classes/typecheck.bbclass
 create mode 100644 meta/lib/oe/data.py
 create mode 100644 meta/lib/oe/maketype.py
 create mode 100644 meta/lib/oe/test_types.py
 create mode 100644 meta/lib/oe/types.py
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Signature · 3 lines

Joshua Lock
        Yocto Project Build Monkey
        Intel Open Source Technology Centre

Richard Purdie

Applied

2011-05-20 16:43 UTC #7 lore raw

In reply to Chris Larson

Quoted 11 lines · Chris Larson

From: Chris Larson <[email protected]>
Pull URL: git://git.openembedded.org/openembedded-core-contrib
Branch: kergoth/oe-sync-base
Browse: http://git.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=kergoth/oe-sync-base
Thanks,
Chris Larson <[email protected]>
---

Quoted 6 lines

Chris Larson (5):
Implement variable typing (sync from OE) oe.data: expand the flags
base.bbclass: switch to current OE's imports handling Shift oe import logic out of the event handler base.bbclass: use oe.data for OE_IMPORTS

Merged to master, thanks.

Richard