Skip to contentSkip to conversation

RFC wic: Support .wks files for multiple machines

PATCH In review Ricardo Ribalda 2 participants 4 messages Apr 24, 2020 – May 13, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 RFC wic: Support .wks files for multiple machines

If you want to make a disk image containing rootfs for different machines, the only way to do it today is by calling wic and passing the different rootfs-paths.

Eg:
combined.wks
part /export --source rootfs --rootfs-dir=target-image part / --source rootfs

bitbake multiconfig:arm:target-image
wic create combined --rootfs-dir target-image=/workdir/build/tmp/work/arm-poky-linux/target-image/1.0-r0/rootfs/ -e main

This has many drawbacks:
1) You need to know the folder location for the target-image
2) It is easy to forget updating the target (only running wic and not bitbake)
3) It is slow
4) It does not scale when you have multiple machines

Instead, wic can be given a hint of what machine to search for:

combined.wks
part /export --source rootfs --rootfs-dir=arm:target-image part / --source rootfs

bitbake main

If we ensoure the dependency of target-image from main via do_image[mcdepends], all the dependencies are automatically handled.

This patch makes wic aware of the machine to use.

---
 scripts/lib/wic/misc.py | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)
scripts/lib/wic/misc.py +13 -3
index 91975ba151..51b43b49aa 100644
@@ -206,7 +206,12 @@ class BitbakeVars(defaultdict):
 
         if image not in self:
             if image and self.vars_dir:
-                fname = os.path.join(self.vars_dir, image + '.env')
+                if image.find(":") != -1:
+                    [machine, img] = image.split(":")
+                    fname = os.path.join(self.vars_dir, "../../",
+                                         machine, "imgdata", img + '.env')
+                else:
+                    fname = os.path.join(self.vars_dir, image + '.env')
                 if os.path.isfile(fname):
                     # parse .env file
                     with open(fname) as varsfile:
@@ -220,11 +225,16 @@ class BitbakeVars(defaultdict):
                 # Get bitbake -e output
                 cmd = "bitbake -e"
                 if image:
-                    cmd += " %s" % image
+                    if image.find(":") != -1:
+                        [machine, img] = image.split(":")
+                        cmd = "MACHINE=%s %s" % (machine, cmd)
+                    else:
+                        img = image
+                    cmd += " %s" % img
 
                 log_level = logger.getEffectiveLevel()
                 logger.setLevel(logging.INFO)
-                ret, lines = _exec_cmd(cmd)
+                ret, lines = _exec_cmd(cmd, as_shell=True)
                 logger.setLevel(log_level)
 
                 if ret:
Signature · 1 line
2.26.1
#1 lore raw

This will only work the way you expect if hte user's local.conf uses ?= or ??= for MACHINE, otherwise setting it in the environment will do nothing.

Quoted 82 lines · Ricardo Ribalda

If you want to make a disk image containing rootfs for different machines, the only way to do it today is by calling wic and passing the different rootfs-paths.

Eg:
combined.wks
part /export --source rootfs --rootfs-dir=target-image part / --source rootfs

bitbake multiconfig:arm:target-image
wic create combined --rootfs-dir target-image=/workdir/build/tmp/work/arm-poky-linux/target-image/1.0-r0/rootfs/ -e main

This has many drawbacks:
1) You need to know the folder location for the target-image
2) It is easy to forget updating the target (only running wic and not bitbake)
3) It is slow
4) It does not scale when you have multiple machines

Instead, wic can be given a hint of what machine to search for:

combined.wks
part /export --source rootfs --rootfs-dir=arm:target-image part / --source rootfs

bitbake main

If we ensoure the dependency of target-image from main via do_image[mcdepends], all the dependencies are automatically handled.

This patch makes wic aware of the machine to use.

---
 scripts/lib/wic/misc.py | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)
scripts/lib/wic/misc.py +13 -4
index 91975ba151..51b43b49aa 100644
@@ -206,7 +206,12 @@ class BitbakeVars(defaultdict):
 
         if image not in self:
             if image and self.vars_dir:
-                fname = os.path.join(self.vars_dir, image + '.env')
+                if image.find(":") != -1:
+                    [machine, img] = image.split(":")
+                    fname = os.path.join(self.vars_dir, "../../",
+                                         machine, "imgdata", img + '.env')
+                else:
+                    fname = os.path.join(self.vars_dir, image + '.env')
                 if os.path.isfile(fname):
                     # parse .env file
                     with open(fname) as varsfile:
@@ -220,11 +225,16 @@ class BitbakeVars(defaultdict):
                 # Get bitbake -e output
                 cmd = "bitbake -e"
                 if image:
-                    cmd += " %s" % image
+                    if image.find(":") != -1:
+                        [machine, img] = image.split(":")
+                        cmd = "MACHINE=%s %s" % (machine, cmd)
+                    else:
+                        img = image
+                    cmd += " %s" % img
 
                 log_level = logger.getEffectiveLevel()
                 logger.setLevel(logging.INFO)
-                ret, lines = _exec_cmd(cmd)
+                ret, lines = _exec_cmd(cmd, as_shell=True)
                 logger.setLevel(log_level)
 
                 if ret:
--

2.26.1

Signature · 4 lines
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Senior Software Engineer, Mentor Graphics
#2 lore raw

Hi Christopher

Quoted 3 lines · Christopher Larson

This will only work the way you expect if hte user's local.conf uses ?= or ??= for MACHINE, otherwise setting it in the environment will do nothing.

Good catch, You are right :(,

Any idea of how could I fix it. I think I would have to change site.conf, or local.conf from wic, and I do not want to do that.

Ignoring this bug we are still covering a big number of usecases:

- bibake
- wic with user provided rootfs-dir
- wic without userprovided rootfs-dir and ?= or ??=

At the end of the day this is just a RFC ;), so i am open to any implementation. I would love to hear your thoughts

Best regards

Quoted 89 lines · Ricardo Ribalda
Quoted 77 lines

If you want to make a disk image containing rootfs for different machines, the only way to do it today is by calling wic and passing the different rootfs-paths.

Eg:
combined.wks
part /export --source rootfs --rootfs-dir=target-image part / --source rootfs

bitbake multiconfig:arm:target-image
wic create combined --rootfs-dir target-image=/workdir/build/tmp/work/arm-poky-linux/target-image/1.0-r0/rootfs/ -e main

This has many drawbacks:
1) You need to know the folder location for the target-image
2) It is easy to forget updating the target (only running wic and not bitbake)
3) It is slow
4) It does not scale when you have multiple machines

Instead, wic can be given a hint of what machine to search for:

combined.wks
part /export --source rootfs --rootfs-dir=arm:target-image part / --source rootfs

bitbake main

If we ensoure the dependency of target-image from main via do_image[mcdepends], all the dependencies are automatically handled.

This patch makes wic aware of the machine to use.

---
 scripts/lib/wic/misc.py | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)
scripts/lib/wic/misc.py +13 -4
index 91975ba151..51b43b49aa 100644
@@ -206,7 +206,12 @@ class BitbakeVars(defaultdict):
 
         if image not in self:
             if image and self.vars_dir:
-                fname = os.path.join(self.vars_dir, image + '.env')
+                if image.find(":") != -1:
+                    [machine, img] = image.split(":")
+                    fname = os.path.join(self.vars_dir, "../../",
+                                         machine, "imgdata", img + '.env')
+                else:
+                    fname = os.path.join(self.vars_dir, image + '.env')
                 if os.path.isfile(fname):
                     # parse .env file
                     with open(fname) as varsfile:
@@ -220,11 +225,16 @@ class BitbakeVars(defaultdict):
                 # Get bitbake -e output
                 cmd = "bitbake -e"
                 if image:
-                    cmd += " %s" % image
+                    if image.find(":") != -1:
+                        [machine, img] = image.split(":")
+                        cmd = "MACHINE=%s %s" % (machine, cmd)
+                    else:
+                        img = image
+                    cmd += " %s" % img
 
                 log_level = logger.getEffectiveLevel()
                 logger.setLevel(logging.INFO)
-                ret, lines = _exec_cmd(cmd)
+                ret, lines = _exec_cmd(cmd, as_shell=True)
                 logger.setLevel(log_level)
 
                 if ret:
--
Quoted 3 lines

2.26.1

--
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Senior Software Engineer, Mentor Graphics

Signature · 1 line
Ricardo Ribalda
#3 lore raw

Hi

Even if it does not cover all the use cases. Can we consider this patch?

Thanks!

Quoted 120 lines · Ricardo Ribalda Delgado

Hi Christopher

Quoted 3 lines · Christopher Larson

This will only work the way you expect if hte user's local.conf uses ?= or ??= for MACHINE, otherwise setting it in the environment will do nothing.

Good catch, You are right :(,

Any idea of how could I fix it. I think I would have to change site.conf, or local.conf from wic, and I do not want to do that.

Ignoring this bug we are still covering a big number of usecases:

- bibake
- wic with user provided rootfs-dir
- wic without userprovided rootfs-dir and ?= or ??=

At the end of the day this is just a RFC ;), so i am open to any implementation. I would love to hear your thoughts

Best regards

Quoted 76 lines · Ricardo Ribalda
Quoted 75 lines

If you want to make a disk image containing rootfs for different machines, the only way to do it today is by calling wic and passing the different rootfs-paths.

Eg:
combined.wks
part /export --source rootfs --rootfs-dir=target-image part / --source rootfs

bitbake multiconfig:arm:target-image
wic create combined --rootfs-dir target-image=/workdir/build/tmp/work/arm-poky-linux/target-image/1.0-r0/rootfs/ -e main

This has many drawbacks:
1) You need to know the folder location for the target-image
2) It is easy to forget updating the target (only running wic and not bitbake)
3) It is slow
4) It does not scale when you have multiple machines

Instead, wic can be given a hint of what machine to search for:

combined.wks
part /export --source rootfs --rootfs-dir=arm:target-image part / --source rootfs

bitbake main

If we ensoure the dependency of target-image from main via do_image[mcdepends], all the dependencies are automatically handled.

This patch makes wic aware of the machine to use.

---
 scripts/lib/wic/misc.py | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)
scripts/lib/wic/misc.py +13 -3
index 91975ba151..51b43b49aa 100644
@@ -206,7 +206,12 @@ class BitbakeVars(defaultdict):
 
         if image not in self:
             if image and self.vars_dir:
-                fname = os.path.join(self.vars_dir, image + '.env')
+                if image.find(":") != -1:
+                    [machine, img] = image.split(":")
+                    fname = os.path.join(self.vars_dir, "../../",
+                                         machine, "imgdata", img + '.env')
+                else:
+                    fname = os.path.join(self.vars_dir, image + '.env')
                 if os.path.isfile(fname):
                     # parse .env file
                     with open(fname) as varsfile:
@@ -220,11 +225,16 @@ class BitbakeVars(defaultdict):
                 # Get bitbake -e output
                 cmd = "bitbake -e"
                 if image:
-                    cmd += " %s" % image
+                    if image.find(":") != -1:
+                        [machine, img] = image.split(":")
+                        cmd = "MACHINE=%s %s" % (machine, cmd)
+                    else:
+                        img = image
+                    cmd += " %s" % img
 
                 log_level = logger.getEffectiveLevel()
                 logger.setLevel(logging.INFO)
-                ret, lines = _exec_cmd(cmd)
+                ret, lines = _exec_cmd(cmd, as_shell=True)
                 logger.setLevel(log_level)
 
Quoted 13 lines
Quoted 5 lines
scripts/lib/wic/misc.py +0 -1
                 if ret:
--

2.26.1

--
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Senior Software Engineer, Mentor Graphics

--
Ricardo Ribalda

Signature · 1 line
Ricardo Ribalda