ScrapeTargets.py: improve target search algorithm

PATCH Superseded Stephano Cetola 2 participants 2 messages Jul 1, 2017 – Jul 3, 2017

Share & tools

Stephano Cetola

author

2017-07-01 22:06 UTC #0 lore raw

PATCH 1/1 ScrapeTargets.py: improve target search algorithm

When scraping the build targets from refkit-ci.inc, grep would only return the first line of the search result. By replacing grep with awk, we are now searching for multi-line variables, and should now scrape build targets even if they have line breaks between them.

Signed-off-by Stephano Cetola <[email protected]>

---
 .../site-packages/autobuilder/buildsteps/ScrapeTargets.py        | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

lib/python2.7/site-packages/autobuilder/buildsteps/ScrapeTargets.py +4 -5

index 80153fe85..c5a2f8d6d 100644
@@ -41,7 +41,9 @@ class ScrapeTargets(ShellCommand):
         workerdir = os.path.join(os.path.join(YOCTO_ABBASE, "yocto-worker"))
         buildername = self.getProperty("buildername")
         src = os.path.join(workerdir, buildername, "build", self.source)
-        self.command = "cat " + src + " | grep " + self.targetsvar
+        # find targetsvar=", skip a line, then return lines up to a quote
+        self.command = ["awk",
+                '/%s="/{flag=1;next}/"/{flag=0}flag' % self.targetsvar, src]
         ShellCommand.start(self)
 
     def commandComplete(self, cmd):

@@ -49,10 +51,7 @@ class ScrapeTargets(ShellCommand):
             return
 
         result = cmd.logs['stdio'].getText()
-        targets = result.replace(self.targetsvar, "")
-        targets = targets.strip()
-        targets = targets.replace('=', '')
-        targets = targets.strip('"')
+        targets = result.strip()
         self.setProperty("scraped_targets",
                          targets,
                          'Targets "%s" scraped from %s' % (targets,

Signature · 1 line

2.13.1

Joshua Lock

2017-07-03 13:48 UTC #1 lore raw

Quoted 20 lines · Stephano Cetola

When scraping the build targets from refkit-ci.inc, grep would only return the first line of the search result. By replacing grep with awk, we are now searching for multi-line variables, and should now scrape build targets even if they have line breaks between them.
Signed-off-by Stephano Cetola <[email protected]>
---
.../site-packages/autobuilder/buildsteps/ScrapeTargets.py | 9
++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)
(quoted diff) +0 -0
packages/autobuilder/buildsteps/ScrapeTargets.py b/lib/python2.7/site-packages/autobuilder/buildsteps/ScrapeTargets.py index 80153fe85..c5a2f8d6d 100644
--- a/lib/python2.7/site- packages/autobuilder/buildsteps/ScrapeTargets.py
+++ b/lib/python2.7/site- packages/autobuilder/buildsteps/ScrapeTargets.py
(quoted diff) +0 -0
@@ -41,7 +41,9 @@ class ScrapeTargets(ShellCommand):

Quoted 8 lines

         workerdir = os.path.join(os.path.join(YOCTO_ABBASE, "yocto- worker"))
         buildername = self.getProperty("buildername") src = os.path.join(workerdir, buildername, "build", self.source)
-        self.command = "cat " + src + " | grep " + self.targetsvar +        # find targetsvar=", skip a line, then return lines up to a quote

What if the first line includes a desired target? Can we change this not to skip the first line?

Does including the line continuation character in the property cause BuildImages to choke later?

Quoted 21 lines

+        self.command = ["awk",
+                '/%s="/{flag=1;next}/"/{flag=0}flag' % self.targetsvar, src]
         ShellCommand.start(self)

     def commandComplete(self, cmd):
(quoted diff) +0 -0
@@ -49,10 +51,7 @@ class ScrapeTargets(ShellCommand):
             return

         result = cmd.logs['stdio'].getText()
-        targets = result.replace(self.targetsvar, "")
-        targets = targets.strip()
-        targets = targets.replace('=', '')
-        targets = targets.strip('"')
+        targets = result.strip() self.setProperty("scraped_targets",
                          targets,
                          'Targets "%s" scraped from %s' % (targets, --
2.13.1