Skip to contentSkip to conversation

Autobuilder data collection for intermittent bugs

DISCUSSION Sakib Sajal 3 participants 8 messages Mar 25, 2021 – Apr 5, 2021

lore
Share & tools
lore
#0 lore raw

Hi Richard,

I have been trying, with help from Randy, Trevor and Steve, to run some experiments on an internal autobuilder instance to collect host data during builds for signs of io latency.

Initially i modified yocto-autobuilder-helper/config.json and added the following 3 lines to existing templates (and created a commit) that were being used so that the builds would start collecting the data.

+                "BB_HEARTBEAT_EVENT = '5'", +                "BB_LOG_HOST_STAT_ON_INTERVAL = '1'", +                "BB_LOG_HOST_STAT_CMDS = 'pwd; ls'"

I also created a template in yocto-autobuilder-helper/config.json

+        "data-collect" : {
+            "BUILDINFO" : true,
+            "BBTARGETS" : "core-image-minimal", +            "extravars" : [
+                "BB_HEARTBEAT_EVENT = '5'", +                "BB_LOG_HOST_STAT_ON_INTERVAL = '1'", +                "BB_LOG_HOST_STAT_CMDS = 'pwd; ls'" +            ]
+        },

and modified a builder to use the template:

        "beaglebone-alt" : {
-            "MACHINE" : "beaglebone-yocto",
-            "TEMPLATE" : "altcfg-hw"
+            "MACHINE" : "qemux86-64",
+            "TEMPLATE" : "data-collect"
         },

Restarted both the controller and the worker, and started the beaglebone-alt build. Once the build started, i checked the conf file in the build directory but those lines were not appended to the local.conf file and hence no log files.

In summary, I have been trying to get the 3 lines from the "extravars" above to appear in one of the local.conf files. However none of the builds seems to have been altered.

I am wondering if the changes that I've made are even being picked up by the controller. Am I missing something?

Is there an easier way to update the conf files?

Initially I'd like collect data on a single node and scale it to all the nodes. Any ideas/suggestions are welcome!

Sincerely,

Sakib Sajal

#1 lore raw

Hi Sakib,

Quoted 34 lines · Sakib Sajal

I have been trying, with help from Randy, Trevor and Steve, to run some experiments on an internal autobuilder instance to collect host data during builds for signs of io latency.

Initially i modified yocto-autobuilder-helper/config.json and added the following 3 lines to existing templates (and created a commit) that were being used so that the builds would start collecting the data.

+                "BB_HEARTBEAT_EVENT = '5'", +                "BB_LOG_HOST_STAT_ON_INTERVAL = '1'", +                "BB_LOG_HOST_STAT_CMDS = 'pwd; ls'"

I also created a template in yocto-autobuilder-helper/config.json

+        "data-collect" : {
+            "BUILDINFO" : true,
+            "BBTARGETS" : "core-image-minimal", +            "extravars" : [
+                "BB_HEARTBEAT_EVENT = '5'", +                "BB_LOG_HOST_STAT_ON_INTERVAL = '1'", +                "BB_LOG_HOST_STAT_CMDS = 'pwd; ls'" +            ]
+        },

and modified a builder to use the template:

         "beaglebone-alt" : {
-            "MACHINE" : "beaglebone-yocto",
-            "TEMPLATE" : "altcfg-hw"
+            "MACHINE" : "qemux86-64",
+            "TEMPLATE" : "data-collect"
          },

Quoted 4 lines

Restarted both the controller and the worker, and started the beaglebone-alt build. Once the build started, i checked the conf file in the build directory but those lines were not appended to the local.conf file and hence no log files.

You shouldn't need to restart the controller, the worker should pull and use the configuration from the helper branch specified.

I'd note that it writes the config to auto.conf, not local.conf.

It should also give output in the logs about what its doing. For example:

https://autobuilder.yoctoproject.org/typhoon/#/builders/106/builds/2144/steps/10/logs/stdio

is the "Write config" step of beaglebone-alt on the autobuilder where it clearly says:

Writing /home/pokybuild/yocto-worker/beaglebone-alt/build/build/conf/auto.conf with contents:

What do your logs say?

Quoted 11 lines

In summary, I have been trying to get the 3 lines from the "extravars" above to appear in one of the local.conf files. However none of the builds seems to have been altered.

I am wondering if the changes that I've made are even being picked up by the controller. Am I missing something?

Is there an easier way to update the conf files?

Initially I'd like collect data on a single node and scale it to all the nodes. Any ideas/suggestions are welcome!

What you're doing should work. Its hard to comment more without seeing to logs but check you're looking at auto.conf.

Cheers,

Richard

#2 lore raw
Quoted 65 lines · Richard Purdie

Hi Sakib,

Quoted 34 lines · Sakib Sajal

I have been trying, with help from Randy, Trevor and Steve, to run some experiments on an internal autobuilder instance to collect host data during builds for signs of io latency.

Initially i modified yocto-autobuilder-helper/config.json and added the following 3 lines to existing templates (and created a commit) that were being used so that the builds would start collecting the data.

+                "BB_HEARTBEAT_EVENT = '5'", +                "BB_LOG_HOST_STAT_ON_INTERVAL = '1'", +                "BB_LOG_HOST_STAT_CMDS = 'pwd; ls'"

I also created a template in yocto-autobuilder-helper/config.json

+        "data-collect" : {
+            "BUILDINFO" : true,
+            "BBTARGETS" : "core-image-minimal", +            "extravars" : [
+                "BB_HEARTBEAT_EVENT = '5'", +                "BB_LOG_HOST_STAT_ON_INTERVAL = '1'", +                "BB_LOG_HOST_STAT_CMDS = 'pwd; ls'" +            ]
+        },

and modified a builder to use the template:

         "beaglebone-alt" : {
-            "MACHINE" : "beaglebone-yocto",
-            "TEMPLATE" : "altcfg-hw"
+            "MACHINE" : "qemux86-64",
+            "TEMPLATE" : "data-collect"
          },

Quoted 4 lines

Restarted both the controller and the worker, and started the beaglebone-alt build. Once the build started, i checked the conf file in the build directory but those lines were not appended to the local.conf file and hence no log files.

You shouldn't need to restart the controller, the worker should pull and use the configuration from the helper branch specified.

I'd note that it writes the config to auto.conf, not local.conf.

It should also give output in the logs about what its doing. For example:

https://autobuilder.yoctoproject.org/typhoon/#/builders/106/builds/2144/steps/10/logs/stdio

is the "Write config" step of beaglebone-alt on the autobuilder where it clearly says:

Writing /home/pokybuild/yocto-worker/beaglebone-alt/build/build/conf/auto.conf with contents:

What do your logs say?

Quoted 7 lines

In summary, I have been trying to get the 3 lines from the "extravars" above to appear in one of the local.conf files. However none of the builds seems to have been altered.

I am wondering if the changes that I've made are even being picked up by the controller. Am I missing something?

Quoted 7 lines
Quoted 4 lines

Is there an easier way to update the conf files?

Initially I'd like collect data on a single node and scale it to all the nodes. Any ideas/suggestions are welcome!

What you're doing should work. Its hard to comment more without seeing to logs but check you're looking at auto.conf.

Thanks for the update Sakib.

We're planning to follow Steve's example of getting commit access to
contrib branches of y-a-h and poky and using the YP AB to start custom workers/job initially. We'll also keep working to duplicate the AB behaviour locally. Trevor has asked Konrad for an additional system
or two since when jobs are running, the web UI is
painfully slow since it's runnig on the same node right now.

The one part that Sakib didn't ask about is how to get all the workers to do the data collection but that's not urgent since we'll be doing some smaller scale experiments tomorrow. If you can point out a helpful example, that'd be good.

Hopefully we'll have a good collection of data showing what processes are running when the IO latency is > 5 seconds sometime next week.

../Randy

Quoted 6 lines

Cheers,

Richard

Signature · 2 lines
# Randy MacLeod
# Wind River Linux
#3 lore raw
Quoted 9 lines · Randy MacLeod

Thanks for the update Sakib.

We're planning to follow Steve's example of getting commit access to contrib branches of y-a-h and poky and using the YP AB to start custom workers/job initially. We'll also keep working to duplicate the AB behaviour locally. Trevor has asked Konrad for an additional system or two since when jobs are running, the web UI is
painfully slow since it's runnig on the same node right now.

Quoted 4 lines

The one part that Sakib didn't ask about is how to get all the workers to do the data collection but that's not urgent since we'll be doing some smaller scale experiments tomorrow. If you can point out a helpful example, that'd be good.

The easiest is the fact that the workers all share an NFS mount which is at BASE_SHAREDDIR in config.json. See also WEBPUBLISH_DIR and WEBPUBLISH_URL.

We use this to share the reproducible build failures and to share the reproducible build failures/diffoscope and test results, e.g.:
OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail/

or perhaps more useful is scripts/collect-results:

"""
#!/bin/bash
WORKDIR=$1
DEST=$2
target=$3

RESFILE=$WORKDIR/tmp/log/oeqa/testresults.json

if [ -e $RESFILE ]; then
mkdir -p $DEST/$target
cp $WORKDIR/tmp/log/oeqa/testresults.json $DEST/$target/
fi
"""

would let you do something similar and DEST here is specific to a given build (date stamp + number). An example would be:

https://autobuilder.yocto.io/pub/non-release/20210129-3/

and then you could create a folder per target. You may need to include the worker name in the log.

Cheers,

Richard

#4 lore raw
Quoted 12 lines · Richard Purdie

[Please note: This e-mail is from an EXTERNAL e-mail address]

Quoted 9 lines · Randy MacLeod

Thanks for the update Sakib.

We're planning to follow Steve's example of getting commit access to contrib branches of y-a-h and poky and using the YP AB to start custom workers/job initially. We'll also keep working to duplicate the AB behaviour locally. Trevor has asked Konrad for an additional system or two since when jobs are running, the web UI is
painfully slow since it's runnig on the same node right now.

Quoted 7 lines
Quoted 4 lines

The one part that Sakib didn't ask about is how to get all the workers to do the data collection but that's not urgent since we'll be doing some smaller scale experiments tomorrow. If you can point out a helpful example, that'd be good.

The easiest is the fact that the workers all share an NFS mount which is at BASE_SHAREDDIR in config.json. See also WEBPUBLISH_DIR and WEBPUBLISH_URL.

Ok. We have not gone there, yet.

Quoted 21 lines

We use this to share the reproducible build failures and to share the reproducible build failures/diffoscope and test results, e.g.:
OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail/

or perhaps more useful is scripts/collect-results:

"""
#!/bin/bash
WORKDIR=$1
DEST=$2
target=$3

RESFILE=$WORKDIR/tmp/log/oeqa/testresults.json

if [ -e $RESFILE ]; then
mkdir -p $DEST/$target
cp $WORKDIR/tmp/log/oeqa/testresults.json $DEST/$target/
fi
"""

Quoted 7 lines

would let you do something similar and DEST here is specific to a given build (date stamp + number). An example would be:

https://autobuilder.yocto.io/pub/non-release/20210129-3/

and then you could create a folder per target. You may need to include the worker name in the log.

We were able to follow the example and collect results in:

https://autobuilder.yocto.io/pub/non-release/20210331-18/testresults/qa-extras2/2021-03-31--20-32/

host_stats.0

host_stats.1

host_stats.2

We are renaming the files to host_stats_<step>.txt so that it would open on the browser.

We were looking at generate-testresult-index.py and would like to append a new column called "AB INT" and hyperlink each file in the folder where "top" was triggered. Normally there should not be any links. Occasionally, we should see 1 or 2 but not more unless things go really bad. Do you agree this is a reasonable approach? Who do we contact about testing the changes? Trevor has an internal instance running, we could try it out there.

Below is the diff of the changes we plan to make for the index. We do not completely understand the script yet, we are working on it.

autobuilder/yocto-autobuilder-helper$ git diff ./scripts/generate-testresult-index.py

scripts/generate-testresult-index.py +0 -0
index d9d577e..27fe06f 100755
@@ -33,6 +33,7 @@ index_templpate = """

   <th>Performance Reports</th>
   <th>ptest Logs</th>
   <th>Buildhistory</th>
+  <th>AB INT</th>
 </tr>
 </thead>
 <tdata>
@@ -57,6 +58,11 @@ index_templpate = """
      <a href="{{bh[0]}}">{{bh[1]}}</a>
    {% endfor %}
    </td>
+   <td>
+   {% for abint in entry[8] %}
+     <a href="{{abint[0]}}">{{abint[1]}}</a>
+   {% endfor %}
+   </td>
 </tr>
 {% endfor %}
 </tdata>
@@ -145,6 +151,11 @@ for build in sorted(os.listdir(path), key=keygen, reverse=True):
     if os.path.exists(buildpath + "/qemuarm/buildhistory.txt"):
         buildhistory.append((reldir + "testresults/qemuarm/buildhistory.txt", "qemuarm"))

+    abint = []
+    for p in glob.glob(buildpath + "/ab-int*/host_stats*"):
+        abintname = os.path.basename(os.path.dirname(p))
+        abintreports.append((reldir + "testresults/" + abintname + "/" + os.path.basename(p), perfname.replace("buildperf-",""))) #todo +
     branch = get_build_branch(buildpath)

Regards,

Sakib, Randy

Quoted 5 lines

Cheers,

Richard

#5 lore raw
Quoted 59 lines · Sakib Sajal

We were able to follow the example and collect results in:

https://autobuilder.yocto.io/pub/non-release/20210331-18/testresults/qa-extras2/2021-03-31--20-32/

host_stats.0

host_stats.1

host_stats.2

We are renaming the files to host_stats_<step>.txt so that it would open on the browser.

We were looking at generate-testresult-index.py and would like to append a new column called "AB INT" and hyperlink each file in the folder where "top" was triggered. Normally there should not be any links. Occasionally, we should see 1 or 2 but not more unless things go really bad. Do you agree this is a reasonable approach? Who do we contact about testing the changes? Trevor has an internal instance running, we could try it out there.

Below is the diff of the changes we plan to make for the index. We do not completely understand the script yet, we are working on it.

autobuilder/yocto-autobuilder-helper$ git diff
./scripts/generate-testresult-index.py

scripts/generate-testresult-index.py +0 -0
index d9d577e..27fe06f 100755
@@ -33,6 +33,7 @@ index_templpate = """

    <th>Performance Reports</th>
    <th>ptest Logs</th>
    <th>Buildhistory</th>
+  <th>AB INT</th>
  </tr>
  </thead>
  <tdata>

scripts/generate-testresult-index.py +0 -0
@@ -57,6 +58,11 @@ index_templpate = """

       <a href="{{bh[0]}}">{{bh[1]}}</a>
     {% endfor %}
     </td>
+   <td>
+   {% for abint in entry[8] %}
+     <a href="{{abint[0]}}">{{abint[1]}}</a>
+   {% endfor %}
+   </td>
  </tr>
  {% endfor %}
  </tdata>

scripts/generate-testresult-index.py +0 -0
@@ -145,6 +151,11 @@ for build in sorted(os.listdir(path), key=keygen, 

reverse=True):
      if os.path.exists(buildpath + "/qemuarm/buildhistory.txt"):
          buildhistory.append((reldir + "testresults/qemuarm/buildhistory.txt", "qemuarm"))

Quoted 8 lines

+    abint = []
+    for p in glob.glob(buildpath + "/ab-int*/host_stats*"):
+        abintname = os.path.basename(os.path.dirname(p))
+        abintreports.append((reldir + "testresults/" + abintname + "/"
+ os.path.basename(p), perfname.replace("buildperf-",""))) #todo +
      branch = get_build_branch(buildpath)

Rather than messing with the main index which is "production", could you just create your own for now for testing? :)

FWIW I added tmpfs testing for qemu images into master-next (needs ab-helper master-next too) so it will be interesting to compare builds running with that with the previous build bug trends.

Cheers,

Richard

#6 lore raw
Quoted 60 lines · Richard Purdie
Quoted 59 lines · Sakib Sajal

We were able to follow the example and collect results in:

https://autobuilder.yocto.io/pub/non-release/20210331-18/testresults/qa-extras2/2021-03-31--20-32/

host_stats.0

host_stats.1

host_stats.2

We are renaming the files to host_stats_<step>.txt so that it would open on the browser.

We were looking at generate-testresult-index.py and would like to append a new column called "AB INT" and hyperlink each file in the folder where "top" was triggered. Normally there should not be any links. Occasionally, we should see 1 or 2 but not more unless things go really bad. Do you agree this is a reasonable approach? Who do we contact about testing the changes? Trevor has an internal instance running, we could try it out there.

Below is the diff of the changes we plan to make for the index. We do not completely understand the script yet, we are working on it.

autobuilder/yocto-autobuilder-helper$ git diff
./scripts/generate-testresult-index.py

scripts/generate-testresult-index.py +6 -0
index d9d577e..27fe06f 100755
@@ -33,6 +33,7 @@ index_templpate = """
     <th>Performance Reports</th>
     <th>ptest Logs</th>
     <th>Buildhistory</th>
+  <th>AB INT</th>
   </tr>
   </thead>
   <tdata>
@@ -57,6 +58,11 @@ index_templpate = """
        <a href="{{bh[0]}}">{{bh[1]}}</a>
      {% endfor %}
      </td>
+   <td>
+   {% for abint in entry[8] %}
+     <a href="{{abint[0]}}">{{abint[1]}}</a>
+   {% endfor %}
+   </td>
   </tr>
   {% endfor %}
   </tdata>
@@ -145,6 +151,11 @@ for build in sorted(os.listdir(path), key=keygen,

reverse=True):
      if os.path.exists(buildpath + "/qemuarm/buildhistory.txt"):
          buildhistory.append((reldir +
"testresults/qemuarm/buildhistory.txt", "qemuarm"))

Quoted 11 lines
Quoted 8 lines

+    abint = []
+    for p in glob.glob(buildpath + "/ab-int*/host_stats*"):
+        abintname = os.path.basename(os.path.dirname(p))
+        abintreports.append((reldir + "testresults/" + abintname + "/"
+ os.path.basename(p), perfname.replace("buildperf-",""))) #todo +
      branch = get_build_branch(buildpath)

Rather than messing with the main index which is "production", could you just create your own for now for testing? :)

Yes, we've figured all that out, thanks.
We'll send you a patch once we've completed testing on our instance of the YP AB, early this week.

Quoted 4 lines

FWIW I added tmpfs testing for qemu images into master-next (needs ab-helper master-next too) so it will be interesting to compare builds running with that with the previous build bug trends.

Any conclusions so far?

../Randy

Quoted 9 lines

Cheers,

Richard

Signature · 2 lines
# Randy MacLeod
# Wind River Linux
#7 lore raw
Quoted 10 lines · Randy MacLeod
Quoted 4 lines

Rather than messing with the main index which is "production", could you just create your own for now for testing? :)

Yes, we've figured all that out, thanks.
We'll send you a patch once we've completed testing on our instance of the YP AB, early this week.

Quoted 1 line
Quoted 5 lines
Quoted 3 lines

FWIW I added tmpfs testing for qemu images into master-next (needs ab-helper master-next too) so it will be interesting to compare builds running with that with the previous build bug trends.

Any conclusions so far?

I think it helps make the runtime testing more consistent but we are seeing a number of "qemu didn't start in 120s" messages instead. That is probably more desirable than random runtime failures in qemu.

We have still seen one valgrind ptest failure with the tmpfs patch too so it doesn't remove all of them.

It also won't fix the bitbake server starting timeout problem.

I'm leaning to merging it.

I did note that in the 120s qemu timeout, we did get a list of other processes that were running:

https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/1973

and it shows webkit being built. Would be interesting to correlate against other failures, see what the pattern of running tasks is.

Cheers,

Richard