build/utils: Add BB_TASK_IONICE_LEVEL support

PATCH In review Richard Purdie 4 participants 5 messages Oct 24, 2015 – Oct 27, 2015

Share & tools

Richard Purdie

author

2015-10-24 10:19 UTC #0 lore raw

PATCH 1/1 build/utils: Add BB_TASK_IONICE_LEVEL support

Similarly to BB_TASK_NICE_LEVEL, add BB_TASK_IONICE_LEVEL which allows the ioprio of tasks to be adjusted. This is in response to various qemu runtime timeouts which have been witnessed on the autobuilder, seemingly due to IO starvation (we already use NICE_LEVEL to adjust tasks). This has a fairly urgent need to deal with certain 'random' failures we're seeing on the autobuilders in testing.

The format of the data in the variable is BB_TASK_IONICE_LEVEL = "<class>.<prio>".

For <class>, 2 is best effort (the default), 1 is real time and 3 is idle. You'd need superuser privileges to use realtime. The <prio> value is a default of 4, and can be set between 0 and 7 with 7 being lowest priority and 0 the highest. The user can set this freely with normal privileges

Note that in order for this to take effect, you need the cfq scheduler selected for the backing block device.

We could use nice wrapper functions for ioprio from modules like psutil however that would complicate bitbake dependencies. This version has some magic numbers but works on the main 32 and 64 bit x86 build architectures and can easily be extended if ever needed. When we move to python 3.x, we can likely replace this with standard calls.

Signed-off-by Richard Purdie <[email protected]>

bitbake/lib/bb/build.py +7 -0

index 948c395..22428a6 100644
@@ -413,6 +413,13 @@ def _exec_task(fn, task, d, quieterr):
         nice = int(nice) - curnice
         newnice = os.nice(nice)
         logger.debug(1, "Renice to %s " % newnice)
+    ionice = localdata.getVar("BB_TASK_IONICE_LEVEL", True)
+    if ionice:
+        try:
+            cls, prio = ionice.split(".", 1)
+            bb.utils.ioprio_set(os.getpid(), int(cls), int(prio))
+        except:
+            bb.warn("Invalid ionice level %s" % ionice)
 
     bb.utils.mkdirhier(tempdir)

bitbake/lib/bb/utils.py +24 -0

index eb219c1..86a5328 100644
@@ -1310,3 +1310,27 @@ def signal_on_parent_exit(signame):
     result = cdll['libc.so.6'].prctl(PR_SET_PDEATHSIG, signum)
     if result != 0:
         raise PrCtlError('prctl failed with error code %s' % result)
+
+#
+# Manually call the ioprio syscall. We could depend on other libs like psutil
+# however this gets us enough of what we need to bitbake for now without the
+# dependency
+#
+_unamearch = os.uname()[4]
+IOPRIO_WHO_PROCESS = 1
+IOPRIO_CLASS_SHIFT = 13
+
+def ioprio_set(who, cls, value):
+    NR_ioprio_set = None
+    if _unamearch == "x86_64":
+      NR_ioprio_set = 251
+    elif _unamearch[0] == "i" and _unamearch[2:3] == "86":
+      NR_ioprio_set = 289
+
+    if NR_ioprio_set:
+        ioprio = value | (cls << IOPRIO_CLASS_SHIFT)
+        rc = cdll['libc.so.6'].syscall(NR_ioprio_set, IOPRIO_WHO_PROCESS, who, ioprio)
+        if rc != 0:
+            raise ValueError("Unable to set ioprio, syscall returned %s" % rc)
+    else:
+        bb.warn("Unable to set IO Prio for arch %s" % _unamearch)

Christopher Larson

2015-10-24 22:26 UTC #1 lore raw

Quoted 33 lines · Richard Purdie

bitbake/lib/bb/utils.py +3 -0

index eb219c1..86a5328 100644
@@ -1310,3 +1310,27 @@ def signal_on_parent_exit(signame):
     result = cdll['libc.so.6'].prctl(PR_SET_PDEATHSIG, signum)
     if result != 0:
         raise PrCtlError('prctl failed with error code %s' % result)
+
+#
+# Manually call the ioprio syscall. We could depend on other libs like

psutil
+# however this gets us enough of what we need to bitbake for now without the

bitbake/lib/bb/utils.py +16 -0

+# dependency
+#
+_unamearch = os.uname()[4]
+IOPRIO_WHO_PROCESS = 1
+IOPRIO_CLASS_SHIFT = 13
+
+def ioprio_set(who, cls, value):
+    NR_ioprio_set = None
+    if _unamearch == "x86_64":
+      NR_ioprio_set = 251
+    elif _unamearch[0] == "i" and _unamearch[2:3] == "86":
+      NR_ioprio_set = 289
+
+    if NR_ioprio_set:
+        ioprio = value | (cls << IOPRIO_CLASS_SHIFT)
+        rc = cdll['libc.so.6'].syscall(NR_ioprio_set, IOPRIO_WHO_PROCESS,

who, ioprio)

bitbake/lib/bb/utils.py +2 -0

+        if rc != 0:
+            raise ValueError("Unable to set ioprio, syscall returned %s"

Quoted 2 lines

% rc)

Question, is the only way the syscall ran return non-zero the case where the ioprio vaue is wrong? If it can fail for any other reason (e.g. interrupted syscall), this probably isn't always a ValueError.

Signature · 5 lines

Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Scott Rifenbark

2015-10-27 13:34 UTC #2 lore raw

In reply to Richard Purdie

Hi,

See
http://www.yoctoproject.org/docs/2.0/bitbake-user-manual/bitbake-user-manual.html#var-BB_TASK_IONICE_LEVEL
for the new variable description in the BitBake User Manual. Note that I did not include the future implementation information at the bottom of Richard's explanation. Also, there is a bit about the 'cfq scheduler' near the end. I did not know how to properly format this string (e.g. CFG Scheduler).

Let me know of any changes or fixes.

Thanks,
Scott

Quoted 94 lines · Richard Purdie

Similarly to BB_TASK_NICE_LEVEL, add BB_TASK_IONICE_LEVEL which allows the ioprio
of tasks to be adjusted. This is in response to various qemu runtime timeouts
which have been witnessed on the autobuilder, seemingly due to IO starvation (we
already use NICE_LEVEL to adjust tasks). This has a fairly urgent need to deal
with certain 'random' failures we're seeing on the autobuilders in testing.
The format of the data in the variable is BB_TASK_IONICE_LEVEL = "<class>.<prio>".
For <class>, 2 is best effort (the default), 1 is real time and 3 is idle. You'd
need superuser privileges to use realtime. The <prio> value is a default of 4,
and can be set between 0 and 7 with 7 being lowest priority and 0 the highest.
The user can set this freely with normal privileges
Note that in order for this to take effect, you need the cfq scheduler selected
for the backing block device.
We could use nice wrapper functions for ioprio from modules like psutil however
that would complicate bitbake dependencies. This version has some magic numbers
but works on the main 32 and 64 bit x86 build architectures and can easily be
extended if ever needed. When we move to python 3.x, we can likely replace this
with standard calls.
Signed-off-by Richard Purdie <[email protected]>
bitbake/lib/bb/build.py +7 -0
index 948c395..22428a6 100644
@@ -413,6 +413,13 @@ def _exec_task(fn, task, d, quieterr):
         nice = int(nice) - curnice
         newnice = os.nice(nice)
         logger.debug(1, "Renice to %s " % newnice)
+    ionice = localdata.getVar("BB_TASK_IONICE_LEVEL", True)
+    if ionice:
+        try:
+            cls, prio = ionice.split(".", 1)
+            bb.utils.ioprio_set(os.getpid(), int(cls), int(prio))
+        except:
+            bb.warn("Invalid ionice level %s" % ionice)
 
     bb.utils.mkdirhier(tempdir)
 
bitbake/lib/bb/utils.py +3 -0
index eb219c1..86a5328 100644
@@ -1310,3 +1310,27 @@ def signal_on_parent_exit(signame):
     result = cdll['libc.so.6'].prctl(PR_SET_PDEATHSIG, signum)
     if result != 0:
         raise PrCtlError('prctl failed with error code %s' % result)
+
+#
+# Manually call the ioprio syscall. We could depend on other libs like
psutil
+# however this gets us enough of what we need to bitbake for now without the
bitbake/lib/bb/utils.py +16 -0
+# dependency
+#
+_unamearch = os.uname()[4]
+IOPRIO_WHO_PROCESS = 1
+IOPRIO_CLASS_SHIFT = 13
+
+def ioprio_set(who, cls, value):
+    NR_ioprio_set = None
+    if _unamearch == "x86_64":
+      NR_ioprio_set = 251
+    elif _unamearch[0] == "i" and _unamearch[2:3] == "86":
+      NR_ioprio_set = 289
+
+    if NR_ioprio_set:
+        ioprio = value | (cls << IOPRIO_CLASS_SHIFT)
+        rc = cdll['libc.so.6'].syscall(NR_ioprio_set, IOPRIO_WHO_PROCESS,
who, ioprio)
bitbake/lib/bb/utils.py +2 -0
+        if rc != 0:
+            raise ValueError("Unable to set ioprio, syscall returned %s"
% rc)
bitbake/lib/bb/utils.py +2 -0
+    else:
+        bb.warn("Unable to set IO Prio for arch %s" % _unamearch)

Avery, Brian

2015-10-27 14:59 UTC #3 lore raw

Hi,

It mostly appears as CFQ scheduler (Completely Fair Queuing). Since we tell them to set it to cfq, it might be nice to tell them to do this by doing sudo sh -c “echo cfq > /sys/block/<device>/queu/scheduler; where device is the backing block device e.g. sda,sdb,…

-b

From: Scott Rifenbark <[email protected]<mailto:[email protected]>>
Date: Tuesday, October 27, 2015 at 6:34 AM
To: Richard Purdie <[email protected]<mailto:[email protected]>>
Cc: bitbake-devel <[email protected]<mailto:[email protected]>>, Brian Avery <[email protected]<mailto:[email protected]>>, "Witt, Randy E" <[email protected]<mailto:[email protected]>>, "Flanagan, Elizabeth" <[email protected]<mailto:[email protected]>>, "Graydon, Tracy" <[email protected]<mailto:[email protected]>>, Michael Halstead <[email protected]<mailto:[email protected]>>
Subject: Re: [PATCH] build/utils: Add BB_TASK_IONICE_LEVEL support

Hi,

See http://www.yoctoproject.org/docs/2.0/bitbake-user-manual/bitbake-user-manual.html#var-BB_TASK_IONICE_LEVEL for the new variable description in the BitBake User Manual. Note that I did not include the future implementation information at the bottom of Richard's explanation. Also, there is a bit about the 'cfq scheduler' near the end. I did not know how to properly format this string (e.g. CFG Scheduler).

Let me know of any changes or fixes.

Thanks,
Scott

On Sat, Oct 24, 2015 at 3:19 AM, Richard Purdie <[email protected]<mailto:[email protected]>> wrote:
Similarly to BB_TASK_NICE_LEVEL, add BB_TASK_IONICE_LEVEL which allows the ioprio
of tasks to be adjusted. This is in response to various qemu runtime timeouts
which have been witnessed on the autobuilder, seemingly due to IO starvation (we
already use NICE_LEVEL to adjust tasks). This has a fairly urgent need to deal
with certain 'random' failures we're seeing on the autobuilders in testing.

The format of the data in the variable is BB_TASK_IONICE_LEVEL = "<class>.<prio>".

Note that in order for this to take effect, you need the cfq scheduler selected for the backing block device.

Signed-off-by Richard Purdie <[email protected]<mailto:[email protected]>>

bitbake/lib/bb/build.py +7 -0

index 948c395..22428a6 100644
@@ -413,6 +413,13 @@ def _exec_task(fn, task, d, quieterr):
         nice = int(nice) - curnice
         newnice = os.nice(nice)
         logger.debug(1, "Renice to %s " % newnice)
+    ionice = localdata.getVar("BB_TASK_IONICE_LEVEL", True)
+    if ionice:
+        try:
+            cls, prio = ionice.split(".", 1)
+            bb.utils.ioprio_set(os.getpid(), int(cls), int(prio))
+        except:
+            bb.warn("Invalid ionice level %s" % ionice)
 
     bb.utils.mkdirhier(tempdir)

bitbake/lib/bb/utils.py +24 -0

index eb219c1..86a5328 100644
@@ -1310,3 +1310,27 @@ def signal_on_parent_exit(signame):
     result = cdll['libc.so.6'].prctl(PR_SET_PDEATHSIG, signum)
     if result != 0:
         raise PrCtlError('prctl failed with error code %s' % result)
+
+#
+# Manually call the ioprio syscall. We could depend on other libs like psutil
+# however this gets us enough of what we need to bitbake for now without the
+# dependency
+#
+_unamearch = os.uname()[4]
+IOPRIO_WHO_PROCESS = 1
+IOPRIO_CLASS_SHIFT = 13
+
+def ioprio_set(who, cls, value):
+    NR_ioprio_set = None
+    if _unamearch == "x86_64":
+      NR_ioprio_set = 251
+    elif _unamearch[0] == "i" and _unamearch[2:3] == "86":
+      NR_ioprio_set = 289
+
+    if NR_ioprio_set:
+        ioprio = value | (cls << IOPRIO_CLASS_SHIFT)
+        rc = cdll['libc.so.6'].syscall(NR_ioprio_set, IOPRIO_WHO_PROCESS, who, ioprio)
+        if rc != 0:
+            raise ValueError("Unable to set ioprio, syscall returned %s" % rc)
+    else:
+        bb.warn("Unable to set IO Prio for arch %s" % _unamearch)

Scott Rifenbark

2015-10-27 15:09 UTC #4 lore raw

Quoted 7 lines · Brian

Hi,
It mostly appears as CFQ scheduler (Completely Fair Queuing). Since we tell them to set it to cfq, it might be nice to tell them to do this by doing sudo sh -c “echo cfq > /sys/block/<device>/queu/scheduler; where device is the backing block device e.g. sda,sdb,…

Brian - I updated that note accordingly. Is "queu" correct? Or should it be "queue"?

Thanks

Quoted 126 lines

-b
From: Scott Rifenbark <[email protected]>
Date: Tuesday, October 27, 2015 at 6:34 AM
To: Richard Purdie <[email protected]>
Cc: bitbake-devel <[email protected]>, Brian Avery < [email protected]>, "Witt, Randy E" <[email protected]>, "Flanagan, Elizabeth" <[email protected]>, "Graydon, Tracy" < [email protected]>, Michael Halstead <[email protected]>
Subject: Re: [PATCH] build/utils: Add BB_TASK_IONICE_LEVEL support
Hi,
See
http://www.yoctoproject.org/docs/2.0/bitbake-user-manual/bitbake-user-manual.html#var-BB_TASK_IONICE_LEVEL
for the new variable description in the BitBake User Manual. Note that I did not include the future implementation information at the bottom of Richard's explanation. Also, there is a bit about the 'cfq scheduler' near the end. I did not know how to properly format this string (e.g. CFG Scheduler).
Let me know of any changes or fixes.
Thanks,
Scott
Quoted 95 lines · Richard Purdie
Similarly to BB_TASK_NICE_LEVEL, add BB_TASK_IONICE_LEVEL which allows the ioprio
of tasks to be adjusted. This is in response to various qemu runtime timeouts
which have been witnessed on the autobuilder, seemingly due to IO starvation (we
already use NICE_LEVEL to adjust tasks). This has a fairly urgent need to deal
with certain 'random' failures we're seeing on the autobuilders in testing.
The format of the data in the variable is BB_TASK_IONICE_LEVEL = "<class>.<prio>".
For <class>, 2 is best effort (the default), 1 is real time and 3 is idle. You'd
need superuser privileges to use realtime. The <prio> value is a default of 4,
and can be set between 0 and 7 with 7 being lowest priority and 0 the highest.
The user can set this freely with normal privileges
Note that in order for this to take effect, you need the cfq scheduler selected
for the backing block device.
We could use nice wrapper functions for ioprio from modules like psutil however
that would complicate bitbake dependencies. This version has some magic numbers
but works on the main 32 and 64 bit x86 build architectures and can easily be
extended if ever needed. When we move to python 3.x, we can likely replace this
with standard calls.
Signed-off-by Richard Purdie <[email protected]>
bitbake/lib/bb/build.py +7 -0
index 948c395..22428a6 100644
@@ -413,6 +413,13 @@ def _exec_task(fn, task, d, quieterr):
         nice = int(nice) - curnice
         newnice = os.nice(nice)
         logger.debug(1, "Renice to %s " % newnice)
+    ionice = localdata.getVar("BB_TASK_IONICE_LEVEL", True)
+    if ionice:
+        try:
+            cls, prio = ionice.split(".", 1)
+            bb.utils.ioprio_set(os.getpid(), int(cls), int(prio))
+        except:
+            bb.warn("Invalid ionice level %s" % ionice)
 
     bb.utils.mkdirhier(tempdir)
 
bitbake/lib/bb/utils.py +3 -0
index eb219c1..86a5328 100644
@@ -1310,3 +1310,27 @@ def signal_on_parent_exit(signame):
     result = cdll['libc.so.6'].prctl(PR_SET_PDEATHSIG, signum)
     if result != 0:
         raise PrCtlError('prctl failed with error code %s' % result)
+
+#
+# Manually call the ioprio syscall. We could depend on other libs like
psutil
+# however this gets us enough of what we need to bitbake for now without the
bitbake/lib/bb/utils.py +16 -0
+# dependency
+#
+_unamearch = os.uname()[4]
+IOPRIO_WHO_PROCESS = 1
+IOPRIO_CLASS_SHIFT = 13
+
+def ioprio_set(who, cls, value):
+    NR_ioprio_set = None
+    if _unamearch == "x86_64":
+      NR_ioprio_set = 251
+    elif _unamearch[0] == "i" and _unamearch[2:3] == "86":
+      NR_ioprio_set = 289
+
+    if NR_ioprio_set:
+        ioprio = value | (cls << IOPRIO_CLASS_SHIFT)
+        rc = cdll['libc.so.6'].syscall(NR_ioprio_set,
IOPRIO_WHO_PROCESS, who, ioprio)
bitbake/lib/bb/utils.py +2 -0
+        if rc != 0:
+            raise ValueError("Unable to set ioprio, syscall returned %s"
% rc)
bitbake/lib/bb/utils.py +2 -0
+    else:
+        bb.warn("Unable to set IO Prio for arch %s" % _unamearch)