bitbake.conf: update way of setting XZ_MEMLIMIT
PATCH 1/1 bitbake.conf: update way of setting XZ_MEMLIMIT
From: Changqing Li <[email protected]>
Update way of setting XZ_MEMLIMIT, considering scenario that running bitbake in container, to avoid OOM Killer of xz.
For example:
Container has memory limit of 30G, and host memory is 300G. 'xz --memlimit=50% ...' will get host memory, so the limit for xz is 150G. And because of the container memory limit is 30G, xz can use nearly up to 30G memory, which may cause oom kill of xz.
--- meta/conf/bitbake.conf | 3 ++- meta/lib/oe/utils.py | 12 ++++++++++++ 2 files changed, 14 insertions(+), 1 deletion(-)
meta/conf/bitbake.conf +2 -1
@@ -809,7 +809,8 @@ BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}" PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}" # Default parallelism and resource usage for xz -XZ_MEMLIMIT ?= "50%" +CONTAINER_MEM_LIMIT = "${@oe.utils.container_mem_limit()}" +XZ_MEMLIMIT ?= "${@ '%d' % (int(d.getVar('CONTAINER_MEM_LIMIT'))/2) if d.getVar('CONTAINER_MEM_LIMIT') != '0' else '50%'}" XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}" XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
meta/lib/oe/utils.py +12 -0
@@ -252,6 +252,18 @@ def cpu_count(at_least=1): cpus = len(os.sched_getaffinity(0)) return max(cpus, at_least) +def container_mem_limit(): + limit_in_bytes = '0' + proc_sched = '/proc/1/sched' + if os.path.exists(proc_sched): + with open(proc_sched, 'r') as fp: + initinfo = fp.readline().strip() + cgroup_limit_in_bytes = '/sys/fs/cgroup/memory/memory.limit_in_bytes' + if initinfo.split(' ')[0] not in ('systemd', 'init',) and os.path.exists(cgroup_limit_in_bytes): + with open(cgroup_limit_in_bytes) as fpc: + limit_in_bytes=fpc.readline().strip() + return limit_in_bytes + def execute_pre_post_process(d, cmds): if cmds is None: return
Signature · 1 line
2.17.1
Quoted 41 lines · Changqing Li
From: Changqing Li <[email protected]>
Update way of setting
XZ_MEMLIMIT, considering scenario that running bitbake in container, to avoid OOM Killer of xz.For example:
Container has memory limit of 30G, and host memory is 300G. 'xz --memlimit=50% ...' will get host memory, so the limit for xz is 150G. And because of the container memory limit is 30G, xz can use nearly up to 30G memory, which may cause oom kill of xz.Signed-off-by Changqing Li <[email protected]>--- meta/conf/bitbake.conf | 3 ++- meta/lib/oe/utils.py | 12 ++++++++++++ 2 files changed, 14 insertions(+), 1 deletion(-)meta/conf/bitbake.conf +2
-1@@ -809,7 +809,8 @@ BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}" PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}" # Default parallelism and resource usage for xz -XZ_MEMLIMIT ?= "50%" +CONTAINER_MEM_LIMIT = "${@oe.utils.container_mem_limit()}" +XZ_MEMLIMIT ?= "${@ '%d' % (int(d.getVar('CONTAINER_MEM_LIMIT'))/2) if d.getVar('CONTAINER_MEM_LIMIT') != '0' else '50%'}" XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}" XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"meta/lib/oe/utils.py +2
-0@@ -252,6 +252,18 @@ def cpu_count(at_least=1): cpus = len(os.sched_getaffinity(0)) return max(cpus, at_least) +def container_mem_limit(): + limit_in_bytes = '0'
meta/lib/oe/utils.py def cpu_count(at_least=1): · line 266
Quoted 10 lines
meta/lib/oe/utils.py +10
-0+ proc_sched = '/proc/1/sched' + if os.path.exists(proc_sched): + with open(proc_sched, 'r') as fp: + initinfo = fp.readline().strip() + cgroup_limit_in_bytes = '/sys/fs/cgroup/memory/memory.limit_in_bytes' + if initinfo.split(' ')[0] not in ('systemd', 'init',) and os.path.exists(cgroup_limit_in_bytes): + with open(cgroup_limit_in_bytes) as fpc: + limit_in_bytes=fpc.readline().strip() + return limit_in_bytes +
Whilst I understand the idea here, this is an awful lot of hardcoded values and "magic". I'm not sure we want all bitbake processes to be making these kinds of queries/decisions every time the config is parsed...
Adding the function to oe/utils may be ok but I'm not keen on making it the default.
Cheers,
Richard
Quoted 41 lines · Richard Purdie
Quoted 39 lines · Changqing Li
From: Changqing Li <[email protected]>
Update way of setting
XZ_MEMLIMIT, considering scenario that running bitbake in container, to avoid OOM Killer of xz.For example:
Container has memory limit of 30G, and host memory is 300G. 'xz --memlimit=50% ...' will get host memory, so the limit for xz is 150G. And because of the container memory limit is 30G, xz can use nearly up to 30G memory, which may cause oom kill of xz.Signed-off-by Changqing Li <[email protected]>--- meta/conf/bitbake.conf | 3 ++- meta/lib/oe/utils.py | 12 ++++++++++++ 2 files changed, 14 insertions(+), 1 deletion(-)meta/conf/bitbake.conf +2
-1@@ -809,7 +809,8 @@ BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}" PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}" # Default parallelism and resource usage for xz -XZ_MEMLIMIT ?= "50%" +CONTAINER_MEM_LIMIT = "${@oe.utils.container_mem_limit()}" +XZ_MEMLIMIT ?= "${@ '%d' % (int(d.getVar('CONTAINER_MEM_LIMIT'))/2) if d.getVar('CONTAINER_MEM_LIMIT') != '0' else '50%'}" XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}" XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"meta/lib/oe/utils.py +0
-0@@ -252,6 +252,18 @@ def cpu_count(at_least=1): cpus = len(os.sched_getaffinity(0)) return max(cpus, at_least)
Quoted 16 lines
meta/lib/oe/utils.py def cpu_count(at_least=1): · line 266
Quoted 12 lines
meta/lib/oe/utils.py +12
-0+def container_mem_limit(): + limit_in_bytes = '0' + proc_sched = '/proc/1/sched' + if os.path.exists(proc_sched): + with open(proc_sched, 'r') as fp: + initinfo = fp.readline().strip() + cgroup_limit_in_bytes = '/sys/fs/cgroup/memory/memory.limit_in_bytes' + if initinfo.split(' ')[0] not in ('systemd', 'init',) and os.path.exists(cgroup_limit_in_bytes): + with open(cgroup_limit_in_bytes) as fpc: + limit_in_bytes=fpc.readline().strip() + return limit_in_bytes +Whilst I understand the idea here, this is an awful lot of hardcoded values and "magic". I'm not sure we want all bitbake processes to be making these kinds of queries/decisions every time the config is parsed...
I'm sure this was discussed before but I forget the answer... how much compression / performance is lost by setting the XZ memory limit to a fixed value which is reasonable for all systems (e.g. 1GB? 256MB * XZ_THREADS?).
As well as making the code simpler should also make builds more reproducible by not depending on some detail which is specific to the host (ie total memory size).
Quoted 12 lines
Adding the function to oe/utils may be ok but I'm not keen on making it the default.
Cheers,
Richard