bitbake: invalidate mtime cache if file doesn't exist
PATCH 1/1 bitbake: invalidate mtime cache if file doesn't exist
Mtime cache is desinged with assumption that files are not removed. Unfortunately it's not always the case for memory-resident bitbake as build/tmp can be removed to perform build from scratch. In this case bitbake crashes when trying to create timestamps if tmp/stamps/ hierarchy doesn't exist.
Simple check of file existance should solve this issue.
[YOCTO: #7562]
--- bitbake/lib/bb/parse/__init__.py | 2 ++ 1 file changed, 2 insertions(+)
bitbake/lib/bb/parse/__init__.py +2 -0
@@ -62,6 +62,8 @@ def cached_mtime(f): return __mtime_cache[f] def cached_mtime_noerror(f): + if not os.path.exists(f): + return 0 if f not in __mtime_cache: try: __mtime_cache[f] = os.stat(f)[stat.ST_MTIME]
Signature · 1 line
2.1.4
Quoted 20 lines · Ed Bartosh
Mtime cache is desinged with assumption that files are not removed. Unfortunately it's not always the case for memory-resident bitbake as build/tmp can be removed to perform build from scratch. In this case bitbake crashes when trying to create timestamps if tmp/stamps/ hierarchy doesn't exist.
Simple check of file existance should solve this issue.
[YOCTO: #7562]
Signed-off-by Ed Bartosh <[email protected]>--- bitbake/lib/bb/parse/__init__.py | 2 ++ 1 file changed, 2 insertions(+)bitbake/lib/bb/parse/__init__.py +0
-0
bitbake/lib/bb/parse/__init__.py def cached_mtime(f): · line 69
Quoted 8 lines
bitbake/lib/bb/parse/__init__.py +2
-0@@ -62,6 +62,8 @@ def cached_mtime(f): return __mtime_cache[f] def cached_mtime_noerror(f): + if not os.path.exists(f): + return 0 if f not in __mtime_cache: try: __mtime_cache[f] = os.stat(f)[stat.ST_MTIME]
I don't think this is right. The whole idea of this cache is so we don't repeatedly hit stat() syscalls and by adding in the exists() check, the cache is made worthless and I suspect we'd see a bad performance change.
Most of the use of this cache is from cooker/cache.py and there, we use inotify to refresh the cache if something in the filesystem changes. We probably need to teach the other users of this cache about the need to deal with inotify (or simply remove any stamp data from the cache entirely at the start of each bitbake invocation).
Cheers,
Richard
PATCH 1/1 bitbake: reset mtime cache before the build
Mtime cache is desinged with assumption that files are not removed. Unfortunately it's not always the case for memory-resident bitbake as build/tmp can be removed to perform build from scratch. In this case bitbake crashes when trying to create timestamps if tmp/stamps/ hierarchy doesn't exist.
Resetting mtime cache should solve this issue.
[YOCTO: #7562]
--- bitbake/lib/bb/cooker.py | 1 + bitbake/lib/bb/parse/__init__.py | 4 ++++ 2 files changed, 5 insertions(+)
bitbake/lib/bb/cooker.py +1 -0
@@ -1343,6 +1343,7 @@ class BBCooker: return True return retval + parse.reset_cache() self.buildSetVars() taskdata, runlist, fulltargetlist = self.buildTaskData(targets, task, self.configuration.abort)
bitbake/lib/bb/parse/__init__.py +4 -0
@@ -78,6 +78,10 @@ def update_cache(f): logger.debug(1, "Updating mtime cache for %s" % f) update_mtime(f) +def reset_cache(): + global __mtime_cache + __mtime_cache = {} + def mark_dependency(d, f): if f.startswith('./'): f = "%s/%s" % (os.getcwd(), f[2:])
Signature · 1 line
2.1.4
PATCH 1/1 bitbake: reset build mtime cache before the build
In reply to Richard PurdieIntroduced build mtime cache structure. Reset it before the build to prevent bitbake from crashing when build/tmp/stamps hierarchy is removed.
[YOCTO: #7562]
--- bitbake/lib/bb/build.py | 17 ++++++++++++++++- bitbake/lib/bb/cooker.py | 3 ++- 2 files changed, 18 insertions(+), 2 deletions(-)
bitbake/lib/bb/build.py +16 -1
@@ -31,6 +31,7 @@ import logging import shlex import glob import time +import stat import bb import bb.msg import bb.process
@@ -42,6 +43,20 @@ logger = logging.getLogger('BitBake.Build') NULL = open(os.devnull, 'r+') +__mtime_cache = {} + +def cached_mtime_noerror(f): + if f not in __mtime_cache: + try: + __mtime_cache[f] = os.stat(f)[stat.ST_MTIME] + except OSError: + return 0 + return __mtime_cache[f] + +def reset_cache(): + global __mtime_cache + __mtime_cache = {} + # When we execute a Python function, we'd like certain things # in all namespaces, hence we add them to __builtins__. # If we do not do this and use the exec globals, they will
@@ -535,7 +550,7 @@ def stamp_internal(taskname, d, file_name, baseonly=False): stamp = bb.parse.siggen.stampfile(stamp, file_name, taskname, extrainfo) stampdir = os.path.dirname(stamp) - if bb.parse.cached_mtime_noerror(stampdir) == 0: + if cached_mtime_noerror(stampdir) == 0: bb.utils.mkdirhier(stampdir) return stamp
bitbake/lib/bb/cooker.py +2 -1
@@ -35,7 +35,7 @@ from contextlib import closing from functools import wraps from collections import defaultdict import bb, bb.exceptions, bb.command -from bb import utils, data, parse, event, cache, providers, taskdata, runqueue +from bb import utils, data, parse, event, cache, providers, taskdata, runqueue, build import Queue import signal import prserv.serv
@@ -1343,6 +1343,7 @@ class BBCooker: return True return retval + build.reset_cache() self.buildSetVars() taskdata, runlist, fulltargetlist = self.buildTaskData(targets, task, self.configuration.abort)
Signature · 1 line
2.1.4