bitbake: parsing time improvement
Hi Richard,
This pull request is to improve parsing time by getting use of the expand cache, please help to review and pull.
Thanks,
Dongxiao
The following changes since commit 0423587db09f6f28cf9d801f5657a84157f42dbe:
hob: disable some menu entries whilst build is in progress (2011-08-24 19:32:42 -0700)
are available in the git repository at:
git://git.pokylinux.org/poky-contrib dxu4/parse
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=dxu4/parse
Dongxiao Xu (1):
data_smart.py: make use of expand cache in getVar()
lib/bb/data_smart.py | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletions(-)
PATCH 1/1 data_smart.py: make use of expand cache in getVar()
Currently if passing expand=True to getVar() function, it will pass the handling to getVarFlag(), which doesn't get any benefit from the expand cache.
Call the expand() function separately in getVar() to make use of the expand cache, which can decrease the parsing time by 40%. (from current 49s to 27s)
--- lib/bb/data_smart.py | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletions(-)
lib/bb/data_smart.py +6 -1
@@ -268,7 +268,12 @@ class DataSmart(MutableMapping): self.dict[var]["content"] = value def getVar(self, var, expand=False, noweakdefault=False): - return self.getVarFlag(var, "content", expand, noweakdefault) + value = self.getVarFlag(var, "content", False, noweakdefault) + + # Call expand() separately to make use of the expand cache + if expand and value: + return self.expand(value, var) + return value def renameVar(self, key, newkey): """
Signature · 1 line
1.7.1
Re: [PATCH 1/1] data_smart.py: make use of expand cache in getVar()
Quoted 19 lines · Dongxiao Xu
Currently if passing expand=True to
getVar()function, it will pass the handling togetVarFlag(), which doesn't get any benefit from the expand cache.Call the
expand()function separately ingetVar()to make use of the expand cache, which can decrease the parsing time by 40%. (from current 49s to 27s)Signed-off-by Dongxiao Xu <[email protected]>--- lib/bb/data_smart.py | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletions(-)lib/bb/data_smart.py +0
-0@@ -268,7 +268,12 @@ class DataSmart(MutableMapping): self.dict[var]["content"] = value
lib/bb/data_smart.py class DataSmart(MutableMapping): · line 277
Quoted 10 lines
lib/bb/data_smart.py +6
-1def getVar(self, var, expand=False, noweakdefault=False): - return self.getVarFlag(var, "content", expand, noweakdefault) + value = self.getVarFlag(var, "content", False, noweakdefault) + + # Call expand() separately to make use of the expand cache + if expand and value: + return self.expand(value, var) + return value
Why can't we have getVarFlag use the expand cache?
Cheers,
Richard
Re: [PATCH 1/1] data_smart.py: make use of expand cache in getVar()
Hi Richard,
Quoted 29 lines
-----Original Message-----
From: Richard Purdie [mailto:[email protected]]
Sent: Monday, August 29, 2011 8:55 PM
To: Xu, Dongxiao
Cc: [email protected]
Subject: Re: [bitbake-devel] [PATCH 1/1]data_smart.py: make use of expand cache ingetVar()Quoted 20 lines · Dongxiao Xu
Currently if passing expand=True to
getVar()function, it will pass the handling togetVarFlag(), which doesn't get any benefit from the expand cache.Call the
expand()function separately ingetVar()to make use of the expand cache, which can decrease the parsing time by 40%. (from current 49s to 27s)Signed-off-by Dongxiao Xu <[email protected]>--- lib/bb/data_smart.py | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletions(-)lib/bb/data_smart.py index renamed from lib/bb/data_smart.py +0
-0301f9e3..d8ba24f 100644
lib/bb/data_smart.py +0
-0@@ -268,7 +268,12 @@ class DataSmart(MutableMapping): self.dict[var]["content"] = value
Quoted 11 lines
lib/bb/data_smart.py class DataSmart(MutableMapping): · line 277
Quoted 9 lines
lib/bb/data_smart.py +6
-1def getVar(self, var, expand=False, noweakdefault=False): - return self.getVarFlag(var, "content", expand, noweakdefault) + value = self.getVarFlag(var, "content", False, noweakdefault) + + # Call expand() separately to make use of the expand cache + if expand and value: + return self.expand(value, var) + return valueWhy can't we have getVarFlag use the expand cache?
For expand cache, the format is a list: expand_cache[varname]. It doesn't contain information of flag values.
Thanks,
Dongxiao
Quoted 4 lines
Cheers,
Richard
Re: [PATCH 1/1] data_smart.py: make use of expand cache in getVar()
Quoted 43 lines · Dongxiao
Hi Richard,
Quoted 29 lines
-----Original Message-----
From: Richard Purdie [mailto:[email protected]]
Sent: Monday, August 29, 2011 8:55 PM
To: Xu, Dongxiao
Cc: [email protected]
Subject: Re: [bitbake-devel] [PATCH 1/1]data_smart.py: make use of expand cache ingetVar()Quoted 20 lines · Dongxiao Xu
Currently if passing expand=True to
getVar()function, it will pass the handling togetVarFlag(), which doesn't get any benefit from the expand cache.Call the
expand()function separately ingetVar()to make use of the expand cache, which can decrease the parsing time by 40%. (from current 49s to 27s)Signed-off-by Dongxiao Xu <[email protected]>--- lib/bb/data_smart.py | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletions(-)lib/bb/data_smart.py index renamed from lib/bb/data_smart.py +0
-0301f9e3..d8ba24f 100644
lib/bb/data_smart.py +0
-0@@ -268,7 +268,12 @@ class DataSmart(MutableMapping): self.dict[var]["content"] = valueQuoted 11 lines
lib/bb/data_smart.py class DataSmart(MutableMapping): · line 277
Quoted 9 lines
lib/bb/data_smart.py +6
-1def getVar(self, var, expand=False, noweakdefault=False): - return self.getVarFlag(var, "content", expand, noweakdefault) + value = self.getVarFlag(var, "content", False, noweakdefault) + + # Call expand() separately to make use of the expand cache + if expand and value: + return self.expand(value, var) + return valueWhy can't we have getVarFlag use the expand cache?
Quoted 2 lines
For expand cache, the format is a list: expand_cache[varname]. It doesn't contain information of flag values.
Good point.
I'm left wondering why we don't key that cache on the unexpanded value rather than a specific variable name (or name+flag).
I'll take this patch but its something we should consider...
Cheers,
Richard
Re: [PATCH 1/1] data_smart.py: make use of expand cache in getVar()
Quoted 56 lines
-----Original Message-----
From: Richard Purdie [mailto:[email protected]]
Sent: Monday, August 29, 2011 9:13 PM
To: Xu, Dongxiao
Cc: [email protected]
Subject: RE: [bitbake-devel] [PATCH 1/1]data_smart.py: make use of expand cache ingetVar()Quoted 44 lines · Dongxiao
Hi Richard,
Quoted 29 lines
-----Original Message-----
From: Richard Purdie [mailto:[email protected]]
Sent: Monday, August 29, 2011 8:55 PM
To: Xu, Dongxiao
Cc: [email protected]
Subject: Re: [bitbake-devel] [PATCH 1/1]data_smart.py: make use of expand cache ingetVar()Quoted 20 lines · Dongxiao Xu
Currently if passing expand=True to
getVar()function, it will pass the handling togetVarFlag(), which doesn't get any benefit from the expand cache.Call the
expand()function separately ingetVar()to make use of the expand cache, which can decrease the parsing time by 40%. (from current 49s to 27s)Signed-off-by Dongxiao Xu <[email protected]>--- lib/bb/data_smart.py | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletions(-)lib/bb/data_smart.py index renamed from lib/bb/data_smart.py +0
-0301f9e3..d8ba24f 100644
lib/bb/data_smart.py +0
-0@@ -268,7 +268,12 @@ class DataSmart(MutableMapping): self.dict[var]["content"] = valueQuoted 12 lines
lib/bb/data_smart.py class DataSmart(MutableMapping): · line 277
Quoted 10 lines
lib/bb/data_smart.py +7
-1def getVar(self, var, expand=False, noweakdefault=False): - return self.getVarFlag(var, "content", expand, noweakdefault) + value = self.getVarFlag(var, "content", False, + noweakdefault) + + # Call expand() separately to make use of the expand cache + if expand and value: + return self.expand(value, var) + return valueWhy can't we have getVarFlag use the expand cache?
Quoted 2 lines
For expand cache, the format is a list: expand_cache[varname]. It doesn't contain information of flag values.
Quoted 6 lines
Good point.
I'm left wondering why we don't key that cache on the unexpanded value rather than a specific variable name (or name+flag).
I'll take this patch but its something we should consider...
Thanks, I will step into this point further and see if (name+flag) type of cache could benefit us more.
Thanks,
Dongxiao
Quoted 6 lines
Cheers,
Richard
Re: [PATCH 1/1] data_smart.py: make use of expand cache in getVar()
In reply to Richard PurdieHi Richard,
Quoted 66 lines
-----Original Message-----
From: Xu, Dongxiao
Sent: Monday, August 29, 2011 9:16 PM
To: Richard Purdie
Cc: [email protected]
Subject: RE: [bitbake-devel] [PATCH 1/1]data_smart.py: make use of expand cache ingetVar()Quoted 58 lines
-----Original Message-----
From: Richard Purdie [mailto:[email protected]]
Sent: Monday, August 29, 2011 9:13 PM
To: Xu, Dongxiao
Cc: [email protected]
Subject: RE: [bitbake-devel] [PATCH 1/1]data_smart.py: make use of expand cache ingetVar()Quoted 44 lines · Dongxiao
Hi Richard,
Quoted 29 lines
-----Original Message-----
From: Richard Purdie [mailto:[email protected]]
Sent: Monday, August 29, 2011 8:55 PM
To: Xu, Dongxiao
Cc: [email protected]
Subject: Re: [bitbake-devel] [PATCH 1/1]data_smart.py: make use of expand cache ingetVar()Quoted 20 lines · Dongxiao Xu
Currently if passing expand=True to
getVar()function, it will pass the handling togetVarFlag(), which doesn't get any benefit from the expand cache.Call the
expand()function separately ingetVar()to make use of the expand cache, which can decrease the parsing time by 40%. (from current 49s to 27s)Signed-off-by Dongxiao Xu <[email protected]>--- lib/bb/data_smart.py | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletions(-)lib/bb/data_smart.py index renamed from lib/bb/data_smart.py +0
-0301f9e3..d8ba24f 100644
lib/bb/data_smart.py +0
-0@@ -268,7 +268,12 @@ class DataSmart(MutableMapping): self.dict[var]["content"] = valueQuoted 12 lines
lib/bb/data_smart.py class DataSmart(MutableMapping): · line 277
Quoted 10 lines
lib/bb/data_smart.py +7
-1def getVar(self, var, expand=False, noweakdefault=False): - return self.getVarFlag(var, "content", expand, noweakdefault) + value = self.getVarFlag(var, "content", False, + noweakdefault) + + # Call expand() separately to make use of the expand cache + if expand and value: + return self.expand(value, var) + return valueWhy can't we have getVarFlag use the expand cache?
Quoted 2 lines
For expand cache, the format is a list: expand_cache[varname]. It doesn't contain information of flag values.
Good point.
Quoted 7 lines
Quoted 4 lines
I'm left wondering why we don't key that cache on the unexpanded value rather than a specific variable name (or name+flag).
I'll take this patch but its something we should consider...
Thanks, I will step into this point further and see if (name+flag) type of cache could benefit us more.
Sorry I must misunderstand your point just now...
For caching of unexpanded value, there are too many/uncertainty of types, like:
${TUNE_ARCH}
${@bb.parse.BBHandler.vars_from_file(bb.data.getVar('FILE',d),d)[0] or 'defaultpkgname'}
${@bb.utils.contains("TUNE_FEATURES", "m32", "${X86ARCH32}", "" ,d)}${@bb.utils.contains("TUNE_FEATURES", "mx32", "${X86ARCH64}", "" ,d)}${@bb.utils.contains("TUNE_FEATURES", "m64", "${X86ARCH64}", "" ,d)}
For simple variables, it is easy to cache, however for the expressions, it may not suitable to cache.
Besides, I also tried the (name+flag) types of cache, but it doesn't bring benefit (from 27s to 28s). Therefore it is suppose that most of the calling should be getVar(). Those specifically getting flag value should be not much.
Thanks,
Dongxiao
Quoted 10 lines
Thanks,
DongxiaoQuoted 6 lines
Cheers,
Richard
Re: [PATCH 1/1] data_smart.py: make use of expand cache in getVar()
Quoted 27 lines · Dongxiao
Quoted 17 lines
-----Original Message-----
From: Xu, Dongxiao
Sent: Monday, August 29, 2011 9:16 PM
To: Richard Purdie
Cc: [email protected]
Subject: RE: [bitbake-devel] [PATCH 1/1]data_smart.py: make use of expand cache ingetVar()Quoted 9 lines
-----Original Message-----
From: Richard Purdie [mailto:[email protected]]
Sent: Monday, August 29, 2011 9:13 PM
To: Xu, Dongxiao
Cc: [email protected]
Subject: RE: [bitbake-devel] [PATCH 1/1]data_smart.py: make use of expand cache ingetVar()Quoted 7 lines
Quoted 4 lines
I'm left wondering why we don't key that cache on the unexpanded value rather than a specific variable name (or name+flag).
I'll take this patch but its something we should consider...
Thanks, I will step into this point further and see if (name+flag) type of cache could benefit us more.
Sorry I must misunderstand your point just now...
Quoted 7 lines
For caching of unexpanded value, there are too many/uncertainty of types, like:
${TUNE_ARCH}
${@bb.parse.BBHandler.vars_from_file(bb.data.getVar('FILE',d),d)[0] or 'defaultpkgname'}
${@bb.utils.contains("TUNE_FEATURES", "m32", "${X86ARCH32}", "" ,d)}${@bb.utils.contains("TUNE_FEATURES", "mx32", "${X86ARCH64}", "" ,d)}${@bb.utils.contains("TUNE_FEATURES", "m64", "${X86ARCH64}", "" ,d)}For simple variables, it is easy to cache, however for the expressions, it may not suitable to cache.
It shouldn't really matter what the expression is, we should just be able to cache the value it evaluates to. We invalidate the cache upon any write to the data store that can potentially change a variables contents.
This shouldn't really be any different to the current behaviour of the code (where caching doesn't cause a problem).
Cheers,
Richard