Skip to contentSkip to conversation

bitbake: parsing time improvement

GIT PULL In review Dongxiao Xu 2 participants 8 messages Aug 29, 2011 – Aug 30, 2011

lore
Share & tools
lore
#0 lore raw

Hi Richard,

This pull request is to improve parsing time by getting use of the expand cache, please help to review and pull.

Thanks,
Dongxiao

The following changes since commit 0423587db09f6f28cf9d801f5657a84157f42dbe:

hob: disable some menu entries whilst build is in progress (2011-08-24 19:32:42 -0700)

are available in the git repository at:
git://git.pokylinux.org/poky-contrib dxu4/parse
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=dxu4/parse

Dongxiao Xu (1):
data_smart.py: make use of expand cache in getVar()

 lib/bb/data_smart.py |    7 ++++++-
 1 files changed, 6 insertions(+), 1 deletions(-)
#1 lore raw

PATCH 1/1 data_smart.py: make use of expand cache in getVar()

Currently if passing expand=True to getVar() function, it will pass the handling to getVarFlag(), which doesn't get any benefit from the expand cache.

Call the expand() function separately in getVar() to make use of the expand cache, which can decrease the parsing time by 40%. (from current 49s to 27s)

---
 lib/bb/data_smart.py |    7 ++++++-
 1 files changed, 6 insertions(+), 1 deletions(-)
lib/bb/data_smart.py +6 -1
index 301f9e3..d8ba24f 100644
@@ -268,7 +268,12 @@ class DataSmart(MutableMapping):
         self.dict[var]["content"] = value
 
     def getVar(self, var, expand=False, noweakdefault=False):
-        return self.getVarFlag(var, "content", expand, noweakdefault)
+        value = self.getVarFlag(var, "content", False, noweakdefault)
+
+        # Call expand() separately to make use of the expand cache
+        if expand and value:
+            return self.expand(value, var)
+        return value
 
     def renameVar(self, key, newkey):
         """
Signature · 1 line
1.7.1
#2 lore raw

Re: [PATCH 1/1] data_smart.py: make use of expand cache in getVar()

Quoted 19 lines · Dongxiao Xu

Currently if passing expand=True to getVar() function, it will pass the handling to getVarFlag(), which doesn't get any benefit from the expand cache.

Call the expand() function separately in getVar() to make use of the expand cache, which can decrease the parsing time by 40%. (from current 49s to 27s)

---
 lib/bb/data_smart.py |    7 ++++++-
 1 files changed, 6 insertions(+), 1 deletions(-)
lib/bb/data_smart.py +0 -0
index 301f9e3..d8ba24f 100644
@@ -268,7 +268,12 @@ class DataSmart(MutableMapping):
         self.dict[var]["content"] = value

lib/bb/data_smart.py class DataSmart(MutableMapping): · line 277

Quoted 10 lines
lib/bb/data_smart.py +6 -1
 
     def getVar(self, var, expand=False, noweakdefault=False):
-        return self.getVarFlag(var, "content", expand, noweakdefault)
+        value = self.getVarFlag(var, "content", False, noweakdefault)
+
+        # Call expand() separately to make use of the expand cache
+        if expand and value:
+            return self.expand(value, var)
+        return value
 

Why can't we have getVarFlag use the expand cache?

Cheers,

Richard

#3 lore raw

Re: [PATCH 1/1] data_smart.py: make use of expand cache in getVar()

Hi Richard,

Quoted 29 lines

-----Original Message-----
From: Richard Purdie [mailto:[email protected]]
Sent: Monday, August 29, 2011 8:55 PM
To: Xu, Dongxiao
Cc: [email protected]
Subject: Re: [bitbake-devel] [PATCH 1/1] data_smart.py: make use of expand cache in getVar()

Quoted 20 lines · Dongxiao Xu

Currently if passing expand=True to getVar() function, it will pass the handling to getVarFlag(), which doesn't get any benefit from the expand cache.

Call the expand() function separately in getVar() to make use of the expand cache, which can decrease the parsing time by 40%. (from current 49s to 27s)

---
 lib/bb/data_smart.py |    7 ++++++-
 1 files changed, 6 insertions(+), 1 deletions(-)
lib/bb/data_smart.py index renamed from lib/bb/data_smart.py +0 -0

301f9e3..d8ba24f 100644

lib/bb/data_smart.py +0 -0
@@ -268,7 +268,12 @@ class DataSmart(MutableMapping):
         self.dict[var]["content"] = value
 
Quoted 11 lines

lib/bb/data_smart.py class DataSmart(MutableMapping): · line 277

Quoted 9 lines
lib/bb/data_smart.py +6 -1
     def getVar(self, var, expand=False, noweakdefault=False):
-        return self.getVarFlag(var, "content", expand, noweakdefault)
+        value = self.getVarFlag(var, "content", False, noweakdefault)
+
+        # Call expand() separately to make use of the expand cache
+        if expand and value:
+            return self.expand(value, var)
+        return value
 

Why can't we have getVarFlag use the expand cache?

For expand cache, the format is a list: expand_cache[varname]. It doesn't contain information of flag values.

Thanks,
Dongxiao

Quoted 4 lines

Cheers,

Richard

#4 lore raw

Re: [PATCH 1/1] data_smart.py: make use of expand cache in getVar()

Quoted 43 lines · Dongxiao

Hi Richard,

Quoted 29 lines

-----Original Message-----
From: Richard Purdie [mailto:[email protected]]
Sent: Monday, August 29, 2011 8:55 PM
To: Xu, Dongxiao
Cc: [email protected]
Subject: Re: [bitbake-devel] [PATCH 1/1] data_smart.py: make use of expand cache in getVar()

Quoted 20 lines · Dongxiao Xu

Currently if passing expand=True to getVar() function, it will pass the handling to getVarFlag(), which doesn't get any benefit from the expand cache.

Call the expand() function separately in getVar() to make use of the expand cache, which can decrease the parsing time by 40%. (from current 49s to 27s)

---
 lib/bb/data_smart.py |    7 ++++++-
 1 files changed, 6 insertions(+), 1 deletions(-)
lib/bb/data_smart.py index renamed from lib/bb/data_smart.py +0 -0

301f9e3..d8ba24f 100644

lib/bb/data_smart.py +0 -0
@@ -268,7 +268,12 @@ class DataSmart(MutableMapping):
         self.dict[var]["content"] = value
 
Quoted 11 lines

lib/bb/data_smart.py class DataSmart(MutableMapping): · line 277

Quoted 9 lines
lib/bb/data_smart.py +6 -1
     def getVar(self, var, expand=False, noweakdefault=False):
-        return self.getVarFlag(var, "content", expand, noweakdefault)
+        value = self.getVarFlag(var, "content", False, noweakdefault)
+
+        # Call expand() separately to make use of the expand cache
+        if expand and value:
+            return self.expand(value, var)
+        return value
 

Why can't we have getVarFlag use the expand cache?

Quoted 2 lines

For expand cache, the format is a list: expand_cache[varname]. It doesn't contain information of flag values.

Good point.

I'm left wondering why we don't key that cache on the unexpanded value rather than a specific variable name (or name+flag).

I'll take this patch but its something we should consider...

Cheers,

Richard

#5 lore raw

Re: [PATCH 1/1] data_smart.py: make use of expand cache in getVar()

Quoted 56 lines

-----Original Message-----
From: Richard Purdie [mailto:[email protected]]
Sent: Monday, August 29, 2011 9:13 PM
To: Xu, Dongxiao
Cc: [email protected]
Subject: RE: [bitbake-devel] [PATCH 1/1] data_smart.py: make use of expand cache in getVar()

Quoted 44 lines · Dongxiao

Hi Richard,

Quoted 29 lines

-----Original Message-----
From: Richard Purdie [mailto:[email protected]]
Sent: Monday, August 29, 2011 8:55 PM
To: Xu, Dongxiao
Cc: [email protected]
Subject: Re: [bitbake-devel] [PATCH 1/1] data_smart.py: make use of expand cache in getVar()

Quoted 20 lines · Dongxiao Xu

Currently if passing expand=True to getVar() function, it will pass the handling to getVarFlag(), which doesn't get any benefit from the expand cache.

Call the expand() function separately in getVar() to make use of the expand cache, which can decrease the parsing time by 40%. (from current 49s to 27s)

---
 lib/bb/data_smart.py |    7 ++++++-
 1 files changed, 6 insertions(+), 1 deletions(-)
lib/bb/data_smart.py index renamed from lib/bb/data_smart.py +0 -0

301f9e3..d8ba24f 100644

lib/bb/data_smart.py +0 -0
@@ -268,7 +268,12 @@ class DataSmart(MutableMapping):
         self.dict[var]["content"] = value
 
Quoted 12 lines

lib/bb/data_smart.py class DataSmart(MutableMapping): · line 277

Quoted 10 lines
lib/bb/data_smart.py +7 -1
     def getVar(self, var, expand=False, noweakdefault=False):
-        return self.getVarFlag(var, "content", expand, noweakdefault)
+        value = self.getVarFlag(var, "content", False,
+ noweakdefault)
+
+        # Call expand() separately to make use of the expand cache
+        if expand and value:
+            return self.expand(value, var)
+        return value
 

Why can't we have getVarFlag use the expand cache?

Quoted 2 lines

For expand cache, the format is a list: expand_cache[varname]. It doesn't contain information of flag values.

Quoted 6 lines

Good point.

I'm left wondering why we don't key that cache on the unexpanded value rather than a specific variable name (or name+flag).

I'll take this patch but its something we should consider...

Thanks, I will step into this point further and see if (name+flag) type of cache could benefit us more.

Thanks,
Dongxiao

Quoted 6 lines

Cheers,

Richard

#6 lore raw

Re: [PATCH 1/1] data_smart.py: make use of expand cache in getVar()

In reply to Richard Purdie

Hi Richard,

Quoted 66 lines

-----Original Message-----
From: Xu, Dongxiao
Sent: Monday, August 29, 2011 9:16 PM
To: Richard Purdie
Cc: [email protected]
Subject: RE: [bitbake-devel] [PATCH 1/1] data_smart.py: make use of expand cache in getVar()

Quoted 58 lines

-----Original Message-----
From: Richard Purdie [mailto:[email protected]]
Sent: Monday, August 29, 2011 9:13 PM
To: Xu, Dongxiao
Cc: [email protected]
Subject: RE: [bitbake-devel] [PATCH 1/1] data_smart.py: make use of expand cache in getVar()

Quoted 44 lines · Dongxiao

Hi Richard,

Quoted 29 lines

-----Original Message-----
From: Richard Purdie [mailto:[email protected]]
Sent: Monday, August 29, 2011 8:55 PM
To: Xu, Dongxiao
Cc: [email protected]
Subject: Re: [bitbake-devel] [PATCH 1/1] data_smart.py: make use of expand cache in getVar()

Quoted 20 lines · Dongxiao Xu

Currently if passing expand=True to getVar() function, it will pass the handling to getVarFlag(), which doesn't get any benefit from the expand cache.

Call the expand() function separately in getVar() to make use of the expand cache, which can decrease the parsing time by 40%. (from current 49s to 27s)

---
 lib/bb/data_smart.py |    7 ++++++-
 1 files changed, 6 insertions(+), 1 deletions(-)
lib/bb/data_smart.py index renamed from lib/bb/data_smart.py +0 -0

301f9e3..d8ba24f 100644

lib/bb/data_smart.py +0 -0
@@ -268,7 +268,12 @@ class DataSmart(MutableMapping):
         self.dict[var]["content"] = value
 
Quoted 12 lines

lib/bb/data_smart.py class DataSmart(MutableMapping): · line 277

Quoted 10 lines
lib/bb/data_smart.py +7 -1
     def getVar(self, var, expand=False, noweakdefault=False):
-        return self.getVarFlag(var, "content", expand, noweakdefault)
+        value = self.getVarFlag(var, "content", False,
+ noweakdefault)
+
+        # Call expand() separately to make use of the expand cache
+        if expand and value:
+            return self.expand(value, var)
+        return value
 

Why can't we have getVarFlag use the expand cache?

Quoted 2 lines

For expand cache, the format is a list: expand_cache[varname]. It doesn't contain information of flag values.

Good point.

Quoted 7 lines
Quoted 4 lines

I'm left wondering why we don't key that cache on the unexpanded value rather than a specific variable name (or name+flag).

I'll take this patch but its something we should consider...

Thanks, I will step into this point further and see if (name+flag) type of cache could benefit us more.

Sorry I must misunderstand your point just now...

For caching of unexpanded value, there are too many/uncertainty of types, like:

${TUNE_ARCH}
${@bb.parse.BBHandler.vars_from_file(bb.data.getVar('FILE',d),d)[0] or 'defaultpkgname'}
${@bb.utils.contains("TUNE_FEATURES", "m32", "${X86ARCH32}", "" ,d)}${@bb.utils.contains("TUNE_FEATURES", "mx32", "${X86ARCH64}", "" ,d)}${@bb.utils.contains("TUNE_FEATURES", "m64", "${X86ARCH64}", "" ,d)}

For simple variables, it is easy to cache, however for the expressions, it may not suitable to cache.

Besides, I also tried the (name+flag) types of cache, but it doesn't bring benefit (from 27s to 28s). Therefore it is suppose that most of the calling should be getVar(). Those specifically getting flag value should be not much.

Thanks,
Dongxiao

Quoted 10 lines

Thanks,
Dongxiao

Quoted 6 lines

Cheers,

Richard

#7 lore raw

Re: [PATCH 1/1] data_smart.py: make use of expand cache in getVar()

Quoted 27 lines · Dongxiao
Quoted 17 lines

-----Original Message-----
From: Xu, Dongxiao
Sent: Monday, August 29, 2011 9:16 PM
To: Richard Purdie
Cc: [email protected]
Subject: RE: [bitbake-devel] [PATCH 1/1] data_smart.py: make use of expand cache in getVar()

Quoted 9 lines

-----Original Message-----
From: Richard Purdie [mailto:[email protected]]
Sent: Monday, August 29, 2011 9:13 PM
To: Xu, Dongxiao
Cc: [email protected]
Subject: RE: [bitbake-devel] [PATCH 1/1] data_smart.py: make use of expand cache in getVar()

Quoted 7 lines
Quoted 4 lines

I'm left wondering why we don't key that cache on the unexpanded value rather than a specific variable name (or name+flag).

I'll take this patch but its something we should consider...

Thanks, I will step into this point further and see if (name+flag) type of cache could benefit us more.

Sorry I must misunderstand your point just now...

Quoted 7 lines

For caching of unexpanded value, there are too many/uncertainty of types, like:

${TUNE_ARCH}
${@bb.parse.BBHandler.vars_from_file(bb.data.getVar('FILE',d),d)[0] or 'defaultpkgname'}
${@bb.utils.contains("TUNE_FEATURES", "m32", "${X86ARCH32}", "" ,d)}${@bb.utils.contains("TUNE_FEATURES", "mx32", "${X86ARCH64}", "" ,d)}${@bb.utils.contains("TUNE_FEATURES", "m64", "${X86ARCH64}", "" ,d)}

For simple variables, it is easy to cache, however for the expressions, it may not suitable to cache.

It shouldn't really matter what the expression is, we should just be able to cache the value it evaluates to. We invalidate the cache upon any write to the data store that can potentially change a variables contents.

This shouldn't really be any different to the current behaviour of the code (where caching doesn't cause a problem).

Cheers,

Richard