using d.expand() in a python function
I have a do_configure() python function which is essentially
python do_configure() {
import fileinput
with open("foo.txt", "w") as out:
for line in fileinput.input(files=d.getVar("FOO_INPUT").split()):
line = d.expand(line)
out.write(line)
}
So some of those input files can contain ${VARIABLE} and it gets replaced appropriately. But to my surprise, when I then change VARIABLE, the recipe doesn't get rebuilt, so it seems that bitbake doesn't automatically realize that the recipe (or task) depends on VARIABLE.
So digging a bit into the code, it seems that d.expand() calls expandWithRefs(), which returns a VariableParse object, but then only uses the .value property, throwing away .references. So I assume that what I really should be doing is call expandWithRefs and then somehow manually pass .references to... something? Or, there must be some other bitbake interface for doing this properly?
Rasmus
Quoted 17 lines · Rasmus Villemoes
I have a
do_configure()python function which is essentiallypython
do_configure(){
import fileinput
withopen("foo.txt", "w")as out:
for line in
fileinput.input(files=d.getVar("FOO_INPUT").split()):
line =d.expand(line)out.write(line)
}So some of those input files can contain ${VARIABLE} and it gets replaced appropriately. But to my surprise, when I then change VARIABLE,
the recipe doesn't get rebuilt, so it seems that bitbake doesn't automatically realize that the recipe (or task) depends on VARIABLE.
Quoted 7 lines
So digging a bit into the code, it seems that
d.expand()callsexpandWithRefs(), which returns a VariableParse object, but then only uses the .value property, throwing away .references. So I assume that what I really should be doing is call expandWithRefs and then somehow manually pass .references to... something? Or, there must be some other
bitbake interface for doing this properly?
Bitbake can only detect direct references to variables. The code above only runs at configure time, not at parse time so its not really surprising that bitbake can't know what its doing.
You'd need to read the data from this file at parse time for bitbake to stand some chance of seeing it, or manually set the variables it depends upon using the vardeps flag.
Something like:
do_configure[vardeps] = "${@my_depends_function(d)}"
where my_depends_function would return a list of variables it references.
Reading files like that at parse time doesn't work out well for performance.
Cheers,
Richard
Quoted 26 lines · Richard Purdie
Quoted 17 lines · Rasmus Villemoes
I have a
do_configure()python function which is essentiallypython
do_configure(){
import fileinput
withopen("foo.txt", "w")as out:
for line in
fileinput.input(files=d.getVar("FOO_INPUT").split()):
line =d.expand(line)out.write(line)
}So some of those input files can contain ${VARIABLE} and it gets replaced appropriately. But to my surprise, when I then change VARIABLE,
the recipe doesn't get rebuilt, so it seems that bitbake doesn't automatically realize that the recipe (or task) depends on VARIABLE.Quoted 7 lines
So digging a bit into the code, it seems that
d.expand()callsexpandWithRefs(), which returns a VariableParse object, but then only uses the .value property, throwing away .references. So I assume that what I really should be doing is call expandWithRefs and then somehow manually pass .references to... something? Or, there must be some other
bitbake interface for doing this properly?
Quoted 3 lines
Bitbake can only detect direct references to variables. The code above only runs at configure time, not at parse time so its not really surprising that bitbake can't know what its doing.
Well, I thought that as well at first. But there must be something done at run-time to feed back the variable dependencies, because if I add a completely dumb
python do_configure() {
import fileinput
+ d.getVar("VARIABLE") # silly
with open("foo.txt", "w") as out:
for line in fileinput.input(files=d.getVar("FOO_INPUT").split()):
line = d.expand(line)
out.write(line)
}
build the recipe (because now it has of course changed), and then change VARIABLE, the recipe does get re-built. Or does bitbake specifically look for d.getVar(<literal>) when it parses a python function?
Quoted 6 lines
You'd need to read the data from this file at parse time for bitbake to stand some chance of seeing it, or manually set the variables it depends upon using the vardeps flag.
Something like:
Quoted 7 lines
do_configure[vardeps] = "${@my_depends_function(d)}"
where
my_depends_functionwould return a list of variables it references.Reading files like that at parse time doesn't work out well for performance.
Yeah, so I'd very much like to avoid that, and the above observation suggests that I might be able to do the expansion manually, just looking for the \$\{...\} regexp and calling getVar to find the replacement. But that seems like something that should be solved already.
Thanks,
Rasmus
Quoted 19 lines · Rasmus Villemoes
Quoted 3 lines · Richard Purdie
Bitbake can only detect direct references to variables. The code above only runs at configure time, not at parse time so its not really surprising that bitbake can't know what its doing.
Well, I thought that as well at first. But there must be something done at run-time to feed back the variable dependencies, because if I add a completely dumb
python
do_configure(){
import fileinput
+d.getVar("VARIABLE")# silly
withopen("foo.txt", "w")as out:
for line in
fileinput.input(files=d.getVar("FOO_INPUT").split()):
line =d.expand(line)out.write(line)
}
Quoted 4 lines
build the recipe (because now it has of course changed), and then change VARIABLE, the recipe does get re-built. Or does bitbake specifically look for
d.getVar(<literal>)when it parses a python function?
Bitbake has ASTs for all its python fragments and can search those for getVar() calls. It will spot getVar("X") but can't know what is in d.expand(line).
Quoted 12 lines
Quoted 12 lines
You'd need to read the data from this file at parse time for bitbake to
stand some chance of seeing it, or manually set the variables it depends upon using the vardeps flag.Something like:
do_configure[vardeps] = "${@my_depends_function(d)}"
where
my_depends_functionwould return a list of variables it references.
Quoted 8 lines
Quoted 2 lines
Reading files like that at parse time doesn't work out well for performance.
Yeah, so I'd very much like to avoid that, and the above observation suggests that I might be able to do the expansion manually, just looking for the \$\{...\} regexp and calling getVar to find the replacement. But that seems like something that should be solved already.
Well, if you have the expansion, just put it in a variable the function references...
Cheers,
Richard
Quoted 20 lines · Richard Purdie
Quoted 19 lines · Rasmus Villemoes
Quoted 3 lines · Richard Purdie
Bitbake can only detect direct references to variables. The code above only runs at configure time, not at parse time so its not really surprising that bitbake can't know what its doing.
Well, I thought that as well at first. But there must be something done at run-time to feed back the variable dependencies, because if I add a completely dumb
python
do_configure(){
import fileinput
+d.getVar("VARIABLE")# silly
withopen("foo.txt", "w")as out:
for line in
fileinput.input(files=d.getVar("FOO_INPUT").split()):
line =d.expand(line)out.write(line)
}
Quoted 8 lines
Quoted 4 lines
build the recipe (because now it has of course changed), and then change VARIABLE, the recipe does get re-built. Or does bitbake specifically look for
d.getVar(<literal>)when it parses a python function?Bitbake has ASTs for all its python fragments and can search those for
getVar()calls. It will spotgetVar("X")but can't know what is ind.expand(line).
Yeah, I just tried changing it to
v1 = "VAR"
v2 = "IABLE"
d.getVar(v1 + v2)
and bitbake no longer knew about the dependency on VARIABLE :(
I can't be the first one to want to substitute bitbake variables in various source files, without necessarily knowing upfront which variables might appear.
So thinking out loud (and apologies if this is very silly): Would it be possible to implement some auto-deps class that a recipe playing the above game could inherit. Something like
AUTO_DEPS_TASKS = "configure"python() {
for t in AUTO_DEPS_TASKS:
f = (tmpdir/cachedir/somewhere)/autodeps/pn/t.inc if f does not exist:
make sure the recipe gets rebuilt return
include f
}
python write_autodeps(d, t):
logic to write a "do_t[vardeps] += ..." line to f, called at the end
of t (preferably automatically, but manually if need be)
python expand_autodeps(d, s):
helper to be used within an autodeps task, recording any variable used
in the expansion of s
Rasmus
Quoted 66 lines · Rasmus Villemoes
Quoted 28 lines · Richard Purdie
Quoted 22 lines · Rasmus Villemoes
Quoted 4 lines · Richard Purdie
Bitbake can only detect direct references to variables. The code
above only runs at configure time, not at parse time so its not really surprising that bitbake can't know what its doing.Well, I thought that as well at first. But there must be something
done at run-time to feed back the variable dependencies, because if I
add a completely dumbpython
do_configure(){
import fileinput
+d.getVar("VARIABLE")# silly
withopen("foo.txt", "w")as out:
for line in
fileinput.input(files=d.getVar("FOO_INPUT").split()):
line =d.expand(line)out.write(line)
}Quoted 4 lines
build the recipe (because now it has of course changed), and then change VARIABLE, the recipe does get re-built. Or does bitbake specifically look for
d.getVar(<literal>)when it parses a python function?Quoted 4 lines
Bitbake has ASTs for all its python fragments and can search those for
getVar()calls. It will spotgetVar("X")but can't know what is ind.expand(line).Yeah, I just tried changing it to
v1 = "VAR"
v2 = "IABLE"
d.getVar(v1 + v2)and bitbake no longer knew about the dependency on VARIABLE :(
I can't be the first one to want to substitute bitbake variables in various source files, without necessarily knowing upfront which variables might appear.
So thinking out loud (and apologies if this is very silly): Would it be
possible to implement some auto-deps class that a recipe playing the above game could inherit. Something like
AUTO_DEPS_TASKS= "configure"python(){
for t inAUTO_DEPS_TASKS:
f = (tmpdir/cachedir/somewhere)/autodeps/pn/t.inc if f does not exist:
make sure the recipe gets rebuilt return
include f
}python
write_autodeps(d, t):
logic to write a "do_t[vardeps] += ..." line to f, called at the
end
of t (preferably automatically, but manually if need be)
Quoted 4 lines
python
expand_autodeps(d, s):
helper to be used within an autodeps task, recording any variable
used
in the expansion of s
Why not do what I already suggested?
I suspect rather than a class you're looking for a helper function in lib/oe to call into? Just move my suggested function somewhere there?
Cheers,
Richard
Quoted 9 lines · Richard Purdie
Quoted 8 lines · Rasmus Villemoes
Quoted 2 lines · Richard Purdie
Quoted 1 line · Rasmus Villemoes
On 22/01/2020 15.35, Richard Purdie wrote:
I can't be the first one to want to substitute bitbake variables in various source files, without necessarily knowing upfront which variables might appear.
Quoted 5 lines
Quoted 3 lines
So thinking out loud (and apologies if this is very silly): Would it be
possible to implement some auto-deps classWhy not do what I already suggested?
Sorry, I'm probably a bit slow. Do you mean this part
Well, if you have the expansion, just put it in a variable the function references...
? That was in response to something that I since discovered wouldn't work [because d.getVar(line) doesn't magically feed back the deps info as my naive experiment led me to believe], so I must admit I didn't think too hard about it. But, I don't see how slurping in the file(s), doing the expansion, and putting the result in some variable that I then do a (dummy?) reference to would help teach bitbake which variables were used during the expansion.
Quoted 2 lines
I suspect rather than a class you're looking for a helper function in lib/oe to call into? Just move my suggested function somewhere there?
Well, yes, I'm certainly looking for or asking for a library function that will do what I want. But I'm not really seeing how to implement that. I do now see how the variable dependencies must be known at parse time.
Rasmus
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 21 lines · Rasmus Villemoes via Lists.Yoctoproject.Org
Quoted 10 lines · Richard Purdie
Quoted 9 lines · Rasmus Villemoes
Quoted 3 lines · Richard Purdie
Bitbake can only detect direct references to variables. The code above only runs at configure time, not at parse time so its not really surprising that bitbake can't know what its doing.
Well, I thought that as well at first. But there must be something done at run-time to feed back the variable dependencies, because if I add a completely dumb
Quoted 10 lines
Quoted 10 lines
python
do_configure(){
import fileinput
+d.getVar("VARIABLE")# silly
withopen("foo.txt", "w")as out:
for line in
fileinput.input(files=d.getVar("FOO_INPUT").split()):
line =d.expand(line)out.write(line)
}
Quoted 2 lines
So thinking out loud (and apologies if this is very silly): Would it be possible to implement some auto-deps class
Alternatively, as I will only need this for files I provide via file:// SRC_URIs, perhaps that fetcher could be taught a ";varexpand=1" parameter. Bitbake must be doing md5sum or similar on local files anyway (? how else are changes in those detected), so having varexpand=1 mean "grep all ${foo} references and add foo to do_unpack[vardeps]" during parsing shouldn't add that much.
And of course having bitbake automatically actually do the expansion as part of placing the file in WORKDIR means one can avoid that logic altogether in the recipe (so my do_configure could become a simple shell function that does "cat ${FOO_INPUT} > foo.txt").
Rasmus