Chasing fetcher issues in circles
Hi Bruce,
I've been chasing a fetcher issue around in circles, I have something concrete to report back now.
We need to have the fetcher code work for the kernel recipes without network access. The problem is that the kernel git recipe is complex, has multiple branches and supports a variety of configurations. The root of some of the problems is the chunk of anonymous python in kernel-yocto.bbclass.
The trouble is we have a race between the variable PV being expanded *anywhere* and that anonymous python running. The race is that other anonymous python may run before the recipes anonymous python and that other code can touch trigger an expansion of PV.
What happens if PV is expanded first?
Basically, the fetcher variables aren't setup and it falls back to the UNDEFINED machine, the revisions are then set based on the undefined machine rather that the true machine branch and confusion ensues. Worse again, the fetcher can cache the first set of values it calculates.
One symptom of this and how it triggers a network access is that SRCREV_machine can be set to "yocto/standard/base". I added:
http://git.pokylinux.org/cgit.cgi/poky/commit/?id=32dbc70e86c3b186658dcaa84a8c7cd76a50e069
a short while ago so at least this is something resolvable (the fetcher would do a "git ls-remote" on this) but it still means a network access and it still means the fetcher config is broken at some point and at risk of being cached.
So firstly, to fulfil our network access concerns when the kernel is used with an undefined machine, we need one more SRCREV to be added to poky-default-revisions.inc. I've done this in the following rfc commit:
That is a valid fix we need regardless but I think this race window is what has been causing various people (like Darren and Tom) a lot of pain and I really want to stop that *ever* happening again.
Even with the above change, it just stops us hitting the network, the race still happens and the data is inconsistent.
So the takeaway is that the anonymous python isn't working. As far as I can tell we can do one of two things:
a) If we really need some python code, there is a nasty way we can hack it in and ensure it runs before SRCPV is expanded. This is hacktastic but the patch is:
and I've confirmed this does actually work as intended.
b) The other option is we reduce the amount of anonymous python and use other means to set the require variables.
For example, the 2.6.34 bits could be turned into a separate class which overrides the default options and is only included by the 2.6.34 recipe(s). For the branch manipulations performed in code it does look a little backwards since it looks like KMACHINE is following the old style kernel still. Shouldn't we be detecting the old style and converting that to new in the 2.6.34 case rather than converting existing variables to the .37 style?
My worry here is that manipulating variables behind the user's back tends to confuse them so its fine for upgrading to a new version but doing it for our new default version is a little concerning.
Anyhow, there is some food for thought here. The patches above did turn out a lot nicer looking than I thought they would when I started writing the email!
Cheers,
Richard
Quoted 69 lines · Richard Purdie
Hi Bruce,
I've been chasing a fetcher issue around in circles, I have something concrete to report back now.
We need to have the fetcher code work for the kernel recipes without network access. The problem is that the kernel git recipe is complex, has multiple branches and supports a variety of configurations. The root of some of the problems is the chunk of anonymous python in kernel-yocto.bbclass.
The trouble is we have a race between the variable PV being expanded *anywhere* and that anonymous python running. The race is that other anonymous python may run before the recipes anonymous python and that other code can touch trigger an expansion of PV.
What happens if PV is expanded first?
Basically, the fetcher variables aren't setup and it falls back to the UNDEFINED machine, the revisions are then set based on the undefined machine rather that the true machine branch and confusion ensues. Worse again, the fetcher can cache the first set of values it calculates.
One symptom of this and how it triggers a network access is that
SRCREV_machinecan be set to "yocto/standard/base". I added:http://git.pokylinux.org/cgit.cgi/poky/commit/?id=32dbc70e86c3b186658dcaa84a8c7cd76a50e069
a short while ago so at least this is something resolvable (the fetcher would do a "git ls-remote" on this) but it still means a network access and it still means the fetcher config is broken at some point and at risk of being cached.
So firstly, to fulfil our network access concerns when the kernel is used with an undefined machine, we need one more SRCREV to be added to poky-default-revisions.inc. I've done this in the following rfc commit:
That is a valid fix we need regardless but I think this race window is what has been causing various people (like Darren and Tom) a lot of pain and I really want to stop that *ever* happening again.
Even with the above change, it just stops us hitting the network, the race still happens and the data is inconsistent.
So the takeaway is that the anonymous python isn't working. As far as I can tell we can do one of two things:
a) If we really need some python code, there is a nasty way we can hack it in and ensure it runs before SRCPV is expanded. This is hacktastic but the patch is:
and I've confirmed this does actually work as intended.
b) The other option is we reduce the amount of anonymous python and use other means to set the require variables.
For example, the 2.6.34 bits could be turned into a separate class which overrides the default options and is only included by the 2.6.34 recipe(s). For the branch manipulations performed in code it does look a little backwards since it looks like KMACHINE is following the old style kernel still. Shouldn't we be detecting the old style and converting that to new in the 2.6.34 case rather than converting existing variables to the .37 style?
Quoted 3 lines
My worry here is that manipulating variables behind the user's back tends to confuse them so its fine for upgrading to a new version but doing it for our new default version is a little concerning.
I already have a complete re-factored set of kernel recipes here, I was holding them back since I wasn't sure about introducing churn in the 1.0 end of cycle. Sounds like I should dust them off.
The anonymous python exists for one reason only:
- to keep the fetcher happy, in particular when you try and build a new board.
Now that the fetcher has changed, the complexity can go. I'll try and spin some re-worked classes shortly.
Cheers,
Bruce
Quoted 16 lines
Anyhow, there is some food for thought here. The patches above did turn out a lot nicer looking than I thought they would when I started writing the email!
Cheers,
Richard
_______________________________________________
poky mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/poky
Signature · 2 lines
"Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end"