RFC: SRC_URI_OVERIDES_PACKAGE_ARCH
Hi,
In trying to make SRCREV work efficiently I recently accidentally changed the behaviour of SRC_URI_OVERIDES_PACKAGE_ARCH.
The old behaviour is that if SRC_URI_OVERIDES_PACKAGE_ARCH is unset (the default) or set to anything other than 0 the auto PACKAGE_ARCH mechanism is enabled.
This code goes through SRC_URI translating each URL into a local path (where the fetched file would be located). It then checks if this local path matches any standard MACHINE specific path. Examples are things like:
PN/files/MACHINE/some_conf_file
PN-PV/MACHINE/some_conf_file
PN/MACHINE/some_conf_file
If its a machine specific path, the .bb file is marked as MACHINE specific with PACKAGE_ARCH=MACHINE_ARCH.
I'd assumed this was opt in and changed the default case, thereby breaking some recipes.
The problem is that translating SRC_URI to localpaths is computationally expensive. In the SRCREV case, it involves hitting the network which is very slow. If we use the localpath calls at all, we hit the database which will slow down parsing.
We have several options:
* restore the old behaviour, accept the parsing slowdown
* limit the parsing to file:// urls only since they're the only ones this applies to anyway. We avoid the network hits, we still hit the database so see some slowdown
* make it opt in, recipes which might use this feature have to set SRC_URI_OVERIDES_PACKAGE_ARCH=1.
The whole OE <-> bitbake fetcher interaction and the associated caching done in the fetcher needs some serious thought yet again. Ideally it needs a total rewrite but I'm not sure I can do it whilst maintaining backwards compatibility :-(.
For reference, I've been trying to postpone the parsing of SRC_URI until the latest possible moment (fetch/unpack tasks) to try and avoid the overhead.
I'll keep thinking but I'm open to opinions of the above options. How much do we value this automatic handling over parsing speed?
Cheers,
Richard
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Richard Purdie schreef:
Quoted 7 lines
The problem is that translating
SRC_URIto localpaths is computationally expensive. In the SRCREV case, it involves hitting the network which is very slow. If we use the localpath calls at all, we hit the database which will slow down parsing.We have several options:
* restore the old behaviour, accept the parsing slowdown
If people want to know how much the slowdown is, add:
SRC_URI_OVERIDES_PACKAGE_ARCH = "1"
to your local.conf. On a dual opteron with U320 scsi disks it becomes unbearably slow.
I think checking only file:// paths is the best intermediate solution.
regards,
Koen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)
iD8DBQFGvbV0MkyGM64RGpERAr3VAJ94xJwqdaWcUE81/2GNFgaxnHh+YQCfS109 kfMBSlqKAVtv4lk2S1SU86U=
=mnp2
-----END PGP SIGNATURE-----
Quoted 9 lines · Koen Kooi
Richard Purdie schreef:
Quoted 7 lines
The problem is that translating
SRC_URIto localpaths is computationally expensive. In the SRCREV case, it involves hitting the network which is very slow. If we use the localpath calls at all, we hit the database which will slow down parsing.We have several options:
* restore the old behaviour, accept the parsing slowdown
Quoted 7 lines
If people want to know how much the slowdown is, add:
SRC_URI_OVERIDES_PACKAGE_ARCH= "1"to your local.conf. On a dual opteron with U320 scsi disks it becomes unbearably slow.
I think checking only file:// paths is the best intermediate solution.
With bitbake 1.8.7 and SRC_URI_OVERIDES_PACKAGE_ARCH just set in recipes that need it parsing OE.dev takes:
real 5m49.043s user 4m35.649s sys 0m11.057s
Whereas if we check file:// urls:
real 9m8.550s user 5m55.542s sys 0m45.847s
Its possible that machine was more loaded when I ran the second test but it was reproducibly taking 9 mins vs 5.5 mins with just that change. Its safe to say there is a 20%+ speed hit, probably worse.
For comparison bitbake 1.8.6 with SRC_URI_OVERIDES_PACKAGE_ARCH just set in recipes that need it:
real 6m56.088s user 4m50.558s sys 0m11.933s
Whereas if we check file:// urls:
real 6m45.358s user 4m52.466s sys 0m12.225s
So we have what looks like a speedup in 1.8.7 if we can sort the SRC_URI_OVERIDES_PACKAGE_ARCH problem...
I suspect I should throw away the sql based primary cache and go back to the ugly global variable as its tons faster...
Cheers,
Richard