Skip to contentSkip to conversation

RFC: SRC_URI_OVERIDES_PACKAGE_ARCH

DISCUSSION Richard Purdie 2 participants 3 messages Aug 11, 2007

lore
Share & tools
lore
#0 lore raw

Hi,

In trying to make SRCREV work efficiently I recently accidentally changed the behaviour of SRC_URI_OVERIDES_PACKAGE_ARCH.

The old behaviour is that if SRC_URI_OVERIDES_PACKAGE_ARCH is unset (the default) or set to anything other than 0 the auto PACKAGE_ARCH mechanism is enabled.

This code goes through SRC_URI translating each URL into a local path (where the fetched file would be located). It then checks if this local path matches any standard MACHINE specific path. Examples are things like:

PN/files/MACHINE/some_conf_file
PN-PV/MACHINE/some_conf_file
PN/MACHINE/some_conf_file

If its a machine specific path, the .bb file is marked as MACHINE specific with PACKAGE_ARCH=MACHINE_ARCH.

I'd assumed this was opt in and changed the default case, thereby breaking some recipes.

The problem is that translating SRC_URI to localpaths is computationally expensive. In the SRCREV case, it involves hitting the network which is very slow. If we use the localpath calls at all, we hit the database which will slow down parsing.

We have several options:
* restore the old behaviour, accept the parsing slowdown
* limit the parsing to file:// urls only since they're the only ones this applies to anyway. We avoid the network hits, we still hit the database so see some slowdown
* make it opt in, recipes which might use this feature have to set SRC_URI_OVERIDES_PACKAGE_ARCH=1.

The whole OE <-> bitbake fetcher interaction and the associated caching done in the fetcher needs some serious thought yet again. Ideally it needs a total rewrite but I'm not sure I can do it whilst maintaining backwards compatibility :-(.

For reference, I've been trying to postpone the parsing of SRC_URI until the latest possible moment (fetch/unpack tasks) to try and avoid the overhead.

I'll keep thinking but I'm open to opinions of the above options. How much do we value this automatic handling over parsing speed?

Cheers,

Richard

#1 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Richard Purdie schreef:

Quoted 7 lines

The problem is that translating SRC_URI to localpaths is computationally expensive. In the SRCREV case, it involves hitting the network which is very slow. If we use the localpath calls at all, we hit the database which will slow down parsing.

We have several options:
* restore the old behaviour, accept the parsing slowdown

If people want to know how much the slowdown is, add:

SRC_URI_OVERIDES_PACKAGE_ARCH = "1"

to your local.conf. On a dual opteron with U320 scsi disks it becomes unbearably slow.

I think checking only file:// paths is the best intermediate solution.

regards,

Koen

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFGvbV0MkyGM64RGpERAr3VAJ94xJwqdaWcUE81/2GNFgaxnHh+YQCfS109 kfMBSlqKAVtv4lk2S1SU86U=
=mnp2
-----END PGP SIGNATURE-----

#2 lore raw
Quoted 9 lines · Koen Kooi

Richard Purdie schreef:

Quoted 7 lines

The problem is that translating SRC_URI to localpaths is computationally expensive. In the SRCREV case, it involves hitting the network which is very slow. If we use the localpath calls at all, we hit the database which will slow down parsing.

We have several options:
* restore the old behaviour, accept the parsing slowdown

Quoted 7 lines

If people want to know how much the slowdown is, add:

SRC_URI_OVERIDES_PACKAGE_ARCH = "1"

to your local.conf. On a dual opteron with U320 scsi disks it becomes unbearably slow.

I think checking only file:// paths is the best intermediate solution.

With bitbake 1.8.7 and SRC_URI_OVERIDES_PACKAGE_ARCH just set in recipes that need it parsing OE.dev takes:

real    5m49.043s
user    4m35.649s
sys     0m11.057s

Whereas if we check file:// urls:

real    9m8.550s
user    5m55.542s
sys     0m45.847s

Its possible that machine was more loaded when I ran the second test but it was reproducibly taking 9 mins vs 5.5 mins with just that change. Its safe to say there is a 20%+ speed hit, probably worse.

For comparison bitbake 1.8.6 with SRC_URI_OVERIDES_PACKAGE_ARCH just set in recipes that need it:

real    6m56.088s
user    4m50.558s
sys     0m11.933s

Whereas if we check file:// urls:

real    6m45.358s
user    4m52.466s
sys     0m12.225s

So we have what looks like a speedup in 1.8.7 if we can sort the SRC_URI_OVERIDES_PACKAGE_ARCH problem...

I suspect I should throw away the sql based primary cache and go back to the ugly global variable as its tons faster...

Cheers,

Richard