The future of "svnnow"
Hi,
As people know the ability to build the latest head revisions of svn (for example) repositories in OE with bitbake has been limited and less than ideal.
Various patches have floated around with good points and bad points and there was some discussion on the bitbake-dev list about them. The multithreading in bitbake 1.8 posed some major challenges.
To solve the problems I have made the following changes:
Addition of an sqlite based persistent datastore to store "information" in. This is divided into "domains" and a domain is easy to create/destroy or clean. Currently we have three domains:
1. BB_URLDATA - this replaced the ugly global variable in the fetcher and means we don't recompute the data from SRC_URI each time we need to use it. This is wiped every bitbake run.
2. BB_URI_HEADREVS - a cache of the head revisions of SCMs
3. BB_URI_LOCALCOUNT - this will be used by SCMs that require a local counter since their revisions don't sort (e.g. git).
Addition of methods the fetchers. We needed three new ones:
1. _revision_key - some key we can uniquely identify a given scm with (the url alone might not be enough)
2. _latest_revision - returns the head revision of a given scm
3. _sortable_revision - returns the sortable version of that revision
Core changes to the fetcher:
1. A get_srcrev() function to return the SRCREV string. This works for since SCM SRC_URIs but needs finishing to support multi SCM SRC_URIs.
2. latest_revision() method as _latest_revision above but uses the cache.
3. sortable_revision() method which will implement BB_URI_LOCALCOUNT for scms like git but this isn't implemented yet
So what does the result look like?
==================================
In OE, you'd set
PV = "x.y.z+svnr${SRCREV}"
and SRCREV is declared in bitbake.conf as:
SRCREV = "${@bb.fetch.get_srcrev()}"
and this is all you do. Now the PV changes when the svn head changes and you have cutting edge packages. The path to the tarball in DL_DIR changes based on the revision too.
How often it checks for changes in the head depends on when BB_URI_HEADREVS is cleared. Currently bitbake does this at each startup but this could easily be controlled by a variable so each user can set their preference.
For distros the plan is that they can lock this down with:
SRCREV_pn-somepackage = "90"
or maybe this as well (if the x.y.z needs to be changed):
PV_pn-somepackage = "x.y.z+svnr90"
I've just noticed this doesn't work yet, that needs fixing. Minor issue ;-).
The problem is dealing with SRCDATE. Since we often set SRCDATE = "something", SRCDATE is set for every package. The only way bitbake can spot the use of SRCDATE is to peek at the unexpanded PV variable and see if it contains the string "DATE". If it does, it uses SRCDATE, if it doesn't is assumes SRCREV is in use. I don't like this behaviour but I can't see the alternative. In due course I'm hoping we stop supporting SRCDATE for svn since SRCREV should be much more powerful.
What still needs to be done (patches welcome)
=============================================
1. Fix revision lockdown SRCREV_pn-somepackage = "90" etc. Thats a problem in localpath() in svn.py.
2. Handling of multiple SCMs in the SRC_URI. The plan here is simple, any package doing this has to set SRCREV_FORMAT specifying the format of the SRCREV string. get_srcrev() can collect the SCM revisions, fill out the string and return it. The format and inner workings of this are open for discussion and to be determined.
3. Finish sortable_revision(). If self._sortable_revision() isn't present, it should read a counter, revision pair from BB_URI_LOCALCOUNT. If the revision is different to _latest_revision, increase the counter and store + return it.
4. Fix the git fetcher to make git "now" work. These "local revision counter" packages will not be shareable between different builds but we always knew that would be the case with git and this is as close as we can get.
5. Examine the other fetchers and see if we can add some kind of "now" support to them too.
6. Add policy control variables to control the wiping of BB_URI_HEADREVS
Migration Path
==============
SRCREV cannot be used by packages in OE.dev until we bump the version requirement to bitbake 1.8.8 assuming that contains these changes. We need to decide how much of the above we want to implement before 1.8.8 and test svn heavily before that release (with people testing SRCREV locally if possible). Once the minimum version is 1.8.8 we can look at converting the svn packages in OE.dev.
Final Note
==========
I realise this comes across more as "how things are" rather than a discussion about how we should do them. This is partly because things were discussed on the bitbake list in the past, partly because I don't see any other way some of this can work and partly because presenting a solution is perhaps the best way to generate discussion. If anyone can see better ways to do any of this, please do speak up! I don't think all of this is perfect, there are bits I strongly dislike. The main thing in its favour is I think we can make it work!
Comments and discussion are very welcome. At the very least we need to agree:
a) this is the right approach
b) how much of the outstanding task list we need to complete before 1.8.8
c) how to migrate to these changes in OE.dev
and due to the invasive nature of some of the changes, testing, testing, testing! :)
Cheers,
Richard
Awesome work -- I'm impressed!
This is yet another example of how good the BitBake core architecture is.
Quoted 4 lines
Comments and discussion are very welcome. At the very least we need to agree:
a) this is the right approach
+1 from me.
Quoted 1 line
b) how much of the outstanding task list we need to complete before 1.8.8
1,2,4
Quoted 1 line
c) how to migrate to these changes in OE.dev
Who has actually used SRCDATE = now? I know I did.
Quoted 2 lines
and due to the invasive nature of some of the changes, testing, testing, testing!
Sure thing.
Thanks a lot Richard!
Regards,
:M:
Signature · 1 line
Michael 'Mickey' Lauer | IT-Freelancer | http://www.vanille-media.de
Quoted 3 lines · Dr. Michael Lauer
Quoted 1 line
c) how to migrate to these changes in OE.dev
Who has actually used SRCDATE = now? I know I did.
Ok, upon rereading the challenge seems to change the pinpointed SRCDATEs in sane-srcdates.inc while making sure that the upgrade path still works (i.e. lexical comparison), right?
Regards,
:M:
Signature · 1 line
Michael 'Mickey' Lauer | IT-Freelancer | http://www.vanille-media.de
Quoted 8 lines · Dr. Michael Lauer
Quoted 3 lines · Dr. Michael Lauer
Quoted 1 line
c) how to migrate to these changes in OE.dev
Who has actually used SRCDATE = now? I know I did.
Ok, upon rereading the challenge seems to change the pinpointed SRCDATEs in sane-srcdates.inc while making sure that the upgrade path still works (i.e. lexical comparison), right?
If we use
PV = "x.y.z+svnrXYZ"
that should sort above
PV = "x.y.z+svnYYYMMDD"
which is as good as we can probably hope for?
Richard
Quoted 1 line · Dr. Michael Lauer
Awesome work -- I'm impressed!
Thanks :)
Quoted 2 lines
This is yet another example of how good the BitBake core architecture is.
Yes!
Quoted 3 lines
Quoted 1 line
b) how much of the outstanding task list we need to complete before 1.8.8
1,2,4
Ok, thats roughly what I was thinking too. 4 needs 3 too so thats 1-4.
Quoted 3 lines
Quoted 1 line
c) how to migrate to these changes in OE.dev
Who has actually used SRCDATE = now? I know I did.
I also mean issues like who's going to do it, when and how? All at once, distro by distro, adhoc? and is the aim to wipe out SRCDATE entirely for svn?
Richard
I should also add, the changes so far are in bitbake trunk and in the bitbake 1.8 svn branch. I've added the definition of SRCREV to bitbake.conf in OE.dev so anyone wishing to experiment with this can do so just by changing the PV as discussed in a .bb file...
Please don't check in such changes yet though.
I guess we could have a "I-use-bitbake-1.8-trunk.conf" that does
PV_pn-somepackage1 = "x.y.z+svnr${SRCREV}"
PV_pn-somepackage2 = "x.y.z+svnr${SRCREV}"
but thats probably insane and a sign I should sleep :)
Richard
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Richard Purdie schreef:
Quoted 23 lines
What still needs to be done (patches welcome)
=============================================1. Fix revision lockdown SRCREV_pn-somepackage = "90" etc. Thats a problem in
localpath()insvn.py.2. Handling of multiple SCMs in the
SRC_URI. The plan here is simple, any package doing this has to setSRCREV_FORMATspecifying the format of the SRCREV string.get_srcrev()can collect the SCM revisions, fill out the string and return it. The format and inner workings of this are open for discussion and to be determined.3. Finish
sortable_revision(). Ifself._sortable_revision()isn't present, it should read a counter, revision pair fromBB_URI_LOCALCOUNT. If the revision is different to_latest_revision, increase the counter and store + return it.4. Fix the git fetcher to make git "now" work. These "local revision counter" packages will not be shareable between different builds but we always knew that would be the case with git and this is as close as we can get.
Quoted 4 lines
5. Examine the other fetchers and see if we can add some kind of "now" support to them too.
6. Add policy control variables to control the wiping of
BB_URI_HEADREVS
7. make a minrev and max rev for patches and try to convert dates to revisions for
existing users of ;mindate=20070707;maxdate=20080808 in SRC_URI
regards,
Koen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)
iD8DBQFGrumgMkyGM64RGpERAvY+AJwIksReey1LVHa6NpFRvcdFvldPOgCfbeLO jvrr6XXcdJFxRjH+GFUsQlU=
=9FHq
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Richard Purdie schreef:
Quoted 5 lines
Hi,
As people know the ability to build the latest head revisions of svn (for example) repositories in OE with bitbake has been limited and less than ideal.
What happens if I have no internet connection during parsing and do have a huge sourcemirror inside my firewall?
regards,
Koen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)
iD8DBQFGrw4zMkyGM64RGpERAgpoAJ4lBWJHarL+qBe8dEysUi69JSz6PwCfddxA 5VRd/dXNGbXlSjU9khkSx88=
=v5vh
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Koen Kooi schreef:
Quoted 9 lines
Richard Purdie schreef:
Quoted 5 lines
Hi,
As people know the ability to build the latest head revisions of svn (for example) repositories in OE with bitbake has been limited and less than ideal.
What happens if I have no internet connection during parsing and do have a huge sourcemirror inside my firewall?
I did:
packages$ for i in */*.bb ; do sed -i -e s:'svn${SRCDATE}':'svnr${SRCREV}':g $i ; done
One of the results:
NOTE: <class 'bb.MalformedUrl'>: while evaluating:
${@bb.fetch.get_srcrev(d)}
NOTE: <class 'bb.MalformedUrl'>: while evaluating:
0.0+svnr${SRCREV}
NOTE: <class 'bb.MalformedUrl'>: while evaluating:
${PN}-${EXTENDPE}${PV}-${PR}
NOTE: <class 'bb.MalformedUrl'>: while evaluating:
${TMPDIR}/work/${MULTIMACH_ARCH}${TARGET_VENDOR}-${TARGET_OS}/${PF}
NOTE: <class 'bb.MalformedUrl'>: while evaluating:
${WORKDIR}/trunk
NOTE: <class 'bb.MalformedUrl'>: while evaluating:
${S}
ERROR: while parsing /home/koen/OE/monotone/org.openembedded.dev/packages/openocd/openocd_svn.bb
The parser seems to dislike:
* recipes with SRCREV that weren't fetched yet (openocd in my case)
* recipes with SRCREV that do ;proto=http (libmatchbox)
Is there anything we can do to fix these cases?
regards,
Koen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)
iD8DBQFGrxT2MkyGM64RGpERAqi0AKCkpdfJuvagyisqUKm3eZm1gS5N2ACfbZN9 iB83sV9Yf0wPoHO9kgX6Lkc=
=QKcE
-----END PGP SIGNATURE-----
Quoted 4 lines · Koen Kooi
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Richard Purdie schreef:
Quoted 8 lines
Quoted 5 lines
Hi,
As people know the ability to build the latest head revisions of svn (for example) repositories in OE with bitbake has been limited and less than ideal.
What happens if I have no internet connection during parsing and do have a huge sourcemirror inside my firewall?
Two possible solutions:
1. You have locked down SRCREVs
2. You have a valid head revisions cache from the last time you had network access and you set the bitbake cache policy not to wipe that cache.
I can't see any other ways around that and its probably better than the current situation.
Cheers,
Richard
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Richard Purdie schreef:
Quoted 10 lines · Koen Kooi
Quoted 9 lines
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Richard Purdie schreef:
Quoted 5 lines
Hi,
As people know the ability to build the latest head revisions of svn (for example) repositories in OE with bitbake has been limited and less than ideal.
Quoted 6 lines
Quoted 2 lines
What happens if I have no internet connection during parsing and do have a huge sourcemirror inside my firewall?
Two possible solutions:
1. You have locked down SRCREVs
bitbake -s | grep svnr | awk '{print "SRCREV_pn-" $1 "=" $2}' | \
sed -e s:\+:\=:g -e s:svnr::g| awk -F "=" '{print $1 "=\"" $3}' | \
awk -F "-r" '{print $1"\""}'
will spit out a list you can use as a .inc file for your favourite distro.
Quoted 3 lines
2. You have a valid head revisions cache from the last time you had network access and you set the bitbake cache policy not to wipe that cache.
Is there a nice way to distribute such a cache, for e.g. evil firewalls?
Quoted 2 lines
I can't see any other ways around that and its probably better than the current situation.
It is indeed better, we just need to recreate all the tweaks we have for the current situation :)
regards,
Koen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)
iD8DBQFGrytsMkyGM64RGpERAoZ7AKCC+2Pue42dUXd6bQ+uhP2oHu6iRQCgjRgo oJFzPUBbj9v4y/3y5YJ2cC0=
=064g
-----END PGP SIGNATURE-----
Quoted 5 lines · Koen Kooi
Quoted 3 lines
2. You have a valid head revisions cache from the last time you had network access and you set the bitbake cache policy not to wipe that cache.
Is there a nice way to distribute such a cache, for e.g. evil firewalls?
Its currently in a very simple SQL database table...
sqlite3 tmp/cache/MACHINE/bb_persist_cache.sqlite3 .dump BB_URI_HEADREVS
to see the info...
Cheers,
Richard
Just an update on the current status:
Quoted 6 lines · Koen Kooi
Richard Purdie schreef:
Quoted 5 lines
What still needs to be done (patches welcome)
=============================================1. Fix revision lockdown SRCREV_pn-somepackage = "90" etc. Thats a problem in
localpath()insvn.py.
Done.
Quoted 5 lines
Quoted 5 lines
2. Handling of multiple SCMs in the
SRC_URI. The plan here is simple, any package doing this has to setSRCREV_FORMATspecifying the format of the SRCREV string.get_srcrev()can collect the SCM revisions, fill out the string and return it. The format and inner workings of this are open for discussion and to be determined.
Not Done. Does anyone have any thoughts on this?
Quoted 4 lines
Quoted 4 lines
3. Finish
sortable_revision(). Ifself._sortable_revision()isn't present, it should read a counter, revision pair fromBB_URI_LOCALCOUNT. If the revision is different to_latest_revision, increase the counter and store + return it.
Done.
Quoted 4 lines
Quoted 4 lines
4. Fix the git fetcher to make git "now" work. These "local revision counter" packages will not be shareable between different builds but we always knew that would be the case with git and this is as close as we can get.
Done.
Quoted 2 lines
Quoted 2 lines
5. Examine the other fetchers and see if we can add some kind of "now" support to them too.
I'm ignoring perforce and svk. It doesn't make sense for ssh/wget/local urls. I doubt we can get any kind of coherent state out of cvs and to be honest I don't care about it. So I guess thats Done.
Quoted 1 line
Quoted 1 line
6. Add policy control variables to control the wiping of
BB_URI_HEADREVS
Not Done.
Quoted 1 line
7. make a minrev and max rev for patches
Done.
Quoted 2 lines
and try to convert dates to revisions for
existing users of ;mindate=20070707;maxdate=20080808 inSRC_URI
Not Done. SEP field invoked here ;-).
Cheers,
Richard
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Richard Purdie schreef:
Quoted 5 lines
2. Handling of multiple SCMs in the
SRC_URI. The plan here is simple, any package doing this has to setSRCREV_FORMATspecifying the format of the SRCREV string.get_srcrev()can collect the SCM revisions, fill out the string and return it. The format and inner workings of this are open for discussion and to be determined.
What about tagging URIs with ;userev=true:
SRC_URI = "svn://foo/bar;userev=true \
svn://foo/baz;userev=false \
svn://snizzle/foo;userev=false"Now bitbake know that it should only use the revision of 'bar' and ideally only rebuild if that changes.
Or what about naming URIs:
SRC_URI = "svn://foo/bar;name=main \
svn://foo/baz;name=baz-module \
svn://snizzle/foo;name=snizzle-foo-support"SRCREV_FORMAT = "main"
regards,
Koen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)
iD8DBQFGtCgwMkyGM64RGpERAv0AAKC7Ghm1j1VNATajHCBGqXHgXWJIYwCfZWiF 8MfFdeVLP5yttEJXUmwAgF8=
=bXR1
-----END PGP SIGNATURE-----
Richard,
Quoted 5 lines · Richard Purdie
As people know the ability to build the latest head revisions of svn (for example) repositories in OE with bitbake has been limited and less than ideal.
Yes, exactly the stuff we run into.
Quoted 2 lines
To solve the problems I have made the following changes:
...A very detailed list of changes...
Could you explain what this means to a not-really-core-developer like me, i.e. what is the proposed exposure of your changes?
Regards,
Leon.
Quoted 5 lines · Leon Woestenberg
Quoted 4 lines · Richard Purdie
As people know the ability to build the latest head revisions of svn (for example) repositories in OE with bitbake has been limited and less than ideal.
Quoted 8 lines
Yes, exactly the stuff we run into.
Quoted 2 lines
To solve the problems I have made the following changes:
...A very detailed list of changes...
Could you explain what this means to a not-really-core-developer like me, i.e. what is the proposed exposure of your changes?
Just considering svn packages to keep this simple, package versions change to become:
PV = "x.y.z+svnr${SRCREV}"
eg. "3.6+svnr90" for svn revision 90
You can set
SRCREV_pn-somepackage = "90"
in a .conf file to force a particular revision or it will automatically increment the revision when svn head changes if its not locked down.
Whilst SRCDATE will still work, use of it will be removed for svn packages in OE and replaced with SRCREV.
Cheers,
Richard
Quoted 7 lines · Koen Kooi
Or what about naming URIs:
SRC_URI = "svn://foo/bar;name=main \ svn://foo/baz;name=baz-module \ svn://snizzle/foo;name=snizzle-foo-support"
SRCREV_FORMAT= "main"
This is more like what I had in mind allowing things like:
SRCREV_FORMAT = "rmain-rbaz-module-rsnizzle-foo-support"
which should result in a:
SRCREV of "r80-r90-r100"
which might even work over package upgrades :)
Cheers,
Richard
Quoted 11 lines · Richard Purdie
Just an update on the current status:
Quoted 6 lines · Koen Kooi
Richard Purdie schreef:
Quoted 5 lines
2. Handling of multiple SCMs in the
SRC_URI. The plan here is simple, any package doing this has to setSRCREV_FORMATspecifying the format of the SRCREV string.get_srcrev()can collect the SCM revisions, fill out the string and return it. The format and inner workings of this are open for discussion and to be determined.Not Done. Does anyone have any thoughts on this?
Now done.
URIs are tagged with a name field e.g.:
SRC_URI "\
svn://someurl/someplace;module=somemodule;name=myname1 \
svn://someurl2/someplace2;module=somemodule2;name=myname2 \
"
and then you set:
SRCREV_FORMAT = "myname1_rmyname2"
and you set it to however you want the SRCREV part of the version field formatted. This could result in:
SRCREV = "123_r456"
Quoted 3 lines
Quoted 1 line
Quoted 1 line
6. Add policy control variables to control the wiping of
BB_URI_HEADREVSNot Done.
Now done.
You can set this in local.conf:
BB_SRCREV_POLICY = "cache"BB_SRCREV_POLICY = "clear"
"cache" preserves the cached data, clear wipes it each time bitbake runs.
That should make SRCREV feature complete, it just needs lots of testing now, the manual needs updating and then we can release bitbake 1.8.8.
Cheers,
Richard