Skip to contentSkip to conversation

systemd Version Going Backwards on Warrior

DISCUSSION [email protected] 4 participants 7 messages Oct 28, 2019 – Oct 30, 2019

lore
Share & tools
lore
#0 lore raw

I'm using buildhistory in one of my builds that creates a package feed, and a recent update to systemd on warrior triggered version-going-backwards errors:

ERROR: systemd-conf-241+AUTOINC+511646b8ac-r0 do_packagedata: QA Issue: Package version for package systemd-conf-src went backwards which would break package feeds from (0:241+0+c1f8ff8d0d-r0 to 0:241+0+511646b8ac-r0) [version-going-backwards]

Should PE have been updated at the same time due to the hash making the version number go backwards? I can send a patch if that's all that's missing. Or is a PR server enough to prevent this? My debug builds do not use a PR server, but my production builds do use a PR server.

Thanks,
Robert

#1 lore raw
Quoted 5 lines · On 28/10/2019 16:25, [email protected] wrote:

I'm using buildhistory in one of my builds that creates a package feed, and a recent update to systemd on warrior triggered version-going-backwards errors:

ERROR: systemd-conf-241+AUTOINC+511646b8ac-r0 do_packagedata: QA Issue: Package version for package systemd-conf-src went backwards which would break package feeds from (0:241+0+c1f8ff8d0d-r0 to 0:241+0+511646b8ac-r0) [version-going-backwards]

Should PE have been updated at the same time due to the hash making the version number go backwards? I can send a patch if that's all that's missing. Or is a PR server enough to prevent this? My debug builds do not use a PR server, but my production builds do use a PR server.

If you're using feeds, you need to use a PR server. This is *exactly* what they are for.

Ross

#2 lore raw
Quoted 16 lines · Ross Burton
Quoted 9 lines · On 28/10/2019 16:25, [email protected] wrote:

I'm using buildhistory in one of my builds that creates a package feed, and a recent update to systemd on warrior triggered version- going-backwards errors:

ERROR: systemd-conf-241+AUTOINC+511646b8ac-r0 do_packagedata: QA
Issue: Package version for package systemd-conf-src went backwards which would break package feeds from (0:241+0+c1f8ff8d0d-r0 to 0:241+0+511646b8ac-r0) [version-going-backwards]

Quoted 5 lines

Should PE have been updated at the same time due to the hash making the version number go backwards? I can send a patch if that's all that's missing. Or is a PR server enough to prevent this? My debug builds do not use a PR server, but my production builds do use a PR server.

Quoted 4 lines

If you're using feeds, you need to use a PR server. This is *exactly* what they are for.

Ross

The part I wasn't sure about was if the PR server helped in the case where PV went backwards. I know it works when PV stays the same but the package was rebuilt. But if it keeps the versions going forward no matter how PV changes, then I should be good. I should probably setup a separate PR server on my debug builds to avoid this kind of error.

Thanks,
Robert

#3 lore raw
Quoted 21 lines · Robert Joslyn
Quoted 10 lines · Ross Burton
Quoted 9 lines · On 28/10/2019 16:25, [email protected] wrote:

I'm using buildhistory in one of my builds that creates a package feed, and a recent update to systemd on warrior triggered version- going-backwards errors:

ERROR: systemd-conf-241+AUTOINC+511646b8ac-r0 do_packagedata: QA
Issue: Package version for package systemd-conf-src went backwards which would break package feeds from (0:241+0+c1f8ff8d0d-r0 to 0:241+0+511646b8ac-r0) [version-going-backwards]

Quoted 8 lines
Quoted 5 lines

Should PE have been updated at the same time due to the hash making the version number go backwards? I can send a patch if that's all that's missing. Or is a PR server enough to prevent this? My debug builds do not use a PR server, but my production builds do use a PR server.

If you're using feeds, you need to use a PR server. This is *exactly* what they are for.

Quoted 5 lines

The part I wasn't sure about was if the PR server helped in the case where PV went backwards. I know it works when PV stays the same but the package was rebuilt. But if it keeps the versions going forward no matter how PV changes, then I should be good. I should probably setup a separate PR server on my debug builds to avoid this kind of error.

If the PV actually goes backwards then the PR service isn't useful, as PV sorts before PR.

However in this case the problem is that SRCREV should have a incrementing counter (the +0+ should be +1+ in the rebuild) which I believe comes from the PR service. I may be wrong...

Ross

#4 lore raw
Quoted 11 lines · Ross Burton
Quoted 10 lines · Robert Joslyn
Quoted 9 lines · Ross Burton
Quoted 8 lines · On 28/10/2019 16:25, [email protected] wrote:

I'm using buildhistory in one of my builds that creates a package feed, and a recent update to systemd on warrior triggered version- going-backwards errors:

ERROR: systemd-conf-241+AUTOINC+511646b8ac-r0 do_packagedata: QA
Issue: Package version for package systemd-conf-src went backwards which would break package feeds from (0:241+0+c1f8ff8d0d-r0 to 0:241+0+511646b8ac-r0) [version-going-backwards]

Isn't this do to the hashes being different and the PR server can't really tell which one is newer?

Quoted 27 lines
Quoted 11 lines
Quoted 9 lines
Quoted 6 lines

Should PE have been updated at the same time due to the hash making the version number go backwards? I can send a patch if that's all that's missing. Or is a PR server enough to prevent this? My debug builds do not use a PR server, but my production builds do use a PR server.

If you're using feeds, you need to use a PR server.  This is *exactly* what they are for.

Quoted 7 lines

The part I wasn't sure about was if the PR server helped in the case where
PV went backwards. I know it works when PV stays the same but the package
was rebuilt. But if it keeps the versions going forward no matter how PV changes, then I should be good. I should probably setup a separate PR server on my debug builds to avoid this kind of error.

If the PV actually goes backwards then the PR service isn't useful, as PV sorts before PR.

However in this case the problem is that SRCREV should have a incrementing counter (the +0+ should be +1+ in the rebuild) which I believe comes from the PR service.  I may be wrong...

Ross

#5 lore raw

PR server never knows which one is really newer (in git).

It just returns max(LOCALCOUNT)+1 when it gets query for hash which isn't stored in the database yet.

Either the build in question didn't use PRserv at all or PRserv's cache was deleted between builds or the builds were using the same buildhistory but different PRserv's or the first systemd SRCREV was reused from sstate created on another server which doesn't share the PRserv (so it didn't build it locally to query local PRserv to store 511646b8ac as LOCALCOUNT).

e.g. I've just built systemd with 511646b8ac hash as:
systemd_241+0+511646b8ac-r0.0webos4_qemux86.ipk
but reusing it from sstate created on another jenkins server as shown by buildstats-summary:

NOTE: Build completion summary:
NOTE:   do_populate_sysroot: 100.0% sstate reuse(124 setscene, 0 scratch)
NOTE:   do_package_qa: 100.0% sstate reuse(10 setscene, 0 scratch)
NOTE:   do_packagedata: 100.0% sstate reuse(51 setscene, 0 scratch)
NOTE:   do_package_write_ipk: 100.0% sstate reuse(10 setscene, 0 scratch)
NOTE:   do_populate_lic: 100.0% sstate reuse(17 setscene, 0 scratch)

Then I've reverted oe-core commit 8b9703454cb2a8a0aa6b7942498f191935d547ea to go back to c1f8ff8d0de7e303b8004b02a0a47d4cc103a7f8 systemd revision.

This time it haven't found valid sstate archive for it:

NOTE: Build completion summary:
NOTE:   do_populate_sysroot: 0.0% sstate reuse(0 setscene, 16 scratch)
NOTE:   do_package_qa: 0.0% sstate reuse(0 setscene, 19 scratch)
NOTE:   do_package: 15.8% sstate reuse(3 setscene, 16 scratch)
NOTE:   do_packagedata: 0.0% sstate reuse(0 setscene, 16 scratch)
NOTE:   do_package_write_ipk: 0.0% sstate reuse(0 setscene, 19 scratch)
NOTE:   do_populate_lic: 100.0% sstate reuse(2 setscene, 0 scratch)

and resulting .ipk has also +0:
systemd_241+0+c1f8ff8d0d-r0.0webos4_qemux86.ipk
but no warning is shown, because in this case it went from 511 to c1f.

Removing the revert again, doesn't trigger the warning again, because it will be again reused from sstate (and QA checks won't get executed):

NOTE: Build completion summary:
NOTE:   do_populate_sysroot: 100.0% sstate reuse(16 setscene, 0 scratch)
NOTE:   do_package_qa: 100.0% sstate reuse(19 setscene, 0 scratch)
NOTE:   do_packagedata: 100.0% sstate reuse(16 setscene, 0 scratch)
NOTE:   do_package_write_ipk: 100.0% sstate reuse(19 setscene, 0 scratch)
NOTE:   do_populate_lic: 100.0% sstate reuse(2 setscene, 0 scratch)

And the local PRserv database still has only the c1f8ff8d0d hash, because 511646b8ac was never really queried against this local PRserv.

cache$ sqlite3 prserv.sqlite3 "select * from PRMAIN_nohist where version like 'AUTOINC-systemd-1%'" AUTOINC-systemd-1_241+|qemux86|AUTOINC+c1f8ff8d0d|0

Also systemd recipe is using strange format with:
PV_append = "+${SRCPV}"

most recipes use "+git${SRCPV}" or "+gitr${SRCPV} to make it more clear where this +0+hash came from.

So long story short: the change is correct, PRserv should handle this, but there are many cases where it will fail (e.g.
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5399), but that's not a reason to start PE bumps everywhere.

Cheers,

Quoted 53 lines · akuster
Quoted 11 lines · Ross Burton
Quoted 10 lines · Robert Joslyn
Quoted 9 lines · Ross Burton
Quoted 8 lines · On 28/10/2019 16:25, [email protected] wrote:

I'm using buildhistory in one of my builds that creates a package feed, and a recent update to systemd on warrior triggered version- going-backwards errors:

ERROR: systemd-conf-241+AUTOINC+511646b8ac-r0 do_packagedata: QA
Issue: Package version for package systemd-conf-src went backwards which would break package feeds from (0:241+0+c1f8ff8d0d-r0 to 0:241+0+511646b8ac-r0) [version-going-backwards]

Isn't this do to the hashes being different and the PR server can't really tell which one is newer?

Quoted 19 lines
Quoted 11 lines
Quoted 9 lines
Quoted 6 lines

Should PE have been updated at the same time due to the hash making the version number go backwards? I can send a patch if that's all that's missing. Or is a PR server enough to prevent this? My debug builds do not use a PR server, but my production builds do use a PR server.

If you're using feeds, you need to use a PR server. This is *exactly* what they are for.

Quoted 7 lines

The part I wasn't sure about was if the PR server helped in the case where
PV went backwards. I know it works when PV stays the same but the package
was rebuilt. But if it keeps the versions going forward no matter how PV changes, then I should be good. I should probably setup a separate PR server on my debug builds to avoid this kind of error.

Quoted 8 lines

If the PV actually goes backwards then the PR service isn't useful, as PV sorts before PR.

However in this case the problem is that SRCREV should have a incrementing counter (the +0+ should be +1+ in the rebuild) which I believe comes from the PR service. I may be wrong...

Ross

--
_______________________________________________
yocto mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/yocto

#6 lore raw
Quoted 63 lines · Martin Jansa

PR server never knows which one is really newer (in git).

It just returns max(LOCALCOUNT)+1 when it gets query for hash which isn't stored in the database yet.

Either the build in question didn't use PRserv at all or PRserv's cache was deleted between builds or the builds were using the same buildhistory but different PRserv's or the first systemd SRCREV was reused from sstate created on another server which doesn't share the PRserv (so it didn't build it locally to query local PRserv to store 511646b8ac as LOCALCOUNT).

e.g. I've just built systemd with 511646b8ac hash as:
systemd_241+0+511646b8ac-r0.0webos4_qemux86.ipk
but reusing it from sstate created on another jenkins server as shown by buildstats-summary:

NOTE: Build completion summary:
NOTE: do_populate_sysroot: 100.0% sstate reuse(124 setscene, 0
scratch)

NOTE:   do_package_qa: 100.0% sstate reuse(10 setscene, 0 scratch)
NOTE:   do_packagedata: 100.0% sstate reuse(51 setscene, 0 scratch)
NOTE:   do_package_write_ipk: 100.0% sstate reuse(10 setscene, 0

scratch)
NOTE: do_populate_lic: 100.0% sstate reuse(17 setscene, 0 scratch)

Then I've reverted oe-core
commit 8b9703454cb2a8a0aa6b7942498f191935d547ea to go back to c1f8ff8d0de7e303b8004b02a0a47d4cc103a7f8 systemd revision.

This time it haven't found valid sstate archive for it:

NOTE: Build completion summary:
NOTE:   do_populate_sysroot: 0.0% sstate reuse(0 setscene, 16 scratch)
NOTE:   do_package_qa: 0.0% sstate reuse(0 setscene, 19 scratch)
NOTE:   do_package: 15.8% sstate reuse(3 setscene, 16 scratch)
NOTE:   do_packagedata: 0.0% sstate reuse(0 setscene, 16 scratch)
NOTE:   do_package_write_ipk: 0.0% sstate reuse(0 setscene, 19 scratch)
NOTE:   do_populate_lic: 100.0% sstate reuse(2 setscene, 0 scratch)

and resulting .ipk has also +0:
systemd_241+0+c1f8ff8d0d-r0.0webos4_qemux86.ipk
but no warning is shown, because in this case it went from 511 to c1f.

Removing the revert again, doesn't trigger the warning again, because it will be again reused from sstate (and QA checks won't get executed):

NOTE: Build completion summary:
NOTE:   do_populate_sysroot: 100.0% sstate reuse(16 setscene, 0 scratch)
NOTE:   do_package_qa: 100.0% sstate reuse(19 setscene, 0 scratch)
NOTE:   do_packagedata: 100.0% sstate reuse(16 setscene, 0 scratch)
NOTE:   do_package_write_ipk: 100.0% sstate reuse(19 setscene, 0

scratch)
NOTE: do_populate_lic: 100.0% sstate reuse(2 setscene, 0 scratch)

And the local PRserv database still has only the c1f8ff8d0d hash, because 511646b8ac was never really queried against this local PRserv.

cache$ sqlite3 prserv.sqlite3 "select * from PRMAIN_nohist where version like 'AUTOINC-systemd-1%'" AUTOINC-systemd-1_241+|qemux86|AUTOINC+c1f8ff8d0d|0

Also systemd recipe is using strange format with:
PV_append = "+${SRCPV}"

Quoted 7 lines

most recipes use "+git${SRCPV}" or "+gitr${SRCPV} to make it more clear where this +0+hash came from.

So long story short: the change is correct, PRserv should handle this, but there are many cases where it will fail (e.g.
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5399), but that's not a reason to start PE bumps everywhere.

I think this explains what I'm seeing and matches what Ross said. I did setup the same test and am able to see the version going forward properly when I have the PR server enabled. The file goes from

systemd_241+0+c1f8ff8d0d-r0.0_core2-64.ipk
to
systemd_241+1+511646b8ac-r0.0_core2-64.ipk

It wasn't obvious to me that the +0 would be incremented by the PR server, I guess I never noticed it before. I already had the PR server running for my production builds, but I didn't have it enabled for my test builds where I got the error. I'll setup another PR server for my test builds to prevent false alarms like this.

Thanks for the help!

Robert