hash generation/PR service problem with xuser-account and other packages
As i mentioned in another thread before we're investigating some problems related with package versions going backwards using a PR service.
Now i have some more information. But i'm not able to understand and fix the problem for myself.
Assume we have a clean workspace and i bitbake the xuser-account package with this command:
MACHINE="foo" bitbake xuser-account
This generates these packages (the 207 comes from our PR service).
tmp-eglibc/deploy/ipk/all/xuser-account-dbg_0.1-r0.207_all.ipk tmp-eglibc/deploy/ipk/all/xuser-account-dev_0.1-r0.207_all.ipk tmp-eglibc/deploy/ipk/all/xuser-account_0.1-r0.207_all.ipk
If i call the same bitbake command again, the same packages are generated. Fine!
Now i call:
MACHINE="bar" bitbake xuser-account
The generated packages now get a new number from the PR server (so they have a different hash i believe).
tmp-eglibc/deploy/ipk/all/xuser-account-dbg_0.1-r0.208_all.ipk tmp-eglibc/deploy/ipk/all/xuser-account-dev_0.1-r0.208_all.ipk tmp-eglibc/deploy/ipk/all/xuser-account_0.1-r0.208_all.ipk
But this package does not seem to be machine dependent!?
If the next build for machine foo is a clean build again (e.g. on our Jenkins continuous integration server which makes clean builds once a week) the PR service generated number wents back to 207 which results in:
"ERROR: QA Issue: Package version for package xuser-account went backwards which would break package feeds"
Can someone confirm this behaviour?
Is this a bug? I think so.
Can someone give some details what went wrong here? Also after studying <http://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#checksums> i was not able to fully understand the background, not to speak of fixing something.
BTW: We see the same behaviour for the cpufreq-tweaks, linux-firmware, run-postinsts, and some of our own packages.
Thanx for any help,
Steffen
PS: We're working on the daisy branches using Angstrom with some more own layers.
Signature · 7 lines
DResearch Fahrzeugelektronik GmbH Otto-Schmirgal-Str. 3, 10319 Berlin, Germany Tel: +49 30 515932-237 mailto:[email protected] Fax: +49 30 515932-299 Geschäftsführer: Dr. Michael Weber, Werner Mögle; Amtsgericht Berlin Charlottenburg; HRB 130120 B; Ust.-IDNr. DE273952058
Quoted 32 lines · Steffen Sledz
As i mentioned in another thread before we're investigating some problems related with package versions going backwards using a PR service.
Now i have some more information. But i'm not able to understand and fix the problem for myself.
Assume we have a clean workspace and i bitbake the xuser-account package with this command:
MACHINE="foo" bitbake xuser-account
This generates these packages (the 207 comes from our PR service).
tmp-eglibc/deploy/ipk/all/xuser-account-dbg_0.1-r0.207_all.ipk tmp-eglibc/deploy/ipk/all/xuser-account-dev_0.1-r0.207_all.ipk tmp-eglibc/deploy/ipk/all/xuser-account_0.1-r0.207_all.ipk
If i call the same bitbake command again, the same packages are generated. Fine!
Now i call:
MACHINE="bar" bitbake xuser-account
The generated packages now get a new number from the PR server (so they have a different hash i believe).
tmp-eglibc/deploy/ipk/all/xuser-account-dbg_0.1-r0.208_all.ipk tmp-eglibc/deploy/ipk/all/xuser-account-dev_0.1-r0.208_all.ipk tmp-eglibc/deploy/ipk/all/xuser-account_0.1-r0.208_all.ipk
But this package does not seem to be machine dependent!?
If the next build for machine foo is a clean build again (e.g. on our Jenkins continuous integration server which makes clean builds once a week) the PR service generated number wents back to 207 which results in:
"ERROR: QA Issue: Package version for package xuser-account went backwards which would break package feeds"
Quoted 7 lines
Can someone confirm this behaviour?
Is this a bug? I think so.
Can someone give some details what went wrong here? Also after studying <http://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#checksums> i was not able to fully understand the background, not to speak of fixing something.
BTW: We see the same behaviour for the cpufreq-tweaks, linux-firmware, run-postinsts, and some of our own packages.
I just took a look at this on master. There are some changes between master and daisy and I want to get this resolved in master before we think about daisy. I thought walking through debugging this may be helpful, so, I did:
MACHINE=qemux86 bitbake xuser-account
MACHINE=qemuarm bitbake xuser-account
and sure enough, you are correct, things rebuild and they should not for an allarch package like this. Looking at the list of packages being built, its pulling in the toolchain. I checked:
MACHINE=qemuarm bitbake xuser-account -e | grep INHIBIT_DEFAULT_DEPS
and sure enough, that is set (as per allarch.bbclass) so why is it building the toolchain?! It has to be a dependency, so:
MACHINE=qemuarm bitbake xuser-account -e | grep DEPENDS=
and in there we see:
DEPENDS=" base-files base-passwd shadow-native shadow-sysroot shadow"
base-files is whitelisted in conf/layer.conf as a dependency we don't need to checksum. The checksum of shadow-native isn't an issue, the others however are a problem. Since they have well behaved APIs, we can add them to the list in layer.conf.
What this does mean is that if shadow rebuilds, xuser-account will not rebuild automatically. I don't believe that should be an issue.
After doing this, I reran the bitbake commands and then:
ls tmp/stamps/all-poky-linux/xuser-account/
shows things like:
0.1-r0.do_populate_sysroot.978b6f9bfd5282121de701a24dc52c80.qemuarm 0.1-r0.do_populate_sysroot.978b6f9bfd5282121de701a24dc52c80.qemux86
basically, the stamp hashes should not change between the two bitbake commands. SO the patch needed to fix xuser is:
meta/conf/layer.conf +3 -0
@@ -31,6 +31,9 @@ SIGGEN_EXCLUDERECIPES_ABISAFE += " \ packagegroup-x11-xserver \ systemd-serialgetty \ initscripts \ + shadow \ + shadow-sysroot \ + base-passwd \ " SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS += " \
The other recipes will need investigating since I doubt its this issue for them.
Cheers,
Richard
Quoted 90 lines · Richard Purdie
Quoted 32 lines · Steffen Sledz
As i mentioned in another thread before we're investigating some problems related with package versions going backwards using a PR service.
Now i have some more information. But i'm not able to understand and fix the problem for myself.
Assume we have a clean workspace and i bitbake the xuser-account package with this command:
MACHINE="foo" bitbake xuser-account
This generates these packages (the 207 comes from our PR service).
tmp-eglibc/deploy/ipk/all/xuser-account-dbg_0.1-r0.207_all.ipk tmp-eglibc/deploy/ipk/all/xuser-account-dev_0.1-r0.207_all.ipk tmp-eglibc/deploy/ipk/all/xuser-account_0.1-r0.207_all.ipk
If i call the same bitbake command again, the same packages are generated. Fine!
Now i call:
MACHINE="bar" bitbake xuser-account
The generated packages now get a new number from the PR server (so they have a different hash i believe).
tmp-eglibc/deploy/ipk/all/xuser-account-dbg_0.1-r0.208_all.ipk tmp-eglibc/deploy/ipk/all/xuser-account-dev_0.1-r0.208_all.ipk tmp-eglibc/deploy/ipk/all/xuser-account_0.1-r0.208_all.ipk
But this package does not seem to be machine dependent!?
If the next build for machine foo is a clean build again (e.g. on our Jenkins continuous integration server which makes clean builds once a week) the PR service generated number wents back to 207 which results in:
"ERROR: QA Issue: Package version for package xuser-account went backwards which would break package feeds"
Quoted 7 lines
Can someone confirm this behaviour?
Is this a bug? I think so.
Can someone give some details what went wrong here? Also after studying <http://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#checksums> i was not able to fully understand the background, not to speak of fixing something.
BTW: We see the same behaviour for the cpufreq-tweaks, linux-firmware, run-postinsts, and some of our own packages.
I just took a look at this on master. There are some changes between master and daisy and I want to get this resolved in master before we think about daisy. I thought walking through debugging this may be helpful, so, I did:
MACHINE=qemux86 bitbake xuser-account
MACHINE=qemuarm bitbake xuser-accountand sure enough, you are correct, things rebuild and they should not for an allarch package like this. Looking at the list of packages being built, its pulling in the toolchain. I checked:
MACHINE=qemuarm bitbake xuser-account -e | grep
INHIBIT_DEFAULT_DEPSand sure enough, that is set (as per allarch.bbclass) so why is it building the toolchain?! It has to be a dependency, so:
MACHINE=qemuarm bitbake xuser-account -e | grep DEPENDS=
and in there we see:
DEPENDS=" base-files base-passwd shadow-native shadow-sysroot shadow"
base-files is whitelisted in
conf/layer.confas a dependency we don't need to checksum. The checksum of shadow-native isn't an issue, the others however are a problem. Since they have well behaved APIs, we can add them to the list in layer.conf.What this does mean is that if shadow rebuilds, xuser-account will not rebuild automatically. I don't believe that should be an issue.
After doing this, I reran the bitbake commands and then:
ls tmp/stamps/all-poky-linux/xuser-account/
shows things like:
0.1-r0.do_populate_sysroot.978b6f9bfd5282121de701a24dc52c80.qemuarm 0.1-r0.do_populate_sysroot.978b6f9bfd5282121de701a24dc52c80.qemux86
basically, the stamp hashes should not change between the two bitbake commands. SO the patch needed to fix xuser is:
meta/conf/layer.conf +0
-0
meta/conf/layer.conf SIGGEN_EXCLUDERECIPES_ABISAFE += " \ · line 39
Quoted 9 lines
meta/conf/layer.conf +3
-0@@ -31,6 +31,9 @@ SIGGEN_EXCLUDERECIPES_ABISAFE += " \ packagegroup-x11-xserver \ systemd-serialgetty \ initscripts \ + shadow \ + shadow-sysroot \ + base-passwd \ " SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS += " \
Hi Richard, thanx for this detailled explanation. It is another step on my way to understand all this. ;-)
Quoted 2 lines
The other recipes will need investigating since I doubt its this issue for them.
Is there someone who can do this work in a similar brightening way Richard did?
Signature · 7 lines
DResearch Fahrzeugelektronik GmbH Otto-Schmirgal-Str. 3, 10319 Berlin, Germany Tel: +49 30 515932-237 mailto:[email protected] Fax: +49 30 515932-299 Geschäftsführer: Dr. Michael Weber, Werner Mögle; Amtsgericht Berlin Charlottenburg; HRB 130120 B; Ust.-IDNr. DE273952058
Quoted 6 lines · Steffen Sledz
Hi Richard, thanx for this detailled explanation. It is another step on my way to understand all this. ;-)
Quoted 2 lines
The other recipes will need investigating since I doubt its this issue for them.
Is there someone who can do this work in a similar brightening way Richard did?
I had a look at run-postinsts:
MACHINE=qemux86 bitbake run-postinsts
MACHINE=qemuarm bitbake run-postinsts
ls tmp/stamps/all-poky-linux/run-postinsts/
Hmm, two different sigdata files for do_package.
bitbake-diffsigs tmp/stamps/all-poky-linux/run-postinsts/1.0-r9.do_package.sigdata.* (assumes only two sigdata files there, idea is to compare them)
Difference is the OVERRIDES variable. We therefore dump one of the files:
bitbake-diffsigs tmp/stamps/all-poky-linux/run-postinsts/1.0-r9.do_package.sigdata.X | grep OVERRIDES
and see that OVERRIDES is depended on in two places:
List of dependencies for variable systemd_populate_packages is set(['systemd_prerm', 'D', 'exec_prefix', 'OVERRIDES', 'sysconfdir', 'SYSTEMD_PACKAGES', 'systemd_postinst', 'nonarch_base_libdir', 'PACKAGES'])
List of dependencies for variable populate_packages_updatercd is set(['D', 'updatercd_postrm', 'OVERRIDES', 'INITSCRIPT_NAME', 'INHIBIT_UPDATERCD_BBCLASS', 'MLPREFIX', 'updatercd_postinst', 'updatercd_preinst', 'updatercd_prerm', 'INIT_D_DIR', 'PACKAGES', 'INITSCRIPT_PACKAGES', 'UPDATERCPN'])
So the patch to fix this is:
meta/classes/systemd.bbclass +2 -0
@@ -55,6 +55,8 @@ fi systemd_populate_packages[vardeps] += "systemd_prerm systemd_postinst" +systemd_populate_packages[vardepsexclude] += "OVERRIDES" + python systemd_populate_packages() { if not bb.utils.contains('DISTRO_FEATURES', 'systemd', True, False, d):
meta/classes/update-rc.d.bbclass +1 -0
@@ -72,6 +72,7 @@ PACKAGESPLITFUNCS_prepend = "populate_packages_updatercd " PACKAGESPLITFUNCS_remove_class-nativesdk = "populate_packages_updatercd " populate_packages_updatercd[vardeps] += "updatercd_prerm updatercd_postrm updatercd_preinst updatercd_postinst" +populate_packages_updatercd[vardepsexclude] += "OVERRIDES" python populate_packages_updatercd () { def update_rcd_auto_depend(pkg):
since OVERRIDES isn't a needed dependency here, its effect is all that matters, not its contents.
Cheers,
Richard
Quoted 58 lines · Richard Purdie
Quoted 6 lines · Steffen Sledz
Hi Richard, thanx for this detailled explanation. It is another step on my way to understand all this. ;-)
Quoted 2 lines
The other recipes will need investigating since I doubt its this issue for them.
Is there someone who can do this work in a similar brightening way Richard did?
I had a look at run-postinsts:
MACHINE=qemux86 bitbake run-postinsts
MACHINE=qemuarm bitbake run-postinstsls tmp/stamps/all-poky-linux/run-postinsts/
Hmm, two different sigdata files for
do_package.bitbake-diffsigs tmp/stamps/all-poky-linux/run-postinsts/1.0-r9.do_package.sigdata.* (assumes only two sigdata files there, idea is to compare them)
Difference is the OVERRIDES variable. We therefore dump one of the files:
bitbake-diffsigs tmp/stamps/all-poky-linux/run-postinsts/1.0-r9.do_package.sigdata.X | grep OVERRIDES
and see that OVERRIDES is depended on in two places:
List of dependencies for variable
systemd_populate_packagesis set(['systemd_prerm', 'D', 'exec_prefix', 'OVERRIDES', 'sysconfdir', 'SYSTEMD_PACKAGES', 'systemd_postinst', 'nonarch_base_libdir', 'PACKAGES'])
List of dependencies for variablepopulate_packages_updatercdis set(['D', 'updatercd_postrm', 'OVERRIDES', 'INITSCRIPT_NAME', 'INHIBIT_UPDATERCD_BBCLASS', 'MLPREFIX', 'updatercd_postinst', 'updatercd_preinst', 'updatercd_prerm', 'INIT_D_DIR', 'PACKAGES', 'INITSCRIPT_PACKAGES', 'UPDATERCPN'])So the patch to fix this is:
meta/classes/systemd.bbclass +2
-0@@ -55,6 +55,8 @@ fi systemd_populate_packages[vardeps] += "systemd_prerm systemd_postinst" +systemd_populate_packages[vardepsexclude] += "OVERRIDES" + python systemd_populate_packages() { if not bb.utils.contains('DISTRO_FEATURES', 'systemd', True, False, d):meta/classes/update-rc.d.bbclass +1
-0@@ -72,6 +72,7 @@ PACKAGESPLITFUNCS_prepend = "populate_packages_updatercd " PACKAGESPLITFUNCS_remove_class-nativesdk = "populate_packages_updatercd " populate_packages_updatercd[vardeps] += "updatercd_prerm updatercd_postrm updatercd_preinst updatercd_postinst" +populate_packages_updatercd[vardepsexclude] += "OVERRIDES" python populate_packages_updatercd () { def update_rcd_auto_depend(pkg):
Quoted 1 line
since OVERRIDES isn't a needed dependency here, its effect is all that matters, not its contents.
This patch fixes the cpufreq-tweaks (from meta-angstrom) problem too. :)
So there's still linux-firmware pending from the packages i mentioned.
Signature · 7 lines
DResearch Fahrzeugelektronik GmbH Otto-Schmirgal-Str. 3, 10319 Berlin, Germany Tel: +49 30 515932-237 mailto:[email protected] Fax: +49 30 515932-299 Geschäftsführer: Dr. Michael Weber, Werner Mögle; Amtsgericht Berlin Charlottenburg; HRB 130120 B; Ust.-IDNr. DE273952058
Quoted 3 lines · Steffen Sledz
This patch fixes the cpufreq-tweaks (from meta-angstrom) problem too. :)
So there's still linux-firmware pending from the packages i mentioned.
Actually, I pushed a patch for that last night. It was the same debug approach as the second case I mentioned, diff the sigs and the issue became clear.
Cheers,
Richard