Questions on HOSTTOOLS and the documented host packages in the manual

Hi everyone

To get repeatable quality builds we ensure the build agents in CI are identical and that they install a _minimal_ list of packages taken from the instructions in the Yocto manual [1].  But from time to time some dependency is missing, especially since things change in Yocto over time (e.g.
gzipnative class, and such). So we were even missing gzip in one build
failure. Now I couldn't confirm that the gzip binary is indeed missing - it seems to be on our standard image, but possibly the configure step was looking for some -devel package, headers or whatnot.

Still we have had our share of "works on my machine" answers from users that don't use a controlled, docker-container like that of the CI build agents.

Sidestepping, since I anticipate this in your response: I know there's continuous work on repeatability, and tried to read what I can in the manual on the new HOSTTOOL variable. The mechanics of $PATH and softlinking the binaries are clear, but I think I missed the intended usage and process
around it. First, is it expected/recommended that each layer modifies this
variable? If so, would layers add new host dependencies, or is the intention to have layers further limit this with its own whitelist but still based on a predefined set coming from the Poky layer?

I get that technically speaking you build in a lot of freedom, but from a process perspective I think if the Yocto project defines a minimal set of required host tools, then a Yocto-based project (like GDP) could take the stance that all other dependencies must come from DEPENDS declaration on a
-native package. Unless there is a clear agreed baseline, it's difficult
to file a bug on a recipe with respect to their -native dependencies.

Please confirm, is the manual (I refer to the Mega Manual [1]) intended to be such a canonical list of host dependencies?

But if so: Is it intentional or a bug that gzip (and bzip2, perhaps others) is listed to be installed for Fedora, but not listed to be installed for Ubuntu/Debian, see [1]?  Maybe the idea was that some things are always there by default on those distros, but that seems brittle. Shouldn't every package be listed very explicitly?

Appreciate your comments...

- Gunnar

[1] http://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html#packages

Layers can extend the variables if they have explicit host dependencies, but I'd be surprised and concerned if this was a common occurrence.

For example I see that meta-oe adds "id" to HOSTTOOLS (for the crash recipe), but if that was submitted to oe-core it would be rejected. The only reason as far as I can tell to call id is to get the current uid/gid, and that then introduces another vector of non-determinism in builds. We recommend that be patched out and replaced with constants: see also the kernel build which by default uses whoami to embed the username of the kernel builder, but we export KBUILD_BUILD_USER=oe-user to remove that source of non-determinism.

Please confirm, is the manual (I refer to the Mega Manual [1]) intended to

Yes, please file a bug for missing ones. The lists are intended to be packages installed over the base install, but container images being *very* minimal are often missing dependencies so we need to be sure everything is covered.

Ross