Skip to contentSkip to conversation

Weird build dependency issue causing missing symbols

DISCUSSION Bills, Jason M 3 participants 8 messages Jul 2, 2020 – Jul 6, 2020

lore
Share & tools
lore
#0 lore raw

Hi All,

We are hitting a weird build dependency issue with Yocto and phosphor-dbus-interfaces and are looking for any help or insight anyone may have on how to fix it. We have not been able to pinpoint exactly when the issue started, but we believe it has come up since the dunfell update.

The symptom of this issue is we see an undefined symbol error at runtime:

[  101.733677] Jul 02 10:37:48 intel-obmc phosphor-ledcontroller[461]: phosph

or-ledcontroller: symbol lookup error: phosphor-ledcontroller: undefined symbol: _ZN9sdbusplus3xyz15openbmc_project3Led6server8Physical17setPropertyByNameERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt7variantIJtNS4_7PaletteEhNS4_6ActionEEEb

Once we hit this error, it persists across rebuilds until we delete the Yocto build directory (likely something in the cache) and start a fresh build.

We have narrowed this down to being caused by two separate issues:
1. When phosphor-dbus-interfaces is rebuilt it will sometimes change the order of the PropertiesVariant in server.hpp.
2. When the order of PropertiesVariant changes on a rebuild, the recipes that already have an old copy of server.hpp are not triggered to rebuild and are left with the old copy of server.hpp.

I have a system that is in this state and have found that if I taint phosphor-dbus-interfaces by running "bitbake -C fetch phosphor-dbus-interfaces", I see many components rebuild and the symbol issue goes away. If I then remove the taint by running "bitbake -c clean phosphor-dbus-interfaces" only phosphor-dbus-interfaces and any components in my devtool status list rebuild and the symbol issue comes back.

We ran an experiment where we compared the contents of ".../Led/Physical/server.hpp" between components by running this command (where the base file came from an existing build):
for fname in $(find . -iname server.hpp|grep -i "led/physical"); do echo "$fname"; diff "./tmp/work/arm1176jzs-openbmc-linux-gnueabi/phosphor-network/1.0+gitAUTOINC+d0679f9bb4-r1/recipe-sysroot/usr/include/xyz/openbmc_project/Led/Physical/server.hpp" "$fname"; done

With the tainted phosphor-dbus-interfaces, there is no diff in any of the server.hpp files.

After cleaning the taint and rebuilding, I get the following results:
./tmp/work/arm1176jzs-openbmc-linux-gnueabi/phosphor-sel-logger/0.1+gitAUTOINC+761bf202ba-r0/recipe-sysroot/usr/include/xyz/openbmc_project/Led/Physical/server.hpp ./tmp/work/arm1176jzs-openbmc-linux-gnueabi/intel-ipmi-oem/0.1+gitAUTOINC+e4f710d7d9-r0/recipe-sysroot/usr/include/xyz/openbmc_project/Led/Physical/server.hpp ./tmp/work/arm1176jzs-openbmc-linux-gnueabi/phosphor-dbus-interfaces/1.0+gitAUTOINC+26ff1c8446-r1/sysroot-destdir/usr/include/xyz/openbmc_project/Led/Physical/server.hpp 66,67c66
< Action,
< uint16_t,
---

Quoted 1 line
                uint8_t,

69c68,69
< uint8_t>;
---

Quoted 2 lines
                Action,
                uint16_t>;

./tmp/work/arm1176jzs-openbmc-linux-gnueabi/phosphor-ipmi-ipmb/0.1+gitAUTOINC+a86059348f-r0/recipe-sysroot/usr/include/xyz/openbmc_project/Led/Physical/server.hpp ./tmp/work/arm1176jzs-openbmc-linux-gnueabi/phosphor-host-postd/0.1+gitAUTOINC+bf002b46d5-r1/recipe-sysroot/usr/include/xyz/openbmc_project/Led/Physical/server.hpp ./tmp/work/arm1176jzs-openbmc-linux-gnueabi/phosphor-network/1.0+gitAUTOINC+d0679f9bb4-r1/recipe-sysroot/usr/include/xyz/openbmc_project/Led/Physical/server.hpp ./tmp/work/arm1176jzs-openbmc-linux-gnueabi/x86-power-control/1.0+gitAUTOINC+b0c613aa88-r0/recipe-sysroot/usr/include/xyz/openbmc_project/Led/Physical/server.hpp 66,67c66
< Action,
< uint16_t,
---

Quoted 1 line
                uint8_t,

69c68,69
< uint8_t>;
---

Quoted 2 lines
                Action,
                uint16_t>;

./tmp/work/arm1176jzs-openbmc-linux-gnueabi/phosphor-post-code-manager/1.0+gitAUTOINC+9d91a39a3a-r0/recipe-sysroot/usr/include/xyz/openbmc_project/Led/Physical/server.hpp ./tmp/work/arm1176jzs-openbmc-linux-gnueabi/obmc-ikvm/1.0+gitAUTOINC+861337e8ec-r0/recipe-sysroot/usr/include/xyz/openbmc_project/Led/Physical/server.hpp ./tmp/work/arm1176jzs-openbmc-linux-gnueabi/service-config-manager/0.1+gitAUTOINC+83241c09ec-r0/recipe-sysroot/usr/include/xyz/openbmc_project/Led/Physical/server.hpp ./tmp/work/arm1176jzs-openbmc-linux-gnueabi/phosphor-ipmi-kcs/1.0+gitAUTOINC+d8594e9a62-r1/recipe-sysroot/usr/include/xyz/openbmc_project/Led/Physical/server.hpp ./tmp/sysroots-components/arm1176jzs/phosphor-dbus-interfaces/usr/include/xyz/openbmc_project/Led/Physical/server.hpp
66,67c66
< Action,
< uint16_t,
---

Quoted 1 line
                uint8_t,

69c68,69
< uint8_t>;
---

Quoted 2 lines
                Action,
                uint16_t>;

The order of the variant changed in server.hpp in phosphor-dbus-interfaces. I had x86-power-control in my devtool status list, so it rebuilt and got the new copy of server.hpp, but everything else still had the old copy.

Does anyone have any ideas on what could be happening or if we're missing something to properly trigger the rebuilds?

Thanks for your help!
-Jason

#1 lore raw
Quoted 3 lines · Jason M

We have narrowed this down to being caused by two separate issues:
1. When phosphor-dbus-interfaces is rebuilt it will sometimes change the order of the PropertiesVariant in server.hpp.

This sounds like a bug in sdbus++. We should be sorting the variant parameters or issuing them in array order. I'll look into it.

Quoted 3 lines

2. When the order of PropertiesVariant changes on a rebuild, the recipes that already have an old copy of server.hpp are not triggered to rebuild and are left with the old copy of server.hpp.

This isn't surprising if what is triggering the rebuild is not a Yocto variable change (or git revision). Yocto doesn't cache the contents of the packages, but caches the variables that went into a build step. A hash of the variables are used to look up the potential 'sstate-cache' files so that it can skip build steps.

If you think a variable or a git-revision should have changed with what you were doing, then maybe it is something else.

Signature · 1 line
Patrick Williams
#2 lore raw
Quoted 7 lines · Patrick Williams
Quoted 3 lines · Jason M

We have narrowed this down to being caused by two separate issues:
1. When phosphor-dbus-interfaces is rebuilt it will sometimes change the order of the PropertiesVariant in server.hpp.

This sounds like a bug in sdbus++. We should be sorting the variant parameters or issuing them in array order. I'll look into it.

sdbus++: server: fix PropertyVariant order
https://gerrit.openbmc-project.xyz/c/openbmc/sdbusplus/+/34344

Signature · 1 line
Patrick Williams
#3 lore raw
Quoted 6 lines · Patrick Williams
Quoted 5 lines · Patrick Williams
Quoted 3 lines · Jason M

We have narrowed this down to being caused by two separate issues:
1. When phosphor-dbus-interfaces is rebuilt it will sometimes change the order of the PropertiesVariant in server.hpp.

Quoted 6 lines
Quoted 2 lines

This sounds like a bug in sdbus++. We should be sorting the variant parameters or issuing them in array order. I'll look into it.

sdbus++: server: fix PropertyVariant order
https://gerrit.openbmc-project.xyz/c/openbmc/sdbusplus/+/34344

Thanks for the quick response! I tested this change with tainting and cleaning phosphor-dbus-interfaces. It still shows this diff after the clean build, but I don't know what is different and it doesn't trigger the symbol error:
./tmp/sysroots-components/arm1176jzs/phosphor-dbus-interfaces/usr/include/xyz/openbmc_project/Led/Physical/server.hpp
67d66
< uint8_t,
68a68

Quoted 1 line
                uint8_t,

This looks good! Thanks again!

#4 lore raw
In reply to Patrick Williams
Quoted 12 lines · Patrick Williams
Quoted 3 lines · Jason M

We have narrowed this down to being caused by two separate issues:
1. When phosphor-dbus-interfaces is rebuilt it will sometimes change the order of the PropertiesVariant in server.hpp.

This sounds like a bug in sdbus++. We should be sorting the variant parameters or issuing them in array order. I'll look into it.

Quoted 3 lines

2. When the order of PropertiesVariant changes on a rebuild, the recipes that already have an old copy of server.hpp are not triggered to rebuild and are left with the old copy of server.hpp.

Quoted 9 lines

This isn't surprising if what is triggering the rebuild is not a Yocto variable change (or git revision). Yocto doesn't cache the contents of the packages, but caches the variables that went into a build step. A hash of the variables are used to look up the potential 'sstate-cache' files so that it can skip build steps.

If you think a variable or a git-revision should have changed with what you were doing, then maybe it is something else.

It seems like a header file change should trigger a rebuild, though? If I manually edited something like a library header file, I'd expect everything that includes that library to be rebuilt with the new header change.

I tried to devtool modify boost to check the behavior, but that causes boost to rebuild every time and correctly triggers the dependent builds.
Maybe the case above of modifying a header file is invalid?

#5 lore raw

Maybe the problem is that this header server.hpp is generated not a source.
Artifact created from the same source *should* be the same (except timestamp)
If the source did not update, just a forced rebuild to create new binaries, I can see Yocto choose not to rebuild things depend on the package. In the case of boost, since it is devtool modified and the header is a source and not build artifact, it make sense to trigger all the rebuild.

-----Original Message-----
From: openbmc <[email protected]> On Behalf Of Bills, Jason M
Sent: Thursday, July 2, 2020 5:00 PM
To: [email protected]
Subject: Re: Weird build dependency issue causing missing symbols

Quoted 12 lines · Patrick Williams
Quoted 3 lines · Jason M

We have narrowed this down to being caused by two separate issues:
1. When phosphor-dbus-interfaces is rebuilt it will sometimes change the order of the PropertiesVariant in server.hpp.

This sounds like a bug in sdbus++. We should be sorting the variant parameters or issuing them in array order. I'll look into it.

Quoted 3 lines

2. When the order of PropertiesVariant changes on a rebuild, the recipes that already have an old copy of server.hpp are not triggered to rebuild and are left with the old copy of server.hpp.

Quoted 9 lines

This isn't surprising if what is triggering the rebuild is not a Yocto variable change (or git revision). Yocto doesn't cache the contents of the packages, but caches the variables that went into a build step. A hash of the variables are used to look up the potential 'sstate-cache' files so that it can skip build steps.

If you think a variable or a git-revision should have changed with what you were doing, then maybe it is something else.

It seems like a header file change should trigger a rebuild, though? If I manually edited something like a library header file, I'd expect everything that includes that library to be rebuilt with the new header change.

I tried to devtool modify boost to check the behavior, but that causes boost to rebuild every time and correctly triggers the dependent builds.
Maybe the case above of modifying a header file is invalid?

#6 lore raw
Quoted 33 lines · Zhikui

Maybe the problem is that this header server.hpp is generated not a source.
Artifact created from the same source *should* be the same (except timestamp)
If the source did not update, just a forced rebuild to create new binaries, I can see Yocto choose not to rebuild things depend on the package. In the case of boost, since it is devtool modified and the header is a source and not build artifact, it make sense to trigger all the rebuild.

-----Original Message-----
From: openbmc <[email protected]> On Behalf Of Bills, Jason M
Sent: Thursday, July 2, 2020 5:00 PM
To: [email protected]
Subject: Re: Weird build dependency issue causing missing symbols

Quoted 18 lines · Patrick Williams
Quoted 3 lines · Jason M

We have narrowed this down to being caused by two separate issues:
1. When phosphor-dbus-interfaces is rebuilt it will sometimes change the order of the PropertiesVariant in server.hpp.

This sounds like a bug in sdbus++. We should be sorting the variant parameters or issuing them in array order. I'll look into it.

Quoted 3 lines

2. When the order of PropertiesVariant changes on a rebuild, the recipes that already have an old copy of server.hpp are not triggered to rebuild and are left with the old copy of server.hpp.

This isn't surprising if what is triggering the rebuild is not a Yocto variable change (or git revision). Yocto doesn't cache the contents of the packages, but caches the variables that went into a build step. A hash of the variables are used to look up the potential 'sstate-cache' files so that it can skip build steps.

Quoted 7 lines
Quoted 3 lines

If you think a variable or a git-revision should have changed with what you were doing, then maybe it is something else.

It seems like a header file change should trigger a rebuild, though? If I manually edited something like a library header file, I'd expect everything that includes that library to be rebuilt with the new header change.

I tried to devtool modify boost to check the behavior, but that causes boost to rebuild every time and correctly triggers the dependent builds.
Maybe the case above of modifying a header file is invalid?

It doesn't matter what the content is: header, library, executable, data file. Yocto does not use contents in the decision of "does this need to be rebuilt". It only uses variables from recipes. If the variables do not change, the package is not rebuilt (unless explicitly tainted).

See https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#dev-invalidating-shared-state-to-force-a-task-to-run for example.

Signature · 1 line
Patrick Williams
#7 lore raw
Quoted 42 lines · Patrick Williams
Quoted 33 lines · Zhikui

Maybe the problem is that this header server.hpp is generated not a source.
Artifact created from the same source *should* be the same (except timestamp)
If the source did not update, just a forced rebuild to create new binaries, I can see Yocto choose not to rebuild things depend on the package. In the case of boost, since it is devtool modified and the header is a source and not build artifact, it make sense to trigger all the rebuild.

-----Original Message-----
From: openbmc <[email protected]> On Behalf Of Bills, Jason M
Sent: Thursday, July 2, 2020 5:00 PM
To: [email protected]
Subject: Re: Weird build dependency issue causing missing symbols

Quoted 18 lines · Patrick Williams
Quoted 3 lines · Jason M

We have narrowed this down to being caused by two separate issues:
1. When phosphor-dbus-interfaces is rebuilt it will sometimes change the order of the PropertiesVariant in server.hpp.

This sounds like a bug in sdbus++. We should be sorting the variant parameters or issuing them in array order. I'll look into it.

Quoted 3 lines

2. When the order of PropertiesVariant changes on a rebuild, the recipes that already have an old copy of server.hpp are not triggered to rebuild and are left with the old copy of server.hpp.

This isn't surprising if what is triggering the rebuild is not a Yocto variable change (or git revision). Yocto doesn't cache the contents of the packages, but caches the variables that went into a build step. A hash of the variables are used to look up the potential 'sstate-cache' files so that it can skip build steps.

Quoted 7 lines
Quoted 3 lines

If you think a variable or a git-revision should have changed with what you were doing, then maybe it is something else.

It seems like a header file change should trigger a rebuild, though? If I manually edited something like a library header file, I'd expect everything that includes that library to be rebuilt with the new header change.

I tried to devtool modify boost to check the behavior, but that causes boost to rebuild every time and correctly triggers the dependent builds.
Maybe the case above of modifying a header file is invalid?

Quoted 7 lines

It doesn't matter what the content is: header, library, executable, data file. Yocto does not use contents in the decision of "does this need to be rebuilt". It only uses variables from recipes. If the variables do not change, the package is not rebuilt (unless explicitly tainted).

See https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#dev-invalidating-shared-state-to-force-a-task-to-run for example.

Thanks, Patrick! I think I get it. I was stuck on the behavior that I see when I'm working with repo using 'devtool modify'. I guess 'devtool modify' must have some additional magic to force rebuilds on file changes that doesn't apply to a standard build?