Skip to contentSkip to conversation

sstate black hole?

DISCUSSION Gary Thomas 3 participants 7 messages Apr 7, 2015

lore
Share & tools
lore
#0 lore raw

I'm building for multiple ARM i.MX6 platforms. These have the same SoC, but slightly different peripherals. As far as I can tell, they should be able to share everything except for a few ${MACHINE} specific packages, e.g. the kernel and u-boot.

Sadly, that doesn't seem to be the case. The architecture specific packages are being split into two categories - plain ARM/Cortex-A9 and those that have i.MX6 specific optimizations. For example, after building a complete image (on the order of core-image-sato), I have this split:

$ ls tmp/work/cortexa9hf-vfp-neon-amltd-linux-gnueabi/
acl                  gst-player                 libsamplerate0         modutils-initscripts  shadow-sysroot
alsa-utils           gst-plugins-bad            libsm                  mpeg2dec              shared-mime-info
apmd                 gst-plugins-good           libsndfile1            mplayer2              speex
atk                  gst-plugins-ugly           libsoup-2.4            mtdev                 sqlite3
attr                 gstreamer                  libtheora              ncurses               startup-notification
base-passwd          gstreamer1.0               libtirpc               neon                  strace
   ...
gst-ffmpeg           libpostproc                matchbox-wm            scrnsaverproto        zlib
gst-fluendo-mpegmux  libproxy                   mkfontdir              settings-daemon
gst-meta-base        libpthread-stubs           mkfontscale            shadow
$ ls tmp/work/cortexa9hf-vfp-neon-mx6qdl-amltd-linux-gnueabi/
alsa-lib      gst-plugins-base           imx-gpu-viv  libfslparser   libsdl      xf86-video-imxfb-vivante
cairo         gstreamer1.0-plugins-bad   libdrm       libfslvpuwrap  mesa        xserver-xorg
firmware-imx  gstreamer1.0-plugins-base  libfslcodec  libglu         pulseaudio

It's the second category that is causing problems. They do not seem to end up in any shareable sstate at all. If I try to rebuild using only sstate, i.e. build my complete image to success, then remove 'tmp' and rebuild, using the sstate-cache from the first go, all of the above packages (alsa-lib, ..., xserver-xorg) are all rebuilt from scratch. Those recipes do seem to end in my sstate-cache, but they are never reused from it.

What would make this happen? How can I prevent it?

As is, sstate is not really shareable between these i.MX6 targets as so much is being rebuilt all the time...

Any ideas or pointers gladly welcomed.

Thanks

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#1 lore raw
Quoted 48 lines · Gary Thomas

I'm building for multiple ARM i.MX6 platforms. These have the same SoC, but slightly different peripherals. As far as I can tell, they should be able to share everything except for a few ${MACHINE} specific packages, e.g. the kernel and u-boot.

Sadly, that doesn't seem to be the case. The architecture specific packages are being split into two categories - plain ARM/Cortex-A9 and those that have i.MX6 specific optimizations. For example, after building a complete image (on the order of core-image-sato), I have this split:

$ ls tmp/work/cortexa9hf-vfp-neon-amltd-linux-gnueabi/
acl                  gst-player                 libsamplerate0
 modutils-initscripts  shadow-sysroot
alsa-utils           gst-plugins-bad            libsm
mpeg2dec              shared-mime-info
apmd                 gst-plugins-good           libsndfile1
mplayer2              speex
atk                  gst-plugins-ugly           libsoup-2.4
mtdev                 sqlite3
attr                 gstreamer                  libtheora
ncurses               startup-notification
base-passwd          gstreamer1.0               libtirpc
 neon                  strace
   ...
gst-ffmpeg           libpostproc                matchbox-wm
scrnsaverproto        zlib
gst-fluendo-mpegmux  libproxy                   mkfontdir
settings-daemon
gst-meta-base        libpthread-stubs           mkfontscale
shadow
$ ls tmp/work/cortexa9hf-vfp-neon-mx6qdl-amltd-linux-gnueabi/
alsa-lib      gst-plugins-base           imx-gpu-viv  libfslparser
 libsdl      xf86-video-imxfb-vivante
cairo         gstreamer1.0-plugins-bad   libdrm       libfslvpuwrap  mesa
      xserver-xorg
firmware-imx  gstreamer1.0-plugins-base  libfslcodec  libglu
 pulseaudio

It's the second category that is causing problems. They do not seem to end up in any shareable sstate at all. If I try to rebuild using only sstate, i.e. build my complete image to success, then remove 'tmp' and rebuild, using the sstate-cache from the first go, all of the above packages (alsa-lib, ..., xserver-xorg) are all rebuilt from scratch. Those recipes do seem to end in my sstate-cache, but they are never reused from it.

Quoted 5 lines

What would make this happen? How can I prevent it?

As is, sstate is not really shareable between these i.MX6 targets as so much is being rebuilt all the time...

bitbake -S printdiff <target> (e.g. a specific recipe, your image, whatever) will tell you why the sstate wasn't used, by finding the ones that are in SSTATE_DIR which most closely match the current configuration, and displaying a detailed delta between the two. You might need
https://gist.github.com/kergoth/3713d779c14dc8b98f36.

Signature · 5 lines
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
#2 lore raw
Quoted 39 lines · Christopher Larson

On Tue, Apr 7, 2015 at 7:52 AM, Gary Thomas <[email protected] <mailto:[email protected]>> wrote:

I'm building for multiple ARM i.MX6 platforms. These have the same SoC, but slightly different peripherals. As far as I can tell, they should be able to share everything except for a few ${MACHINE} specific packages, e.g. the kernel and u-boot.

Sadly, that doesn't seem to be the case. The architecture specific packages are being split into two categories - plain ARM/Cortex-A9 and those that have i.MX6 specific optimizations. For example, after building a complete image (on the order of core-image-sato), I have this split:
$ ls tmp/work/cortexa9hf-vfp-neon-__amltd-linux-gnueabi/
acl gst-player libsamplerate0 modutils-initscripts shadow-sysroot
alsa-utils gst-plugins-bad libsm mpeg2dec shared-mime-info
apmd gst-plugins-good libsndfile1 mplayer2 speex
atk gst-plugins-ugly libsoup-2.4 mtdev sqlite3
attr gstreamer libtheora ncurses startup-notification
base-passwd gstreamer1.0 libtirpc neon strace
...
gst-ffmpeg libpostproc matchbox-wm scrnsaverproto zlib
gst-fluendo-mpegmux libproxy mkfontdir settings-daemon
gst-meta-base libpthread-stubs mkfontscale shadow

$ ls tmp/work/cortexa9hf-vfp-neon-__mx6qdl-amltd-linux-gnueabi/
alsa-lib gst-plugins-base imx-gpu-viv libfslparser libsdl xf86-video-imxfb-vivante
cairo gstreamer1.0-plugins-bad libdrm libfslvpuwrap mesa xserver-xorg
firmware-imx gstreamer1.0-plugins-base libfslcodec libglu pulseaudio

It's the second category that is causing problems. They do not seem to end up in any shareable sstate at all. If I try to rebuild using only sstate, i.e. build my complete image to success, then remove 'tmp' and rebuild, using the sstate-cache from the first go, all of the above packages (alsa-lib, ..., xserver-xorg) are all rebuilt from scratch. Those recipes do seem to end in my sstate-cache, but they are never reused from it.

Quoted 8 lines

What would make this happen? How can I prevent it?

As is, sstate is not really shareable between these i.MX6 targets as so much is being rebuilt all the time...

bitbake -S printdiff <target> (e.g. a specific recipe, your image, whatever) will tell you why the sstate wasn't used, by finding the ones that are in SSTATE_DIR which most closely match the current configuration, and displaying a detailed delta between the two. You might need https://gist.github.com/kergoth/3713d779c14dc8b98f36.

Sorry, that didn't seem to do anything (I didn't apply your patch since I'm not looking at any native or sdk packages).

Here's what I got:

  $ bitbake -S printdiffs xserver-xorg
  ...
  NOTE: Preparing RunQueue
  NOTE: Reparsing files to collect dependency data
  Writing locked sigs to /local/imx6_2015-03-26/locked-sigs.inc
  NOTE: Tasks Summary: Attempted 0 tasks of which 0 didn't need to be rerun and all succeeded.

No other output.

I can see [manually] that there are a number of cache entries, e.g.

$ find sstate-cache/ -name "*xserver-xorg*package.tgz"
sstate-cache/48/sstate:xserver-xorg:cortexa9hf-vfp-neon-mx6qdl-amltd-linux-gnueabi:1.16.3:r0:cortexa9hf-vfp-neon-mx6qdl:3:487ec958840dd9ed48ff3da86e2dabdb_package.tgz
sstate-cache/4f/sstate:xserver-xorg:cortexa9hf-vfp-neon-mx6qdl-amltd-linux-gnueabi:1.16.3:r0:cortexa9hf-vfp-neon-mx6qdl:3:4f7a493630c7f29762714430ea6be4d1_package.tgz
sstate-cache/80/sstate:xserver-xorg:cortexa9hf-vfp-neon-mx6qdl-amltd-linux-gnueabi:1.16.3:r0:cortexa9hf-vfp-neon-mx6qdl:3:8041b9f7bda788038e011829939e2049_package.tgz
sstate-cache/2d/sstate:xserver-xorg:cortexa9hf-vfp-neon-mx6qdl-amltd-linux-gnueabi:1.16.3:r0:cortexa9hf-vfp-neon-mx6qdl:3:2de173b1c4875889f124ce4560f42ed7_package.tgz
sstate-cache/b4/sstate:xserver-xorg:cortexa9hf-vfp-neon-mx6qdl-amltd-linux-gnueabi:1.16.3:r0:cortexa9hf-vfp-neon-mx6qdl:3:b492f53c05ca6cbeb51b2e574dd60495_package.tgz

So I would think that something would be printed. Do I need to do something else to make this work, e.g. cleaning the target package, etc?

Note: I looked at the 'locked-sigs.inc' that was generated and I noticed that there is no section for the recipes I'm interested in:

  $ grep ^SIG locked-sigs.inc
  SIGGEN_LOCKEDSIGS_t-imx6qsabresd = "\
  SIGGEN_LOCKEDSIGS_t-x86-64 = "\
  SIGGEN_LOCKEDSIGS_t-x86-64-arm = "\
  SIGGEN_LOCKEDSIGS_t-cortexa9hf-vfp-neon = "\
  SIGGEN_LOCKEDSIGS_TYPES_imx6qsabresd = "t-imx6qsabresd t-x86-64 t-x86-64-arm t-cortexa9hf-vfp-neon"

I would think there would need to be a "SIGGEN_LOCKEDSIGS_t-cortexa9hf-vfp-neon-mx6qdl" section?

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#3 lore raw
In reply to Gary Thomas
Quoted 36 lines · Gary Thomas

I'm building for multiple ARM i.MX6 platforms. These have the same SoC, but slightly different peripherals. As far as I can tell, they should be able to share everything except for a few ${MACHINE} specific packages, e.g. the kernel and u-boot.

Sadly, that doesn't seem to be the case. The architecture specific packages are being split into two categories - plain ARM/Cortex-A9 and those that have i.MX6 specific optimizations. For example, after building a complete image (on the order of core-image-sato), I have this split:

$ ls tmp/work/cortexa9hf-vfp-neon-amltd-linux-gnueabi/
acl                  gst-player                 libsamplerate0         modutils-initscripts  shadow-sysroot
alsa-utils           gst-plugins-bad            libsm                  mpeg2dec              shared-mime-info
apmd                 gst-plugins-good           libsndfile1            mplayer2              speex
atk                  gst-plugins-ugly           libsoup-2.4            mtdev                 sqlite3
attr                 gstreamer                  libtheora              ncurses               startup-notification
base-passwd          gstreamer1.0               libtirpc               neon                  strace
    ...
gst-ffmpeg           libpostproc                matchbox-wm            scrnsaverproto        zlib
gst-fluendo-mpegmux  libproxy                   mkfontdir              settings-daemon
gst-meta-base        libpthread-stubs           mkfontscale            shadow
$ ls tmp/work/cortexa9hf-vfp-neon-mx6qdl-amltd-linux-gnueabi/
alsa-lib      gst-plugins-base           imx-gpu-viv  libfslparser   libsdl      xf86-video-imxfb-vivante
cairo         gstreamer1.0-plugins-bad   libdrm       libfslvpuwrap  mesa        xserver-xorg
firmware-imx  gstreamer1.0-plugins-base  libfslcodec  libglu         pulseaudio

It's the second category that is causing problems. They do not seem to end up in any shareable sstate at all. If I try to rebuild using only sstate, i.e. build my complete image to success, then remove 'tmp' and rebuild, using the sstate-cache from the first go, all of the above packages (alsa-lib, ..., xserver-xorg) are all rebuilt from scratch. Those recipes do seem to end in my sstate-cache, but they are never reused from it.

Quoted 6 lines

What would make this happen? How can I prevent it?

As is, sstate is not really shareable between these i.MX6 targets as so much is being rebuilt all the time...

Any ideas or pointers gladly welcomed.

Try openembedded-core/scripts/sstate-diff-machines.sh to see why.

Signature · 1 line
Martin 'JaMa' Jansa     jabber: [email protected]
#4 lore raw
Quoted 39 lines · Martin Jansa
Quoted 38 lines · Gary Thomas

I'm building for multiple ARM i.MX6 platforms. These have the same SoC, but slightly different peripherals. As far as I can tell, they should be able to share everything except for a few ${MACHINE} specific packages, e.g. the kernel and u-boot.

Sadly, that doesn't seem to be the case. The architecture specific packages are being split into two categories - plain ARM/Cortex-A9 and those that have i.MX6 specific optimizations. For example, after building a complete image (on the order of core-image-sato), I have this split:

$ ls tmp/work/cortexa9hf-vfp-neon-amltd-linux-gnueabi/
acl                  gst-player                 libsamplerate0         modutils-initscripts  shadow-sysroot
alsa-utils           gst-plugins-bad            libsm                  mpeg2dec              shared-mime-info
apmd                 gst-plugins-good           libsndfile1            mplayer2              speex
atk                  gst-plugins-ugly           libsoup-2.4            mtdev                 sqlite3
attr                 gstreamer                  libtheora              ncurses               startup-notification
base-passwd          gstreamer1.0               libtirpc               neon                  strace
     ...
gst-ffmpeg           libpostproc                matchbox-wm            scrnsaverproto        zlib
gst-fluendo-mpegmux  libproxy                   mkfontdir              settings-daemon
gst-meta-base        libpthread-stubs           mkfontscale            shadow
$ ls tmp/work/cortexa9hf-vfp-neon-mx6qdl-amltd-linux-gnueabi/
alsa-lib      gst-plugins-base           imx-gpu-viv  libfslparser   libsdl      xf86-video-imxfb-vivante
cairo         gstreamer1.0-plugins-bad   libdrm       libfslvpuwrap  mesa        xserver-xorg
firmware-imx  gstreamer1.0-plugins-base  libfslcodec  libglu         pulseaudio

It's the second category that is causing problems. They do not seem to end up in any shareable sstate at all. If I try to rebuild using only sstate, i.e. build my complete image to success, then remove 'tmp' and rebuild, using the sstate-cache from the first go, all of the above packages (alsa-lib, ..., xserver-xorg) are all rebuilt from scratch. Those recipes do seem to end in my sstate-cache, but they are never reused from it.

What would make this happen? How can I prevent it?

Quoted 8 lines
Quoted 4 lines

As is, sstate is not really shareable between these i.MX6 targets as so much is being rebuilt all the time...

Any ideas or pointers gladly welcomed.

Try openembedded-core/scripts/sstate-diff-machines.sh to see why.

Can this work if I build for the two machines in separate trees? Also, I'm really trying to find out why a build for the same machine doesn't reuse sstate, in the same build tree, back-to-back builds (no metadata changes)

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#5 lore raw
Quoted 43 lines · Gary Thomas
Quoted 42 lines · Martin Jansa
Quoted 41 lines · Gary Thomas

I'm building for multiple ARM i.MX6 platforms. These have the same SoC, but slightly different peripherals. As far as I can tell, they should be able to share everything except for a few ${MACHINE} specific packages, e.g. the kernel and u-boot.

Sadly, that doesn't seem to be the case. The architecture specific packages are being split into two categories - plain ARM/Cortex-A9 and those that have i.MX6 specific optimizations. For example, after building a complete image (on the order of core-image-sato), I have this split:

$ ls tmp/work/cortexa9hf-vfp-neon-amltd-linux-gnueabi/
acl                  gst-player                 libsamplerate0         modutils-initscripts  shadow-sysroot
alsa-utils           gst-plugins-bad            libsm                  mpeg2dec              shared-mime-info
apmd                 gst-plugins-good           libsndfile1            mplayer2              speex
atk                  gst-plugins-ugly           libsoup-2.4            mtdev                 sqlite3
attr                 gstreamer                  libtheora              ncurses               startup-notification
base-passwd          gstreamer1.0               libtirpc               neon                  strace
     ...
gst-ffmpeg           libpostproc                matchbox-wm            scrnsaverproto        zlib
gst-fluendo-mpegmux  libproxy                   mkfontdir              settings-daemon
gst-meta-base        libpthread-stubs           mkfontscale            shadow
$ ls tmp/work/cortexa9hf-vfp-neon-mx6qdl-amltd-linux-gnueabi/
alsa-lib      gst-plugins-base           imx-gpu-viv  libfslparser   libsdl      xf86-video-imxfb-vivante
cairo         gstreamer1.0-plugins-bad   libdrm       libfslvpuwrap  mesa        xserver-xorg
firmware-imx  gstreamer1.0-plugins-base  libfslcodec  libglu         pulseaudio

It's the second category that is causing problems. They do not seem to end up in any shareable sstate at all. If I try to rebuild using only sstate, i.e. build my complete image to success, then remove 'tmp' and rebuild, using the sstate-cache from the first go, all of the above packages (alsa-lib, ..., xserver-xorg) are all rebuilt from scratch. Those recipes do seem to end in my sstate-cache, but they are never reused from it.

What would make this happen? How can I prevent it?

As is, sstate is not really shareable between these i.MX6 targets as so much is being rebuilt all the time...

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

Any ideas or pointers gladly welcomed.

Try openembedded-core/scripts/sstate-diff-machines.sh to see why.

Can this work if I build for the two machines in separate trees?

Yes, you can generate the report in each tree separately and then compare them in one of them.

Quoted 3 lines

Also, I'm really trying to find out why a build for the same machine doesn't reuse sstate, in the same build tree, back-to-back builds (no metadata changes)

Sorry, is "the same build tree" something else than "the two machines in separate trees" or are you testing both?

Signature · 1 line
Martin 'JaMa' Jansa     jabber: [email protected]
#6 lore raw
Quoted 55 lines · Martin Jansa
Quoted 43 lines · Gary Thomas
Quoted 42 lines · Martin Jansa
Quoted 41 lines · Gary Thomas

I'm building for multiple ARM i.MX6 platforms. These have the same SoC, but slightly different peripherals. As far as I can tell, they should be able to share everything except for a few ${MACHINE} specific packages, e.g. the kernel and u-boot.

Sadly, that doesn't seem to be the case. The architecture specific packages are being split into two categories - plain ARM/Cortex-A9 and those that have i.MX6 specific optimizations. For example, after building a complete image (on the order of core-image-sato), I have this split:

$ ls tmp/work/cortexa9hf-vfp-neon-amltd-linux-gnueabi/
acl                  gst-player                 libsamplerate0         modutils-initscripts  shadow-sysroot
alsa-utils           gst-plugins-bad            libsm                  mpeg2dec              shared-mime-info
apmd                 gst-plugins-good           libsndfile1            mplayer2              speex
atk                  gst-plugins-ugly           libsoup-2.4            mtdev                 sqlite3
attr                 gstreamer                  libtheora              ncurses               startup-notification
base-passwd          gstreamer1.0               libtirpc               neon                  strace
      ...
gst-ffmpeg           libpostproc                matchbox-wm            scrnsaverproto        zlib
gst-fluendo-mpegmux  libproxy                   mkfontdir              settings-daemon
gst-meta-base        libpthread-stubs           mkfontscale            shadow
$ ls tmp/work/cortexa9hf-vfp-neon-mx6qdl-amltd-linux-gnueabi/
alsa-lib      gst-plugins-base           imx-gpu-viv  libfslparser   libsdl      xf86-video-imxfb-vivante
cairo         gstreamer1.0-plugins-bad   libdrm       libfslvpuwrap  mesa        xserver-xorg
firmware-imx  gstreamer1.0-plugins-base  libfslcodec  libglu         pulseaudio

It's the second category that is causing problems. They do not seem to end up in any shareable sstate at all. If I try to rebuild using only sstate, i.e. build my complete image to success, then remove 'tmp' and rebuild, using the sstate-cache from the first go, all of the above packages (alsa-lib, ..., xserver-xorg) are all rebuilt from scratch. Those recipes do seem to end in my sstate-cache, but they are never reused from it.

What would make this happen? How can I prevent it?

As is, sstate is not really shareable between these i.MX6 targets as so much is being rebuilt all the time...

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

Any ideas or pointers gladly welcomed.

Try openembedded-core/scripts/sstate-diff-machines.sh to see why.

Can this work if I build for the two machines in separate trees?

Yes, you can generate the report in each tree separately and then compare them in one of them.

Quoted 7 lines
Quoted 3 lines

Also, I'm really trying to find out why a build for the same machine doesn't reuse sstate, in the same build tree, back-to-back builds (no metadata changes)

Sorry, is "the same build tree" something else than "the two machines in separate trees" or are you testing both?

Both actually, but at the moment the most interesting problem is for a given machine that sstate is not being [re]used for these recipes.

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------