Skip to contentSkip to conversation

Question about rebuilding RPM package index for updated RPMs when bitbake completes

DISCUSSION Alex J Lennon 2 participants 5 messages Apr 20, 2014 – Apr 22, 2014

lore
Share & tools
lore
#0 lore raw

Hi,

I'm trying to put in place a development workflow using the PR server, RPM package feeds and smart update/install on a target.

I see that when I modify and rebuild my recipe, foo, the PR server increments its count within the RPM filename, but the RPM feed data doesn't seem to update.

e.g. in my deploy/rpm/cortexa9_vfp_neon folder I see a change (say) from

foo-1.7.11-r0.1.cortexa9_vfp_neon.rpm

to

foo-1.7.11-r0.2.cortexa9_vfp_neon.rpm

However if I then run smart on the target (configured to use that rpm package feed) I don't see new packages

smart update

Similarly if I remove the package and reinstall it I get an error

smart remove foo
smart install foo

Fetching packages...
-> http://pkgrepo.my.local/.../foo-1.7.11-r0.1.cortexa9_vfp_neon.rpm
foo-1.7.11-r0.1... [ 0%]

error: Failed to download packages:
error:
http://pkgrepo.my.local/cortexa9_vfp_neon/foo-1.7.11-r0.1.cortexa9_vfp_neon.rpm:
File not found

I believe this is because the old package has been removed, and the new package added to the folder,
but the repodata has not been rebuilt, leaving the packages out of sync with the feed representation.

If then try running

bitbake package-index

When I now run an update on the target it picks up the new package and I can fetch and install this

smart update

Updating cache... ########################################
[100%]

Channels have 3 new packages:

    foo-1.7.11-r0.3@cortexa9_vfp_neon

...

So if I am understanding the above correctly, when I make a change to a recipe and build it, PR automatically
updates, the old RPM is removed and the new RPM added to the feed directory. However the package index
for the feed is not updated.

So if at that point I try to make use of the feed on a target I am likely to find something is broken.

If that is true would it make sense to leave the old RPM in the feed directory until package-index
is re-ran, or to run package-index automatically at the end of a build when RPMs have changed?

Thanks,

Alex

#1 lore raw
Quoted 9 lines · Alex J Lennon

Hi,

I'm trying to put in place a development workflow using the PR server, RPM package feeds and smart update/install on a target.

I see that when I modify and rebuild my recipe, foo, the PR server increments its count within the RPM filename, but the RPM feed data doesn't seem to update.

...

Quoted 8 lines

So if I am understanding the above correctly, when I make a change to a recipe and build it, PR automatically
updates, the old RPM is removed and the new RPM added to the feed directory. However the package index
for the feed is not updated.

So if at that point I try to make use of the feed on a target I am likely to find something is broken.

The feed is normally indexed (createrepo) either when you manually run the package-index operation, or when you construct a filesystem. Until you do that, the feed directories are transient.

Quoted 4 lines

If that is true would it make sense to leave the old RPM in the feed directory until package-index
is re-ran, or to run package-index automatically at the end of a build when RPMs have changed?

I -never- export the feed directories from the project directory. Instead, I copy the packages from the feed directory to where I share them, and then run the indexer against the external repository. This preserves the older versions and also makes the new ones available.

To run the indexer I have to configure and run it manually...

PATH=/home/mhatle/build-6.0-RCPL-test/build/tmp/sysroots/x86_64-linux/bin:/home/mhatle/build-6.0-RCPL-test/bitbake/tmp/sysroots/x86_64-linux/usr/bin:$PATH /home/mhatle/build-6.0-RCPL-test/bitbake/tmp/sysroots/x86_64-linux/usr/bin/createrepo --update -q <path to feed>

So for qemux86_64, I end up running the above three times. all, x86_64 and qemux86_64.

Then on the target I just do:

smart update
smart upgrade -y

--Mark

Quoted 6 lines

Thanks,

Alex

#2 lore raw
Quoted 19 lines · Mark Hatle
Quoted 9 lines · Alex J Lennon

Hi,

I'm trying to put in place a development workflow using the PR server, RPM package feeds and smart update/install on a target.

I see that when I modify and rebuild my recipe, foo, the PR server increments its count within the RPM filename, but the RPM feed data doesn't seem to update.

...

Quoted 6 lines

So if I am understanding the above correctly, when I make a change to a recipe and build it, PR automatically
updates, the old RPM is removed and the new RPM added to the feed directory. However the package index
for the feed is not updated.

Quoted 7 lines
Quoted 2 lines

So if at that point I try to make use of the feed on a target I am likely to find something is broken.

The feed is normally indexed (createrepo) either when you manually run the package-index operation, or when you construct a filesystem. Until you do that, the feed directories are transient.

Is there a reason for this? Surely having a package index that is out of sync with the packages until a manual operation is performed isn't ideal?

Quoted 17 lines
Quoted 4 lines

If that is true would it make sense to leave the old RPM in the feed directory until package-index
is re-ran, or to run package-index automatically at the end of a build when RPMs have changed?

I -never- export the feed directories from the project directory. Instead, I copy the packages from the feed directory to where I share them, and then run the indexer against the external repository. This preserves the older versions and also makes the new ones available.

To run the indexer I have to configure and run it manually...

PATH=/home/mhatle/build-6.0-RCPL-test/build/tmp/sysroots/x86_64-linux/bin:/home/mhatle/build-6.0-RCPL-test/bitbake/tmp/sysroots/x86_64-linux/usr/bin:$PATH

/home/mhatle/build-6.0-RCPL-test/bitbake/tmp/sysroots/x86_64-linux/usr/bin/createrepo --update -q <path to feed>

Quoted 8 lines

So for qemux86_64, I end up running the above three times. all, x86_64 and qemux86_64.

Then on the target I just do:

smart update
smart upgrade -y

Thanks, that's useful

However I still believe the core question stands, which is why the package index is out of sync, and why it's a good thing to have to run bitbake package-index manually to bring the package index back into sync with the packages?

I too export the feed directories to a server for 3rd party consumption via compressed rsync over SSH. I don't want to expose more on the server than a simple SSH endpoint, and rsync seems a sensible way to minimise copying times, so either I need the package index to be correct prior to the rsync, I need to export, say, an NFS filesystem which I don't want to do, or I need some createrepo specifics on what I would prefer to be a dumb web-server.

Best,

Alex

Quoted 9 lines

--Mark

Quoted 6 lines

Thanks,

Alex

Signature · 22 lines
Dynamic Devices Ltd <http://www.dynamicdevices.co.uk/>

Alex J Lennon / Director
1 Queensway, Liverpool L22 4RA

mobile: +44 (0)7956 668178

Linkedin <http://www.linkedin.com/in/alexjlennon> Skype
<skype:alexjlennon?add>

This e-mail message may contain confidential or legally privileged
information and is intended only for the use of the intended
recipient(s). Any unauthorized disclosure, dissemination, distribution,
copying or the taking of any action in reliance on the information
herein is prohibited. E-mails are not secure and cannot be guaranteed to
be error free as they can be intercepted, amended, or contain viruses.
Anyone who communicates with us by e-mail is deemed to have accepted
these risks. Company Name is not responsible for errors or omissions in
this message and denies any responsibility for any damage arising from
the use of e-mail. Any opinion and other statement contained in this
message and any attachment are solely those of the author and do not
necessarily represent those of the company.
#3 lore raw
Quoted 24 lines · Alex J Lennon
Quoted 22 lines · Mark Hatle
Quoted 9 lines · Alex J Lennon

Hi,

I'm trying to put in place a development workflow using the PR server, RPM package feeds and smart update/install on a target.

I see that when I modify and rebuild my recipe, foo, the PR server increments its count within the RPM filename, but the RPM feed data doesn't seem to update.

...

Quoted 8 lines

So if I am understanding the above correctly, when I make a change to a recipe and build it, PR automatically
updates, the old RPM is removed and the new RPM added to the feed directory. However the package index
for the feed is not updated.

So if at that point I try to make use of the feed on a target I am likely to find something is broken.

Quoted 7 lines
Quoted 4 lines

The feed is normally indexed (createrepo) either when you manually run the package-index operation, or when you construct a filesystem. Until you do that, the feed directories are transient.

Is there a reason for this? Surely having a package index that is out of sync with the packages until a manual operation is performed isn't ideal?

Yes, if you re-indexed the set of packages after each package, you will add multiple minutes to the construction of each package, and introduce locking into the system. (You won't be able to generate multiple recipes in parallel due to this.) (I don't consider it a package feed until all of the packages are built and the index is constructed.)

Quoted 22 lines
Quoted 22 lines
Quoted 4 lines

If that is true would it make sense to leave the old RPM in the feed directory until package-index
is re-ran, or to run package-index automatically at the end of a build when RPMs have changed?

I -never- export the feed directories from the project directory. Instead, I copy the packages from the feed directory to where I share them, and then run the indexer against the external repository. This preserves the older versions and also makes the new ones available.

To run the indexer I have to configure and run it manually...

PATH=/home/mhatle/build-6.0-RCPL-test/build/tmp/sysroots/x86_64-linux/bin:/home/mhatle/build-6.0-RCPL-test/bitbake/tmp/sysroots/x86_64-linux/usr/bin:$PATH

/home/mhatle/build-6.0-RCPL-test/bitbake/tmp/sysroots/x86_64-linux/usr/bin/createrepo --update -q <path to feed>

So for qemux86_64, I end up running the above three times. all, x86_64 and qemux86_64.

Then on the target I just do:

Quoted 8 lines
Quoted 3 lines

smart update
smart upgrade -y

Thanks, that's useful

However I still believe the core question stands, which is why the package index is out of sync, and why it's a good thing to have to run bitbake package-index

The other option is to remove the repo indexes when packages are written. Again, I don't consider these out of sync though, as the files have different purposes and constraints. They are simply constructed to enable the rootfs generation. Upgrades are external of the build system.

Quoted 8 lines

manually to bring the package index back into sync with the packages?

I too export the feed directories to a server for 3rd party consumption via compressed rsync over SSH. I don't want to expose more on the server than a simple SSH endpoint, and rsync seems a sensible way to minimise copying times, so either I need the package index to be correct prior to the rsync, I need to export, say, an NFS filesystem which I don't want to do, or I need some createrepo specifics on what I would prefer to be a dumb web-server.

For an rsync, I would either index on the server using a cron-job.. so when new packages appear they get added to the index, or have a local mirror of the directory including the updated index... you just want to make sure the index files are synced last somehow.

--Mark

Quoted 37 lines

Best,

Alex

Quoted 9 lines

--Mark

Quoted 6 lines

Thanks,

Alex

--

Dynamic Devices Ltd <http://www.dynamicdevices.co.uk/>

Alex J Lennon / Director
1 Queensway, Liverpool L22 4RA

mobile: +44 (0)7956 668178

Linkedin <http://www.linkedin.com/in/alexjlennon> Skype <skype:alexjlennon?add>

This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorized disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Company Name is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company.

#4 lore raw
Quoted 37 lines · Mark Hatle
Quoted 32 lines · Alex J Lennon
Quoted 20 lines · Mark Hatle
Quoted 9 lines · Alex J Lennon

Hi,

I'm trying to put in place a development workflow using the PR server, RPM package feeds and smart update/install on a target.

I see that when I modify and rebuild my recipe, foo, the PR server increments its count within the RPM filename, but the RPM feed data doesn't seem to update.

...

Quoted 7 lines

So if I am understanding the above correctly, when I make a change to a
recipe and build it, PR automatically
updates, the old RPM is removed and the new RPM added to the feed directory. However the package index
for the feed is not updated.

Quoted 9 lines
Quoted 2 lines

So if at that point I try to make use of the feed on a target I am likely to find something is broken.

The feed is normally indexed (createrepo) either when you manually run the
package-index operation, or when you construct a filesystem. Until you do
that, the feed directories are transient.

Quoted 3 lines

Is there a reason for this? Surely having a package index that is out of sync
with the packages until a manual operation is performed isn't ideal?

Quoted 7 lines

Yes, if you re-indexed the set of packages after each package, you will add multiple minutes to the construction of each package, and introduce locking into the system. (You won't be able to generate multiple recipes in parallel due to this.) (I don't consider it a package feed until all of the packages are built and the index is constructed.)

I wasn't suggesting that the index is built mid-run . Not knowing the internals of the bitbake dependency chain, presumably the package index would/could be rebuilt at the end of the build process and so would not block parallelisation (which I agree would be a time-sink)

Quoted 38 lines
Quoted 31 lines
Quoted 21 lines
Quoted 4 lines

If that is true would it make sense to leave the old RPM in the feed directory until package-index
is re-ran, or to run package-index automatically at the end of a build when RPMs have changed?

I -never- export the feed directories from the project directory. Instead, I
copy the packages from the feed directory to where I share them, and then run
the indexer against the external repository. This preserves the older versions and also makes the new ones available.

To run the indexer I have to configure and run it manually...

PATH=/home/mhatle/build-6.0-RCPL-test/build/tmp/sysroots/x86_64-linux/bin:/home/mhatle/build-6.0-RCPL-test/bitbake/tmp/sysroots/x86_64-linux/usr/bin:$PATH

/home/mhatle/build-6.0-RCPL-test/bitbake/tmp/sysroots/x86_64-linux/usr/bin/createrepo

--update -q <path to feed>

Quoted 9 lines

So for qemux86_64, I end up running the above three times. all, x86_64 and
qemux86_64.

Then on the target I just do:

smart update
smart upgrade -y

Quoted 6 lines

Thanks, that's useful

However I still believe the core question stands, which is why the package index
is out of sync, and why it's a good thing to have to run bitbake package-index

Quoted 6 lines

The other option is to remove the repo indexes when packages are written. Again, I don't consider these out of sync though, as the files have different purposes and constraints. They are simply constructed to enable the rootfs generation. Upgrades are external of the build system.

Fair comment. I take the view that these feeds are available, could easily be relied upon not to be transient, and I'd like to use them locally to speed up my development workflow. I think other users would too.

Much of the value of the build environment is the automation. If bitbake can do a task that means I don't have to remember to do it, and I don't have to explain to somebody else that they need to do it, then that seems like a win to me (even if that is an option I can turn on).

Quoted 13 lines
Quoted 12 lines

manually to bring the package index back into sync with the packages?

I too export the feed directories to a server for 3rd party consumption via
compressed rsync over SSH. I don't want to expose more on the server than a
simple SSH endpoint, and rsync seems a sensible way to minimise copying times,
so either I need the package index to be correct prior to the rsync, I need to
export, say, an NFS filesystem which I don't want to do, or I need some createrepo specifics on what I would prefer to be a dumb web-server.

Quoted 5 lines

For an rsync, I would either index on the server using a cron-job.. so when new packages appear they get added to the index, or have a local mirror of the directory including the updated index... you just want to make sure the index files are synced last somehow.

To me the server side customisation brings in in extra work that shouldn't be needed in all cases, (although will undoubtely be needed in some cases).

I would much prefer not to have to worry about it.

If the package feed could be relied upon to be in sync with the contents of the tmp/deploy/rpm/* tree at the end of a bitbake build then the problem simply goes away, potentially at the cost of a little lost time during the build.

Understanding that other use-cases will be different from mine I have been wondering today,

- Can I add something into local.conf that would add a dependency to cause the package-index task to run at the end of the bitbake build (preferably only if the contents of the package directories have changed?

- Does the build environment have the concept of post-build triggers? i.e. Understanding that I could script something myself, is there a recommended
method for defining a post-build trigger script for bitbake to run on
certain conditions e.g to do the the package reindex and feed upload in this case?

Thanks,

Alex

Quoted 53 lines

--Mark

Quoted 48 lines

Best,

Alex

Quoted 9 lines

--Mark

Quoted 6 lines

Thanks,

Alex

Signature · 31 lines
Dynamic Devices Ltd <http://www.dynamicdevices.co.uk/>

Alex J Lennon / Director
1 Queensway, Liverpool L22 4RA

mobile: +44 (0)7956 668178

Linkedin <http://www.linkedin.com/in/alexjlennon> Skype
<skype:alexjlennon?add>

This e-mail message may contain confidential or legally privileged
information
and is intended only for the use of the intended recipient(s). Any
unauthorized
disclosure, dissemination, distribution, copying or the taking of any
action in
reliance on the information herein is prohibited. E-mails are not
secure and
cannot be guaranteed to be error free as they can be intercepted,
amended, or
contain viruses. Anyone who communicates with us by e-mail is deemed
to have
accepted these risks. Company Name is not responsible for errors or
omissions in
this message and denies any responsibility for any damage arising
from the use
of e-mail. Any opinion and other statement contained in this message
and any
attachment are solely those of the author and do not necessarily
represent those
of the company.
Signature · 22 lines
Dynamic Devices Ltd <http://www.dynamicdevices.co.uk/>

Alex J Lennon / Director
1 Queensway, Liverpool L22 4RA

mobile: +44 (0)7956 668178

Linkedin <http://www.linkedin.com/in/alexjlennon> Skype
<skype:alexjlennon?add>

This e-mail message may contain confidential or legally privileged
information and is intended only for the use of the intended
recipient(s). Any unauthorized disclosure, dissemination, distribution,
copying or the taking of any action in reliance on the information
herein is prohibited. E-mails are not secure and cannot be guaranteed to
be error free as they can be intercepted, amended, or contain viruses.
Anyone who communicates with us by e-mail is deemed to have accepted
these risks. Company Name is not responsible for errors or omissions in
this message and denies any responsibility for any damage arising from
the use of e-mail. Any opinion and other statement contained in this
message and any attachment are solely those of the author and do not
necessarily represent those of the company.