Skip to contentSkip to conversation

scripts/contrib/make_ipk_repo.sh: Add new image-specific package feed generator

PATCH In review Bryan Evenson 3 participants 4 messages Aug 11, 2014 – Aug 12, 2014

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 scripts/contrib/make_ipk_repo.sh: Add new image-specific package feed generator

Adding a new script which generates a reduced package feed directory. The script uses the list of installed packages for a specific build image and creates a new package feed directory with only the packages relevant to that image. This is useful for distributing firmware upgrades.

At this time the script only works for opkg repositories. Further modifications are needed for the script to work with other package management systems.

---
 scripts/contrib/make_ipk_repo.sh |   93 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 93 insertions(+)
 create mode 100755 scripts/contrib/make_ipk_repo.sh
scripts/contrib/make_ipk_repo.sh new file +93 -0
new file mode 100755
index 0000000..b04535e
@@ -0,0 +1,93 @@
+#!/bin/bash
+
+# Creates a package feed directory that only contains the packages that were
+# installed with the referenced image.  This script depends on image build
+# history to properly operate.  In your conf/local.conf, add the following two
+# lines:
+#   INHERIT += "buildhistory"
+#   BUILDHISTORY_COMMIT = "1"
+#
+# At this time this script is only setup to create opkg package feeds.
+# Further script modification is needed if you use a different package
+# managment system.
+#
+# This script assumes it is being run from your image build directory.  Some
+# paths may need to be modified if it is run from a different directory.
+#
+# Assumes that the relevant image was recently built.  If you have done any
+# package builds since the last image build, it is suggested that you re-build
+# relevant image.
+#
+# Script input variables:
+# $1 - The image name (i.e. "core-image-minimal")
+# $2 - The target machine (i.e. "beaglebone")
+# $3 - The target architecture (i.e. "arm926ejste")
+# $4 - The target directory (i.e. "feed")
+#
+
+# Validate the number of input variables
+if [[ "$#" -ne 4 ]]; then
+    echo "Unexpected number of arguments";
+    echo "Usage: make_ipk_repo.sh IMAGE_NAME TARGET_MACHINE TARGET_ARCH TARGET_DIR";
+    exit 1;
+fi
+
+# Some packages inexplicably contain a "1:" in the package name that does not
+# belong there.  Remove this text from these package names.
+
+sed -i 's/1://g' tmp/buildhistory/images/"$2"/eglibc/"$1"/installed-packages.txt
+
+# Read in the list of packages used by the image.  This includes the
+# machine/architecture independent packages ("all"), the architecture specific
+# packages, and the machine specific packages.
+ALL_PKG_LIST=$(grep '_all\.ipk' tmp/buildhistory/images/"$2"/eglibc/"$1"/installed-packages.txt)
+ARCH_PKG_LIST=$(grep '_'"$3"'\.ipk' tmp/buildhistory/images/"$2"/eglibc/"$1"/installed-packages.txt)
+MACHINE_PKG_LIST=$(grep '_'"$2"'\.ipk' tmp/buildhistory/images/"$2"/eglibc/"$1"/installed-packages.txt)
+
+echo "ALL_PKG_LIST is $ALL_PKG_LIST";
+echo "ARCH_PKG_LIST is $ARCH_PKG_LIST";
+echo "MACHINE_PKG_LIST is $MACHINE_PKG_LIST";
+
+# Create the new directory for the feeds, making sure there is no previous copy
+rm -r "$4"/
+rm "$4".zip
+mkdir -p "$4"/
+
+# Copy over the "all" packages
+mkdir -p "$4"/all/
+echo "Copying packages to 'all' directory"
+for i in $ALL_PKG_LIST; do
+    cp tmp/deploy/ipk/all/"$i" "$4"/all/
+done
+
+# Copy over the architecture packages
+mkdir -p "$4"/$3/
+echo "Copying packages to '$3' directory"
+for i in $ARCH_PKG_LIST; do
+    cp tmp/deploy/ipk/"$3"/"$i" "$4"/"$3"/
+done
+
+# Copy over the machine packages
+mkdir -p "$4"/"$2"/
+echo "Copying packages to '$2' directory"
+for i in $MACHINE_PKG_LIST; do
+    cp tmp/deploy/ipk/"$2"/"$i" "$4"/"$2"/
+done
+
+# Build the package index files
+echo "Building package index files"
+touch "$4"/Packages
+flock "$4"/Packages.flock -c "tmp/sysroots/i686-linux/usr/bin/opkg-make-index -r $4/Packages -p $4/Packages -m $4/"
+touch "$4"/all/Packages
+flock "$4"/all/Packages.flock -c "tmp/sysroots/i686-linux/usr/bin/opkg-make-index -r $4/all/Packages -p $4/all/Packages -m $4/all/"
+touch "$4"/"$3"/Packages
+flock "$4"/"$3"/Packages.flock -c "tmp/sysroots/i686-linux/usr/bin/opkg-make-index -r $4/$3/Packages -p $4/$3/Packages -m $4/$3/"
+touch "$4"/"$2"/Packages
+flock "$4"/"$2"/Packages.flock -c "tmp/sysroots/i686-linux/usr/bin/opkg-make-index -r $4/$2/Packages -p $4/$2/Packages -m $4/$2/"
+
+#Create a zip file of the directory for archive purposes
+cd "$4"/
+zip -r "$4" .
+
+echo "A reduced package repository has been created at $4"
+
Signature · 1 line
1.7.9.5
#1 lore raw
Quoted 9 lines · Bryan Evenson

Adding a new script which generates a reduced package feed directory. The script uses the list of installed packages for a specific build image and creates a new package feed directory with only the packages relevant to that image. This is useful for distributing firmware upgrades.

At this time the script only works for opkg repositories. Further modifications are needed for the script to work with other package management systems.

This belongs to oe-core ML

Quoted 114 lines
---
 scripts/contrib/make_ipk_repo.sh |   93 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 93 insertions(+)
 create mode 100755 scripts/contrib/make_ipk_repo.sh
scripts/contrib/make_ipk_repo.sh new file +93 -0
new file mode 100755
index 0000000..b04535e
@@ -0,0 +1,93 @@
+#!/bin/bash
+
+# Creates a package feed directory that only contains the packages that were
+# installed with the referenced image.  This script depends on image build
+# history to properly operate.  In your conf/local.conf, add the following two
+# lines:
+#   INHERIT += "buildhistory"
+#   BUILDHISTORY_COMMIT = "1"
+#
+# At this time this script is only setup to create opkg package feeds.
+# Further script modification is needed if you use a different package
+# managment system.
+#
+# This script assumes it is being run from your image build directory.  Some
+# paths may need to be modified if it is run from a different directory.
+#
+# Assumes that the relevant image was recently built.  If you have done any
+# package builds since the last image build, it is suggested that you re-build
+# relevant image.
+#
+# Script input variables:
+# $1 - The image name (i.e. "core-image-minimal")
+# $2 - The target machine (i.e. "beaglebone")
+# $3 - The target architecture (i.e. "arm926ejste")
+# $4 - The target directory (i.e. "feed")
+#
+
+# Validate the number of input variables
+if [[ "$#" -ne 4 ]]; then
+    echo "Unexpected number of arguments";
+    echo "Usage: make_ipk_repo.sh IMAGE_NAME TARGET_MACHINE TARGET_ARCH TARGET_DIR";
+    exit 1;
+fi
+
+# Some packages inexplicably contain a "1:" in the package name that does not
+# belong there.  Remove this text from these package names.
+
+sed -i 's/1://g' tmp/buildhistory/images/"$2"/eglibc/"$1"/installed-packages.txt
+
+# Read in the list of packages used by the image.  This includes the
+# machine/architecture independent packages ("all"), the architecture specific
+# packages, and the machine specific packages.
+ALL_PKG_LIST=$(grep '_all\.ipk' tmp/buildhistory/images/"$2"/eglibc/"$1"/installed-packages.txt)
+ARCH_PKG_LIST=$(grep '_'"$3"'\.ipk' tmp/buildhistory/images/"$2"/eglibc/"$1"/installed-packages.txt)
+MACHINE_PKG_LIST=$(grep '_'"$2"'\.ipk' tmp/buildhistory/images/"$2"/eglibc/"$1"/installed-packages.txt)
+
+echo "ALL_PKG_LIST is $ALL_PKG_LIST";
+echo "ARCH_PKG_LIST is $ARCH_PKG_LIST";
+echo "MACHINE_PKG_LIST is $MACHINE_PKG_LIST";
+
+# Create the new directory for the feeds, making sure there is no previous copy
+rm -r "$4"/
+rm "$4".zip
+mkdir -p "$4"/
+
+# Copy over the "all" packages
+mkdir -p "$4"/all/
+echo "Copying packages to 'all' directory"
+for i in $ALL_PKG_LIST; do
+    cp tmp/deploy/ipk/all/"$i" "$4"/all/
+done
+
+# Copy over the architecture packages
+mkdir -p "$4"/$3/
+echo "Copying packages to '$3' directory"
+for i in $ARCH_PKG_LIST; do
+    cp tmp/deploy/ipk/"$3"/"$i" "$4"/"$3"/
+done
+
+# Copy over the machine packages
+mkdir -p "$4"/"$2"/
+echo "Copying packages to '$2' directory"
+for i in $MACHINE_PKG_LIST; do
+    cp tmp/deploy/ipk/"$2"/"$i" "$4"/"$2"/
+done
+
+# Build the package index files
+echo "Building package index files"
+touch "$4"/Packages
+flock "$4"/Packages.flock -c "tmp/sysroots/i686-linux/usr/bin/opkg-make-index -r $4/Packages -p $4/Packages -m $4/"
+touch "$4"/all/Packages
+flock "$4"/all/Packages.flock -c "tmp/sysroots/i686-linux/usr/bin/opkg-make-index -r $4/all/Packages -p $4/all/Packages -m $4/all/"
+touch "$4"/"$3"/Packages
+flock "$4"/"$3"/Packages.flock -c "tmp/sysroots/i686-linux/usr/bin/opkg-make-index -r $4/$3/Packages -p $4/$3/Packages -m $4/$3/"
+touch "$4"/"$2"/Packages
+flock "$4"/"$2"/Packages.flock -c "tmp/sysroots/i686-linux/usr/bin/opkg-make-index -r $4/$2/Packages -p $4/$2/Packages -m $4/$2/"
+
+#Create a zip file of the directory for archive purposes
+cd "$4"/
+zip -r "$4" .
+
+echo "A reduced package repository has been created at $4"
+
Signature · 7 lines
1.7.9.5

-- 
_______________________________________________
poky mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/poky
Signature · 1 line
Martin 'JaMa' Jansa     jabber: [email protected]
#2 lore raw
Quoted 7 lines · Martin Jansa
Quoted 6 lines · Bryan Evenson

Adding a new script which generates a reduced package feed directory. The script uses the list of installed packages for a specific build image and creates a new package feed directory with only the packages relevant to that image. This is useful for distributing firmware upgrades.

Quoted 5 lines
Quoted 3 lines

At this time the script only works for opkg repositories. Further modifications are needed for the script to work with other package management systems.

This belongs to oe-core ML

Agreed.

Quoted 91 lines
Quoted 91 lines
---
 scripts/contrib/make_ipk_repo.sh |   93 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 93 insertions(+)
 create mode 100755 scripts/contrib/make_ipk_repo.sh
scripts/contrib/make_ipk_repo.sh new file +78 -0
new file mode 100755
index 0000000..b04535e
@@ -0,0 +1,93 @@
+#!/bin/bash
+
+# Creates a package feed directory that only contains the packages that were
+# installed with the referenced image.  This script depends on image build
+# history to properly operate.  In your conf/local.conf, add the following two
+# lines:
+#   INHERIT += "buildhistory"
+#   BUILDHISTORY_COMMIT = "1"
+#
+# At this time this script is only setup to create opkg package feeds.
+# Further script modification is needed if you use a different package
+# managment system.
+#
+# This script assumes it is being run from your image build directory.  Some
+# paths may need to be modified if it is run from a different directory.
+#
+# Assumes that the relevant image was recently built.  If you have done any
+# package builds since the last image build, it is suggested that you re-build
+# relevant image.
+#
+# Script input variables:
+# $1 - The image name (i.e. "core-image-minimal")
+# $2 - The target machine (i.e. "beaglebone")
+# $3 - The target architecture (i.e. "arm926ejste")
+# $4 - The target directory (i.e. "feed")
+#
+
+# Validate the number of input variables
+if [[ "$#" -ne 4 ]]; then
+    echo "Unexpected number of arguments";
+    echo "Usage: make_ipk_repo.sh IMAGE_NAME TARGET_MACHINE TARGET_ARCH TARGET_DIR";
+    exit 1;
+fi
+
+# Some packages inexplicably contain a "1:" in the package name that does not
+# belong there.  Remove this text from these package names.
+
+sed -i 's/1://g' tmp/buildhistory/images/"$2"/eglibc/"$1"/installed-packages.txt
+
+# Read in the list of packages used by the image.  This includes the
+# machine/architecture independent packages ("all"), the architecture specific
+# packages, and the machine specific packages.
+ALL_PKG_LIST=$(grep '_all\.ipk' tmp/buildhistory/images/"$2"/eglibc/"$1"/installed-packages.txt)
+ARCH_PKG_LIST=$(grep '_'"$3"'\.ipk' tmp/buildhistory/images/"$2"/eglibc/"$1"/installed-packages.txt)
+MACHINE_PKG_LIST=$(grep '_'"$2"'\.ipk' tmp/buildhistory/images/"$2"/eglibc/"$1"/installed-packages.txt)
+
+echo "ALL_PKG_LIST is $ALL_PKG_LIST";
+echo "ARCH_PKG_LIST is $ARCH_PKG_LIST";
+echo "MACHINE_PKG_LIST is $MACHINE_PKG_LIST";
+
+# Create the new directory for the feeds, making sure there is no previous copy
+rm -r "$4"/
+rm "$4".zip
+mkdir -p "$4"/
+
+# Copy over the "all" packages
+mkdir -p "$4"/all/
+echo "Copying packages to 'all' directory"
+for i in $ALL_PKG_LIST; do
+    cp tmp/deploy/ipk/all/"$i" "$4"/all/
+done
+
+# Copy over the architecture packages
+mkdir -p "$4"/$3/
+echo "Copying packages to '$3' directory"
+for i in $ARCH_PKG_LIST; do
+    cp tmp/deploy/ipk/"$3"/"$i" "$4"/"$3"/
+done
+
+# Copy over the machine packages
+mkdir -p "$4"/"$2"/
+echo "Copying packages to '$2' directory"
+for i in $MACHINE_PKG_LIST; do
+    cp tmp/deploy/ipk/"$2"/"$i" "$4"/"$2"/
+done
+
+# Build the package index files
+echo "Building package index files"
Quoted 8 lines

scripts/contrib/make_ipk_repo.sh line 86

Quoted 8 lines
scripts/contrib/make_ipk_repo.sh new file +8 -0
+touch "$4"/Packages
+flock "$4"/Packages.flock -c "tmp/sysroots/i686-linux/usr/bin/opkg-make-index -r $4/Packages -p $4/Packages -m $4/"
+touch "$4"/all/Packages
+flock "$4"/all/Packages.flock -c "tmp/sysroots/i686-linux/usr/bin/opkg-make-index -r $4/all/Packages -p $4/all/Packages -m $4/all/"
+touch "$4"/"$3"/Packages
+flock "$4"/"$3"/Packages.flock -c "tmp/sysroots/i686-linux/usr/bin/opkg-make-index -r $4/$3/Packages -p $4/$3/Packages -m $4/$3/"
+touch "$4"/"$2"/Packages
+flock "$4"/"$2"/Packages.flock -c "tmp/sysroots/i686-linux/usr/bin/opkg-make-index -r $4/$2/Packages -p $4/$2/Packages -m $4/$2/"

This is also hardcoded to be 32 bit build systems :/. There are a number of hardcoded assumptions in the script.

Cheers,

Richard

#3 lore raw

Richard and Martin,

Quoted 16 lines

-----Original Message-----
From: Richard Purdie [mailto:[email protected]]
Sent: Tuesday, August 12, 2014 8:41 AM
To: Martin Jansa
Cc: Bryan Evenson; [email protected]
Subject: Re: [poky] [PATCH] scripts/contrib/make_ipk_repo.sh: Add new image-specific package feed generator

Quoted 7 lines · Martin Jansa
Quoted 6 lines · Bryan Evenson

Adding a new script which generates a reduced package feed directory. The script uses the list of installed packages for a specific build image and creates a new package feed directory with only the packages relevant to that image. This is useful for distributing firmware upgrades.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

At this time the script only works for opkg repositories. Further modifications are needed for the script to work with other package management systems.

This belongs to oe-core ML

Agreed.

From browsing the the openembedded repository and the poky repository, I'd thought the poky mailing list was more appropriate on this item. I'll re-submit to the oe-core mailing list.

Quoted 102 lines
Quoted 5 lines
Quoted 5 lines
---
 scripts/contrib/make_ipk_repo.sh |   93

++++++++++++++++++++++++++++++++++++++

Quoted 17 lines
Quoted 17 lines
 1 file changed, 93 insertions(+)
 create mode 100755 scripts/contrib/make_ipk_repo.sh
scripts/contrib/make_ipk_repo.sh new file +7 -0
new file mode 100755
index 0000000..b04535e
@@ -0,0 +1,93 @@
+#!/bin/bash
+
+# Creates a package feed directory that only contains the packages
+that were # installed with the referenced image.  This script
+depends on image build # history to properly operate.  In your
+conf/local.conf, add the following two # lines:
+#   INHERIT += "buildhistory"
Quoted 8 lines

scripts/contrib/make_ipk_repo.sh

Quoted 8 lines
scripts/contrib/make_ipk_repo.sh new file +8 -0
+#   BUILDHISTORY_COMMIT = "1"
+#
+# At this time this script is only setup to create opkg package feeds.
+# Further script modification is needed if you use a different
+package # managment system.
+#
+# This script assumes it is being run from your image build
+directory.  Some # paths may need to be modified if it is run from a

different directory.

Quoted 5 lines
Quoted 5 lines
scripts/contrib/make_ipk_repo.sh new file +5 -0
+#
+# Assumes that the relevant image was recently built.  If you have
+done any # package builds since the last image build, it is
+suggested that you re-build # relevant image.
+#
Quoted 8 lines

scripts/contrib/make_ipk_repo.sh line 28

Quoted 8 lines
scripts/contrib/make_ipk_repo.sh new file +8 -0
+# Script input variables:
+# $1 - The image name (i.e. "core-image-minimal") # $2 - The target
+machine (i.e. "beaglebone") # $3 - The target architecture (i.e.
+"arm926ejste") # $4 - The target directory (i.e. "feed") #
+
+# Validate the number of input variables if [[ "$#" -ne 4 ]]; then
+    echo "Unexpected number of arguments";
+    echo "Usage: make_ipk_repo.sh IMAGE_NAME TARGET_MACHINE

TARGET_ARCH TARGET_DIR";

Quoted 5 lines

scripts/contrib/make_ipk_repo.sh

Quoted 5 lines
scripts/contrib/make_ipk_repo.sh new file +5 -0
+    exit 1;
+fi
+
+# Some packages inexplicably contain a "1:" in the package name
+that does not # belong there.  Remove this text from these package

names.

Quoted 8 lines
Quoted 8 lines
scripts/contrib/make_ipk_repo.sh new file +8 -0
+
+sed -i 's/1://g'
+tmp/buildhistory/images/"$2"/eglibc/"$1"/installed-packages.txt
+
+# Read in the list of packages used by the image.  This includes
+the # machine/architecture independent packages ("all"), the
+architecture specific # packages, and the machine specific packages.
+ALL_PKG_LIST=$(grep '_all\.ipk'
Quoted 8 lines

scripts/contrib/make_ipk_repo.sh line 49

Quoted 8 lines
scripts/contrib/make_ipk_repo.sh new file +8 -0
+tmp/buildhistory/images/"$2"/eglibc/"$1"/installed-packages.txt)
+ARCH_PKG_LIST=$(grep '_'"$3"'\.ipk'
+tmp/buildhistory/images/"$2"/eglibc/"$1"/installed-packages.txt)
+MACHINE_PKG_LIST=$(grep '_'"$2"'\.ipk'
+tmp/buildhistory/images/"$2"/eglibc/"$1"/installed-packages.txt)
+
+echo "ALL_PKG_LIST is $ALL_PKG_LIST"; echo "ARCH_PKG_LIST is
+$ARCH_PKG_LIST"; echo "MACHINE_PKG_LIST is

$MACHINE_PKG_LIST";

Quoted 17 lines
Quoted 17 lines
scripts/contrib/make_ipk_repo.sh new file +17 -0
+
+# Create the new directory for the feeds, making sure there is no
+previous copy rm -r "$4"/ rm "$4".zip mkdir -p "$4"/
+
+# Copy over the "all" packages
+mkdir -p "$4"/all/
+echo "Copying packages to 'all' directory"
+for i in $ALL_PKG_LIST; do
+    cp tmp/deploy/ipk/all/"$i" "$4"/all/ done
+
+# Copy over the architecture packages mkdir -p "$4"/$3/ echo
+"Copying packages to '$3' directory"
+for i in $ARCH_PKG_LIST; do
+    cp tmp/deploy/ipk/"$3"/"$i" "$4"/"$3"/ done
+
+# Copy over the machine packages
+mkdir -p "$4"/"$2"/
Quoted 8 lines

scripts/contrib/make_ipk_repo.sh

Quoted 8 lines
scripts/contrib/make_ipk_repo.sh new file +8 -0
+echo "Copying packages to '$2' directory"
+for i in $MACHINE_PKG_LIST; do
+    cp tmp/deploy/ipk/"$2"/"$i" "$4"/"$2"/ done
+
+# Build the package index files
+echo "Building package index files"
+touch "$4"/Packages
+flock "$4"/Packages.flock -c "tmp/sysroots/i686-linux/usr/bin/opkg-

make-index -r $4/Packages -p $4/Packages -m $4/"

Quoted 2 lines

scripts/contrib/make_ipk_repo.sh line 76

Quoted 2 lines
scripts/contrib/make_ipk_repo.sh new file +2 -0
+touch "$4"/all/Packages
+flock "$4"/all/Packages.flock -c "tmp/sysroots/i686-linux/usr/bin/opkg-

make-index -r $4/all/Packages -p $4/all/Packages -m $4/all/"

Quoted 2 lines

scripts/contrib/make_ipk_repo.sh line 78

Quoted 2 lines
scripts/contrib/make_ipk_repo.sh new file +2 -0
+touch "$4"/"$3"/Packages
+flock "$4"/"$3"/Packages.flock -c "tmp/sysroots/i686-

linux/usr/bin/opkg-make-index -r $4/$3/Packages -p $4/$3/Packages -m $4/$3/"

Quoted 7 lines
Quoted 2 lines

scripts/contrib/make_ipk_repo.sh line 80

Quoted 2 lines
scripts/contrib/make_ipk_repo.sh new file +2 -0
+touch "$4"/"$2"/Packages
+flock "$4"/"$2"/Packages.flock -c "tmp/sysroots/i686-

linux/usr/bin/opkg-make-index -r $4/$2/Packages -p $4/$2/Packages -m $4/$2/"

This is also hardcoded to be 32 bit build systems :/. There are a number of hardcoded assumptions in the script.

Agreed, I make a number of assumptions in the script and I list the assumptions that I knew I made. But, it’s a script I've found useful for my purposes and wanted to share with others. If you feel there is potential and if you have specific suggestions on what to change, I'll make the modifications before resubmitting.

Regards,
Bryan

Quoted 5 lines

Cheers,

Richard