package_rpm: Fix useradd preinst ordering issues

PATCH In review Richard Purdie 3 participants 11 messages Apr 11, 2012 – Apr 12, 2012

Share & tools

Richard Purdie

author

2012-04-11 21:40 UTC #0 lore raw

PATCH 1/1 package_rpm: Fix useradd preinst ordering issues

We were already having occasional ordering issues with package_rpm. Fixing the ldconfig postinstall issue pushed rpm over the cliff and totally broke rpm builds with the packages getting installed in effectively a random order and the useradd preinstalls getting executed out of order and breaking.

The only explanation I can find for this is that rpm is special. It will happily run a preinst for a package without any of that package's dependencies being present regardless of whether there are any circular dependency issues or not. I attempted various ways of solving this such as ordering the total_solution.manifest in creative ways but the bottom line is RPM ignores this. It takes little account of any request to ignore /bin/sh dependencies for the purposes of constructing the final image.

The end result is we're having to install the base-passwd, base-files and shadow packages first (if there is a request to install them), then install any other packages.

It this wasn't in the middle of a release I'd be rewriting this bbclass file, its horrible.

Signed-off-by Richard Purdie <[email protected]>

meta/classes/package_rpm.bbclass +46 ~~-16~~

index 16a2c87..1b0f6f2 100644
@@ -166,22 +167,23 @@ rpm_common_comand () {
 # install or remove the pkg
 rpm_update_pkg () {
 
+    manifest=$1
+    btmanifest=$manifest.bt
     local target_rootfs="${INSTALL_ROOTFS_RPM}"
 
     # Save the rpm's build time for incremental image generation, and the file
     # would be moved to ${T}
-    rm -f ${target_rootfs}/install/total_solution_bt.manifest
-    for i in `cat ${target_rootfs}/install/total_solution.manifest`; do
+    rm -f $btmanifest
+    for i in `cat $manifest`; do
         # Use "rpm" rather than "${RPM}" here, since we don't need the
         # '--dbpath' option
-        echo "$i `rpm -qp --qf '%{BUILDTIME}\n' $i`" >> \
-            ${target_rootfs}/install/total_solution_bt.manifest
+        echo "$i `rpm -qp --qf '%{BUILDTIME}\n' $i`" >> $btmanifest
     done
 
     # Only install the different pkgs if incremental image generation is set
     if [ "${INC_RPM_IMAGE_GEN}" = "1" -a -f ${T}/total_solution_bt.manifest -a \
         "${IMAGE_PKGTYPE}" = "rpm" ]; then
-        cur_list="${target_rootfs}/install/total_solution_bt.manifest"
+        cur_list="$btmanifest"
         pre_list="${T}/total_solution_bt.manifest"
         sort -u $cur_list -o $cur_list
         sort -u $pre_list -o $pre_list

@@ -203,8 +205,7 @@ rpm_update_pkg () {
             -Uvh ${target_rootfs}/install/incremental.manifest
     else
         # Attempt to install
-        rpm_common_comand --replacepkgs \
-            -Uhv ${target_rootfs}/install/total_solution.manifest
+        rpm_common_comand --replacepkgs -Uhv $manifest
     fi
 }

@@ -440,14 +441,7 @@ package_install_internal_rpm () {
 
 	fi
 
-	# If base-passwd or shadow are in the list of packages to install,
-	# ensure they are installed first to support later packages that
-	# may create custom users/groups (fixes Yocto bug #2127)
-	infile=${target_rootfs}/install/install_solution.manifest
-	outfile=${target_rootfs}/install/total_solution.manifest
-	cat $infile | grep /base-passwd-[0-9] > $outfile || true
-	cat $infile | grep /shadow-[0-9] >> $outfile || true
-	cat $infile | grep -v /shadow-[0-9] | grep -v /base-passwd-[0-9] >> $outfile || true
+	cat ${target_rootfs}/install/install_solution.manifest > ${target_rootfs}/install/total_solution.manifest
 	cat ${target_rootfs}/install/install_multilib_solution.manifest >> ${target_rootfs}/install/total_solution.manifest
 
 	# Construct install scriptlet wrapper

@@ -474,8 +468,45 @@ EOF
 
 	chmod 0755 ${WORKDIR}/scriptlet_wrapper
 
-    rpm_update_pkg
+	# RPM is special. It can't handle dependencies and preinstall scripts correctly. Its
+	# probably a feature. The only way to convince rpm to actually run the preinstall scripts 
+	# for base-passwd and shadow first before installing packages that depend on these packages 
+	# is to do two image installs, installing one set of packages, then the other.
+	if [ "${INC_RPM_IMAGE_GEN}" = "1" -a -f ${T}/total_solution_bt.manifest ]; then
+		echo "Skipping pre install due to exisitng image"
+	else
+		echo "# Intial Install manifest" > ${target_rootfs}/install/initial_install.manifest
+		echo "Installing base dependencies first (base-passwd, base-files and shadow) since rpm is special"
+		grep /base-passwd-[0-9] ${target_rootfs}/install/total_solution.manifest >> ${target_rootfs}/install/initial_install.manifest || true
+		grep /base-files-[0-9] ${target_rootfs}/install/total_solution.manifest >> ${target_rootfs}/install/initial_install.manifest || true		
+		grep /shadow-[0-9] ${target_rootfs}/install/total_solution.manifest >> ${target_rootfs}/install/initial_install.manifest || true
+
+		# Generate an install solution by doing a --justdb install, then recreate it with
+		# an actual package install!
+		mkdir -p ${target_rootfs}/initial
+
+		${RPM} --predefine "_rpmds_sysinfo_path ${target_rootfs}/etc/rpm/sysinfo" \
+			--predefine "_rpmrc_platform_path ${target_rootfs}/etc/rpm/platform" \
+			-D "_dbpath ${target_rootfs}/initial" -D "`cat ${confbase}-base_archs.macro`" \
+			-D "__dbi_txn create nofsync" \
+			-U --justdb --noscripts --notriggers --noparentdirs --nolinktos --ignoresize \
+			${target_rootfs}/install/initial_install.manifest
+
+		${RPM} -D "_dbpath ${target_rootfs}/initial" -qa --yaml \
+			-D "__dbi_txn create nofsync private" \
+			| grep -i 'Packageorigin' | cut -d : -f 2 > ${target_rootfs}/install/initial_solution.manifest
+
+		rpm_update_pkg ${target_rootfs}/install/initial_solution.manifest
+		
+		grep -Fv -f ${target_rootfs}/install/initial_solution.manifest ${target_rootfs}/install/total_solution.manifest > ${target_rootfs}/install/total_solution.manifest.new
+		mv ${target_rootfs}/install/total_solution.manifest.new ${target_rootfs}/install/total_solution.manifest
+		
+		rm -rf ${target_rootfs}/initial
+	fi
+
+	echo "Installing main solution manifest (${target_rootfs}/install/total_solution.manifest)"
 
+	rpm_update_pkg ${target_rootfs}/install/total_solution.manifest
 }
 
 python write_specfile () {

Mark Hatle

2012-04-11 21:52 UTC #1 lore raw

Quoted 15 lines · Richard Purdie

We were already having occasional ordering issues with package_rpm. Fixing the ldconfig postinstall issue pushed rpm over the cliff and totally broke rpm builds with the packages getting installed in effectively a random order and the useradd preinstalls getting executed out of order and breaking.
The only explanation I can find for this is that rpm is special. It will happily run a preinst for a package without any of that package's dependencies being present regardless of whether there are any circular dependency issues or not. I attempted various ways of solving this such as ordering the total_solution.manifest in creative ways but the bottom line is RPM ignores this. It takes little account of any request to ignore /bin/sh dependencies for the purposes of constructing the final image.

Quoted 8 lines

The end result is we're having to install the base-passwd, base-files and shadow packages first (if there is a request to install them), then install any other packages.
It this wasn't in the middle of a release I'd be rewriting this bbclass file, its horrible.
Signed-off-by Richard Purdie<[email protected]>

I've reviewed this. I don't see anything wrong with this, other then the pain of having to do this.

Signed-off-by Mark Hatle <[email protected]>

--Mark

Quoted 108 lines

meta/classes/package_rpm.bbclass +46 ~~-16~~

index 16a2c87..1b0f6f2 100644
@@ -166,22 +167,23 @@ rpm_common_comand () {
  # install or remove the pkg
  rpm_update_pkg () {
 
+    manifest=$1
+    btmanifest=$manifest.bt
      local target_rootfs="${INSTALL_ROOTFS_RPM}"
 
      # Save the rpm's build time for incremental image generation, and the file
      # would be moved to ${T}
-    rm -f ${target_rootfs}/install/total_solution_bt.manifest
-    for i in `cat ${target_rootfs}/install/total_solution.manifest`; do
+    rm -f $btmanifest
+    for i in `cat $manifest`; do
          # Use "rpm" rather than "${RPM}" here, since we don't need the
          # '--dbpath' option
-        echo "$i `rpm -qp --qf '%{BUILDTIME}\n' $i`">>  \
-            ${target_rootfs}/install/total_solution_bt.manifest
+        echo "$i `rpm -qp --qf '%{BUILDTIME}\n' $i`">>  $btmanifest
      done
 
      # Only install the different pkgs if incremental image generation is set
      if [ "${INC_RPM_IMAGE_GEN}" = "1" -a -f ${T}/total_solution_bt.manifest -a \
          "${IMAGE_PKGTYPE}" = "rpm" ]; then
-        cur_list="${target_rootfs}/install/total_solution_bt.manifest"
+        cur_list="$btmanifest"
          pre_list="${T}/total_solution_bt.manifest"
          sort -u $cur_list -o $cur_list
          sort -u $pre_list -o $pre_list

@@ -203,8 +205,7 @@ rpm_update_pkg () {
              -Uvh ${target_rootfs}/install/incremental.manifest
      else
          # Attempt to install
-        rpm_common_comand --replacepkgs \
-            -Uhv ${target_rootfs}/install/total_solution.manifest
+        rpm_common_comand --replacepkgs -Uhv $manifest
      fi
  }

@@ -440,14 +441,7 @@ package_install_internal_rpm () {
 
  	fi
 
-	# If base-passwd or shadow are in the list of packages to install,
-	# ensure they are installed first to support later packages that
-	# may create custom users/groups (fixes Yocto bug #2127)
-	infile=${target_rootfs}/install/install_solution.manifest
-	outfile=${target_rootfs}/install/total_solution.manifest
-	cat $infile | grep /base-passwd-[0-9]>  $outfile || true
-	cat $infile | grep /shadow-[0-9]>>  $outfile || true
-	cat $infile | grep -v /shadow-[0-9] | grep -v /base-passwd-[0-9]>>  $outfile || true
+	cat ${target_rootfs}/install/install_solution.manifest>  ${target_rootfs}/install/total_solution.manifest
  	cat ${target_rootfs}/install/install_multilib_solution.manifest>>  ${target_rootfs}/install/total_solution.manifest
 
  	# Construct install scriptlet wrapper

@@ -474,8 +468,45 @@ EOF
 
  	chmod 0755 ${WORKDIR}/scriptlet_wrapper
 
-    rpm_update_pkg
+	# RPM is special. It can't handle dependencies and preinstall scripts correctly. Its
+	# probably a feature. The only way to convince rpm to actually run the preinstall scripts
+	# for base-passwd and shadow first before installing packages that depend on these packages
+	# is to do two image installs, installing one set of packages, then the other.
+	if [ "${INC_RPM_IMAGE_GEN}" = "1" -a -f ${T}/total_solution_bt.manifest ]; then
+		echo "Skipping pre install due to exisitng image"
+	else
+		echo "# Intial Install manifest">  ${target_rootfs}/install/initial_install.manifest
+		echo "Installing base dependencies first (base-passwd, base-files and shadow) since rpm is special"
+		grep /base-passwd-[0-9] ${target_rootfs}/install/total_solution.manifest>>  ${target_rootfs}/install/initial_install.manifest || true
+		grep /base-files-[0-9] ${target_rootfs}/install/total_solution.manifest>>  ${target_rootfs}/install/initial_install.manifest || true		
+		grep /shadow-[0-9] ${target_rootfs}/install/total_solution.manifest>>  ${target_rootfs}/install/initial_install.manifest || true
+
+		# Generate an install solution by doing a --justdb install, then recreate it with
+		# an actual package install!
+		mkdir -p ${target_rootfs}/initial
+
+		${RPM} --predefine "_rpmds_sysinfo_path ${target_rootfs}/etc/rpm/sysinfo" \
+			--predefine "_rpmrc_platform_path ${target_rootfs}/etc/rpm/platform" \
+			-D "_dbpath ${target_rootfs}/initial" -D "`cat ${confbase}-base_archs.macro`" \
+			-D "__dbi_txn create nofsync" \
+			-U --justdb --noscripts --notriggers --noparentdirs --nolinktos --ignoresize \
+			${target_rootfs}/install/initial_install.manifest
+
+		${RPM} -D "_dbpath ${target_rootfs}/initial" -qa --yaml \
+			-D "__dbi_txn create nofsync private" \
+			| grep -i 'Packageorigin' | cut -d : -f 2>  ${target_rootfs}/install/initial_solution.manifest
+
+		rpm_update_pkg ${target_rootfs}/install/initial_solution.manifest
+		
+		grep -Fv -f ${target_rootfs}/install/initial_solution.manifest ${target_rootfs}/install/total_solution.manifest>  ${target_rootfs}/install/total_solution.manifest.new
+		mv ${target_rootfs}/install/total_solution.manifest.new ${target_rootfs}/install/total_solution.manifest
+		
+		rm -rf ${target_rootfs}/initial
+	fi
+
+	echo "Installing main solution manifest (${target_rootfs}/install/total_solution.manifest)"
 
+	rpm_update_pkg ${target_rootfs}/install/total_solution.manifest
  }
 
  python write_specfile () {

Steve Sakoman

2012-04-12 13:49 UTC #2 lore raw

Quoted 24 lines · Mark Hatle

Quoted 23 lines · Richard Purdie
We were already having occasional ordering issues with package_rpm. Fixing the ldconfig postinstall issue pushed rpm over the cliff and totally broke rpm builds with the packages getting installed in effectively a random order and the useradd preinstalls getting executed out of order and breaking.
The only explanation I can find for this is that rpm is special. It will happily run a preinst for a package without any of that package's dependencies being present regardless of whether there are any circular dependency issues or not. I attempted various ways of solving this such as ordering the total_solution.manifest in creative ways but the bottom line is RPM ignores this. It takes little account of any request to ignore /bin/sh dependencies for the purposes of constructing the final image.
The end result is we're having to install the base-passwd, base-files and shadow packages first (if there is a request to install them), then install any other packages.
It this wasn't in the middle of a release I'd be rewriting this bbclass file, its horrible.

Quoted 7 lines

Quoted 1 line
Signed-off-by Richard Purdie<[email protected]>
I've reviewed this. I don't see anything wrong with this, other then the pain of having to do this.
Signed-off-by Mark Hatle <[email protected]>

FWIW, after pulling current poky this morning all of my image builds are failing with errors like this:

| Installing base dependencies first (base-passwd, base-files and shadow) since rpm is special

| error: Failed dependencies:
| 	/bin/sh is needed by base-passwd-3.5.24-r0.armv7a
| 	/bin/sh is needed by shadow-4.1.4.3-r8.armv7a
| 	/bin/sh is needed by libgcc1-4.6.3+svnr184847-r24.armv7a
| 	/bin/sh is needed by libc6-2.13-r23+svnr15508.armv7a

I'll investigate, but since it is late in the release process thought I would mention the issue.

Steve

Quoted 145 lines

Quoted 134 lines
meta/classes/package_rpm.bbclass +2 -0
index 16a2c87..1b0f6f2 100644
@@ -166,22 +167,23 @@ rpm_common_comand () {
  # install or remove the pkg
  rpm_update_pkg () {
 
+    manifest=$1
+    btmanifest=$manifest.bt
local target_rootfs="${INSTALL_ROOTFS_RPM}"
# Save the rpm's build time for incremental image generation, and the file
     # would be moved to ${T}
-    rm -f ${target_rootfs}/install/total_solution_bt.manifest
-    for i in `cat ${target_rootfs}/install/total_solution.manifest`; do
+    rm -f $btmanifest
+    for i in `cat $manifest`; do
         # Use "rpm" rather than "${RPM}" here, since we don't need the
         # '--dbpath' option
-        echo "$i `rpm -qp --qf '%{BUILDTIME}\n' $i`">>  \
-            ${target_rootfs}/install/total_solution_bt.manifest
+        echo "$i `rpm -qp --qf '%{BUILDTIME}\n' $i`">>  $btmanifest
     done
# Only install the different pkgs if incremental image generation is set
if [ "${INC_RPM_IMAGE_GEN}" = "1" -a -f ${T}/total_solution_bt.manifest -a \
"${IMAGE_PKGTYPE}" = "rpm" ]; then
- cur_list="${target_rootfs}/install/total_solution_bt.manifest"
+ cur_list="$btmanifest" pre_list="${T}/total_solution_bt.manifest"
sort -u $cur_list -o $cur_list
sort -u $pre_list -o $pre_list
meta/classes/package_rpm.bbclass +0 -0
@@ -203,8 +205,7 @@ rpm_update_pkg () {
-Uvh ${target_rootfs}/install/incremental.manifest else
# Attempt to install
- rpm_common_comand --replacepkgs \
- -Uhv ${target_rootfs}/install/total_solution.manifest
+ rpm_common_comand --replacepkgs -Uhv $manifest fi
}
meta/classes/package_rpm.bbclass +0 -0
@@ -440,14 +441,7 @@ package_install_internal_rpm () {
 
fi
- # If base-passwd or shadow are in the list of packages to install,
- # ensure they are installed first to support later packages that
- # may create custom users/groups (fixes Yocto bug #2127)
- infile=${target_rootfs}/install/install_solution.manifest
- outfile=${target_rootfs}/install/total_solution.manifest
-       cat $infile | grep /base-passwd-[0-9]>  $outfile || true
-       cat $infile | grep /shadow-[0-9]>>  $outfile || true
-       cat $infile | grep -v /shadow-[0-9] | grep -v /base-passwd-[0-9]>>
$outfile || true
+ cat ${target_rootfs}/install/install_solution.manifest> ${target_rootfs}/install/total_solution.manifest
cat ${target_rootfs}/install/install_multilib_solution.manifest>> ${target_rootfs}/install/total_solution.manifest
# Construct install scriptlet wrapper
meta/classes/package_rpm.bbclass +0 -0
@@ -474,8 +468,45 @@ EOF
 
chmod 0755 ${WORKDIR}/scriptlet_wrapper
- rpm_update_pkg
+ # RPM is special. It can't handle dependencies and preinstall scripts correctly. Its
+ # probably a feature. The only way to convince rpm to actually run the preinstall scripts
+ # for base-passwd and shadow first before installing packages that depend on these packages
+ # is to do two image installs, installing one set of packages, then the other.
+ if [ "${INC_RPM_IMAGE_GEN}" = "1" -a -f ${T}/total_solution_bt.manifest ]; then
+ echo "Skipping pre install due to exisitng image"
+ else
+ echo "# Intial Install manifest"> ${target_rootfs}/install/initial_install.manifest
+ echo "Installing base dependencies first (base-passwd, base-files and shadow) since rpm is special"
+ grep /base-passwd-[0-9] ${target_rootfs}/install/total_solution.manifest>> ${target_rootfs}/install/initial_install.manifest || true
+ grep /base-files-[0-9] ${target_rootfs}/install/total_solution.manifest>> ${target_rootfs}/install/initial_install.manifest || true
+ grep /shadow-[0-9] ${target_rootfs}/install/total_solution.manifest>> ${target_rootfs}/install/initial_install.manifest || true
+
+ # Generate an install solution by doing a --justdb install, then recreate it with
+ # an actual package install!
+ mkdir -p ${target_rootfs}/initial
+
+ ${RPM} --predefine "_rpmds_sysinfo_path ${target_rootfs}/etc/rpm/sysinfo" \
+ --predefine "_rpmrc_platform_path ${target_rootfs}/etc/rpm/platform" \
+ -D "_dbpath ${target_rootfs}/initial" -D "cat ${confbase}-base_archs.macro" \
+ -D "__dbi_txn create nofsync" \
+ -U --justdb --noscripts --notriggers --noparentdirs --nolinktos --ignoresize \
+ ${target_rootfs}/install/initial_install.manifest +
+ ${RPM} -D "_dbpath ${target_rootfs}/initial" -qa --yaml \
+ -D "__dbi_txn create nofsync private" \
+ | grep -i 'Packageorigin' | cut -d : -f 2> ${target_rootfs}/install/initial_solution.manifest
+
+ rpm_update_pkg ${target_rootfs}/install/initial_solution.manifest
+
+ grep -Fv -f ${target_rootfs}/install/initial_solution.manifest ${target_rootfs}/install/total_solution.manifest> ${target_rootfs}/install/total_solution.manifest.new
+ mv ${target_rootfs}/install/total_solution.manifest.new ${target_rootfs}/install/total_solution.manifest
meta/classes/package_rpm.bbclass +5 -0
+
+               rm -rf ${target_rootfs}/initial
+       fi
+
+       echo "Installing main solution manifest
(${target_rootfs}/install/total_solution.manifest)"
Quoted 5 lines
+ rpm_update_pkg ${target_rootfs}/install/total_solution.manifest }
python write_specfile () {
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Steve Sakoman

2012-04-12 15:45 UTC #3 lore raw

Quoted 5 lines · Steve Sakoman

FWIW, after pulling current poky this morning all of my image builds are failing with errors like this:
| Installing base dependencies first (base-passwd, base-files and shadow) since rpm is special

Quoted 8 lines

| error: Failed dependencies:
|       /bin/sh is needed by base-passwd-3.5.24-r0.armv7a
|       /bin/sh is needed by shadow-4.1.4.3-r8.armv7a
|       /bin/sh is needed by libgcc1-4.6.3+svnr184847-r24.armv7a
|       /bin/sh is needed by libc6-2.13-r23+svnr15508.armv7a

I'll investigate, but since it is late in the release process thought I would mention the issue.

I have no idea if this is the proper fix, but adding bash to the list of packages to install first fixed the issue for me:

meta/classes/package_rpm.bbclass +0 -0

index fd00fb1..a5f0b06 100644
@@ -475,7 +475,8 @@ EOF
                echo "Skipping pre install due to exisitng image"
        else
                echo "# Initial Install manifest" >

${target_rootfs}/install/initial_install.manifest
- echo "Installing base dependencies first (base-passwd,
base-files and shadow) since rpm is special"
+ echo "Installing base dependencies first (bash,
base-passwd, base-files and shadow) since rpm is special"
+ grep /bash-[0-9]
${target_rootfs}/install/total_solution.manifest >> ${target_rootfs}/install/initial_install.manife

                grep /base-passwd-[0-9]
${target_rootfs}/install/total_solution.manifest >>
${target_rootfs}/install/initial_install
                grep /base-files-[0-9]
${target_rootfs}/install/total_solution.manifest >>
${target_rootfs}/install/initial_install.
                grep /shadow-[0-9]

${target_rootfs}/install/total_solution.manifest >> ${target_rootfs}/install/initial_install.mani

I'm concerned we are on a slippery slope here!

Steve

Quoted 145 lines

Quoted 145 lines
Quoted 134 lines
meta/classes/package_rpm.bbclass +2 -0
index 16a2c87..1b0f6f2 100644
@@ -166,22 +167,23 @@ rpm_common_comand () {
  # install or remove the pkg
  rpm_update_pkg () {
 
+    manifest=$1
+    btmanifest=$manifest.bt
local target_rootfs="${INSTALL_ROOTFS_RPM}"
# Save the rpm's build time for incremental image generation, and the file
     # would be moved to ${T}
-    rm -f ${target_rootfs}/install/total_solution_bt.manifest
-    for i in `cat ${target_rootfs}/install/total_solution.manifest`; do
+    rm -f $btmanifest
+    for i in `cat $manifest`; do
         # Use "rpm" rather than "${RPM}" here, since we don't need the
         # '--dbpath' option
-        echo "$i `rpm -qp --qf '%{BUILDTIME}\n' $i`">>  \
-            ${target_rootfs}/install/total_solution_bt.manifest
+        echo "$i `rpm -qp --qf '%{BUILDTIME}\n' $i`">>  $btmanifest
     done
# Only install the different pkgs if incremental image generation is set
if [ "${INC_RPM_IMAGE_GEN}" = "1" -a -f ${T}/total_solution_bt.manifest -a \
"${IMAGE_PKGTYPE}" = "rpm" ]; then
- cur_list="${target_rootfs}/install/total_solution_bt.manifest"
+ cur_list="$btmanifest" pre_list="${T}/total_solution_bt.manifest"
sort -u $cur_list -o $cur_list
sort -u $pre_list -o $pre_list
meta/classes/package_rpm.bbclass +0 -0
@@ -203,8 +205,7 @@ rpm_update_pkg () {
-Uvh ${target_rootfs}/install/incremental.manifest else
# Attempt to install
- rpm_common_comand --replacepkgs \
- -Uhv ${target_rootfs}/install/total_solution.manifest
+ rpm_common_comand --replacepkgs -Uhv $manifest fi
}
meta/classes/package_rpm.bbclass +0 -0
@@ -440,14 +441,7 @@ package_install_internal_rpm () {
 
fi
- # If base-passwd or shadow are in the list of packages to install,
- # ensure they are installed first to support later packages that
- # may create custom users/groups (fixes Yocto bug #2127)
- infile=${target_rootfs}/install/install_solution.manifest
- outfile=${target_rootfs}/install/total_solution.manifest
-       cat $infile | grep /base-passwd-[0-9]>  $outfile || true
-       cat $infile | grep /shadow-[0-9]>>  $outfile || true
-       cat $infile | grep -v /shadow-[0-9] | grep -v /base-passwd-[0-9]>>
$outfile || true
+ cat ${target_rootfs}/install/install_solution.manifest> ${target_rootfs}/install/total_solution.manifest
cat ${target_rootfs}/install/install_multilib_solution.manifest>> ${target_rootfs}/install/total_solution.manifest
# Construct install scriptlet wrapper
meta/classes/package_rpm.bbclass +0 -0
@@ -474,8 +468,45 @@ EOF
 
chmod 0755 ${WORKDIR}/scriptlet_wrapper
- rpm_update_pkg
+ # RPM is special. It can't handle dependencies and preinstall scripts correctly. Its
+ # probably a feature. The only way to convince rpm to actually run the preinstall scripts
+ # for base-passwd and shadow first before installing packages that depend on these packages
+ # is to do two image installs, installing one set of packages, then the other.
+ if [ "${INC_RPM_IMAGE_GEN}" = "1" -a -f ${T}/total_solution_bt.manifest ]; then
+ echo "Skipping pre install due to exisitng image"
+ else
+ echo "# Intial Install manifest"> ${target_rootfs}/install/initial_install.manifest
+ echo "Installing base dependencies first (base-passwd, base-files and shadow) since rpm is special"
+ grep /base-passwd-[0-9] ${target_rootfs}/install/total_solution.manifest>> ${target_rootfs}/install/initial_install.manifest || true
+ grep /base-files-[0-9] ${target_rootfs}/install/total_solution.manifest>> ${target_rootfs}/install/initial_install.manifest || true
+ grep /shadow-[0-9] ${target_rootfs}/install/total_solution.manifest>> ${target_rootfs}/install/initial_install.manifest || true
+
+ # Generate an install solution by doing a --justdb install, then recreate it with
+ # an actual package install!
+ mkdir -p ${target_rootfs}/initial
+
+ ${RPM} --predefine "_rpmds_sysinfo_path ${target_rootfs}/etc/rpm/sysinfo" \
+ --predefine "_rpmrc_platform_path ${target_rootfs}/etc/rpm/platform" \
+ -D "_dbpath ${target_rootfs}/initial" -D "cat ${confbase}-base_archs.macro" \
+ -D "__dbi_txn create nofsync" \
+ -U --justdb --noscripts --notriggers --noparentdirs --nolinktos --ignoresize \
+ ${target_rootfs}/install/initial_install.manifest +
+ ${RPM} -D "_dbpath ${target_rootfs}/initial" -qa --yaml \
+ -D "__dbi_txn create nofsync private" \
+ | grep -i 'Packageorigin' | cut -d : -f 2> ${target_rootfs}/install/initial_solution.manifest
+
+ rpm_update_pkg ${target_rootfs}/install/initial_solution.manifest
+
+ grep -Fv -f ${target_rootfs}/install/initial_solution.manifest ${target_rootfs}/install/total_solution.manifest> ${target_rootfs}/install/total_solution.manifest.new
+ mv ${target_rootfs}/install/total_solution.manifest.new ${target_rootfs}/install/total_solution.manifest
meta/classes/package_rpm.bbclass +5 -0
+
+               rm -rf ${target_rootfs}/initial
+       fi
+
+       echo "Installing main solution manifest
(${target_rootfs}/install/total_solution.manifest)"
Quoted 5 lines
+ rpm_update_pkg ${target_rootfs}/install/total_solution.manifest }
python write_specfile () {
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Mark Hatle

2012-04-12 15:55 UTC #4 lore raw

Quoted 1 line · Steve Sakoman

On Thu, Apr 12, 2012 at 6:39 AM, Steve Sakoman<[email protected]> wrote:

That is very odd, the system is supposed to identify all of the dependencies needed for the first set of packages... (the three base-passwd, base-files and shadow).

The end result is a list of 10-12 binaries to be installed that meet those requirements, as well as the original three requested packages.

We also shouldn't specify bash because busybox provides /bin/sh in a lot of configurations.

Which image did you try to build and I'll see what I can replicate here.

--Mark

Quoted 193 lines

Quoted 5 lines
FWIW, after pulling current poky this morning all of my image builds are failing with errors like this:
| Installing base dependencies first (base-passwd, base-files and shadow) since rpm is special
Quoted 8 lines
| error: Failed dependencies:
|       /bin/sh is needed by base-passwd-3.5.24-r0.armv7a
|       /bin/sh is needed by shadow-4.1.4.3-r8.armv7a
|       /bin/sh is needed by libgcc1-4.6.3+svnr184847-r24.armv7a
|       /bin/sh is needed by libc6-2.13-r23+svnr15508.armv7a
I'll investigate, but since it is late in the release process thought I would mention the issue.
I have no idea if this is the proper fix, but adding bash to the list of packages to install first fixed the issue for me:
meta/classes/package_rpm.bbclass +0 -0
index fd00fb1..a5f0b06 100644
@@ -475,7 +475,8 @@ EOF
                 echo "Skipping pre install due to exisitng image"
         else
                 echo "# Initial Install manifest">
${target_rootfs}/install/initial_install.manifest
- echo "Installing base dependencies first (base-passwd,
base-files and shadow) since rpm is special"
+ echo "Installing base dependencies first (bash,
base-passwd, base-files and shadow) since rpm is special"
+ grep /bash-[0-9]
${target_rootfs}/install/total_solution.manifest>> ${target_rootfs}/install/initial_install.manife
                 grep /base-passwd-[0-9]
${target_rootfs}/install/total_solution.manifest>>
${target_rootfs}/install/initial_install
                 grep /base-files-[0-9]
${target_rootfs}/install/total_solution.manifest>>
${target_rootfs}/install/initial_install.
                 grep /shadow-[0-9]
${target_rootfs}/install/total_solution.manifest>> ${target_rootfs}/install/initial_install.mani
I'm concerned we are on a slippery slope here!
Steve
Quoted 145 lines
Quoted 145 lines
Quoted 134 lines
meta/classes/package_rpm.bbclass +2 -0
index 16a2c87..1b0f6f2 100644
@@ -166,22 +167,23 @@ rpm_common_comand () {
  # install or remove the pkg
  rpm_update_pkg () {
 
+    manifest=$1
+    btmanifest=$manifest.bt
      local target_rootfs="${INSTALL_ROOTFS_RPM}"
 
      # Save the rpm's build time for incremental image generation, and the
file
      # would be moved to ${T}
meta/classes/package_rpm.bbclass +3 -4
-    rm -f ${target_rootfs}/install/total_solution_bt.manifest
-    for i in `cat ${target_rootfs}/install/total_solution.manifest`; do
+    rm -f $btmanifest
+    for i in `cat $manifest`; do
          # Use "rpm" rather than "${RPM}" here, since we don't need the
          # '--dbpath' option
-        echo "$i `rpm -qp --qf '%{BUILDTIME}\n' $i`">>    \
-            ${target_rootfs}/install/total_solution_bt.manifest
+        echo "$i `rpm -qp --qf '%{BUILDTIME}\n' $i`">>    $btmanifest
      done
 
      # Only install the different pkgs if incremental image generation is
set
      if [ "${INC_RPM_IMAGE_GEN}" = "1" -a -f
${T}/total_solution_bt.manifest -a \
          "${IMAGE_PKGTYPE}" = "rpm" ]; then
meta/classes/package_rpm.bbclass +3 ~~-11~~
-        cur_list="${target_rootfs}/install/total_solution_bt.manifest"
+        cur_list="$btmanifest"
          pre_list="${T}/total_solution_bt.manifest"
          sort -u $cur_list -o $cur_list
          sort -u $pre_list -o $pre_list
@@ -203,8 +205,7 @@ rpm_update_pkg () {
              -Uvh ${target_rootfs}/install/incremental.manifest
      else
          # Attempt to install
-        rpm_common_comand --replacepkgs \
-            -Uhv ${target_rootfs}/install/total_solution.manifest
+        rpm_common_comand --replacepkgs -Uhv $manifest
      fi
  }
 
@@ -440,14 +441,7 @@ package_install_internal_rpm () {
 
        fi
 
-       # If base-passwd or shadow are in the list of packages to install,
-       # ensure they are installed first to support later packages that
-       # may create custom users/groups (fixes Yocto bug #2127)
-       infile=${target_rootfs}/install/install_solution.manifest
-       outfile=${target_rootfs}/install/total_solution.manifest
-       cat $infile | grep /base-passwd-[0-9]>    $outfile || true
-       cat $infile | grep /shadow-[0-9]>>    $outfile || true
-       cat $infile | grep -v /shadow-[0-9] | grep -v /base-passwd-[0-9]>>
  $outfile || true
+       cat ${target_rootfs}/install/install_solution.manifest>
  ${target_rootfs}/install/total_solution.manifest
        cat ${target_rootfs}/install/install_multilib_solution.manifest>>
  ${target_rootfs}/install/total_solution.manifest
# Construct install scriptlet wrapper
meta/classes/package_rpm.bbclass +1 -1
@@ -474,8 +468,45 @@ EOF
 
        chmod 0755 ${WORKDIR}/scriptlet_wrapper
 
-    rpm_update_pkg
+       # RPM is special. It can't handle dependencies and preinstall
scripts correctly. Its
+ # probably a feature. The only way to convince rpm to actually run
the preinstall scripts
+ # for base-passwd and shadow first before installing packages that
depend on these packages
+ # is to do two image installs, installing one set of packages,
then the other.
+ if [ "${INC_RPM_IMAGE_GEN}" = "1" -a -f
${T}/total_solution_bt.manifest ]; then
meta/classes/package_rpm.bbclass +4 -0
+               echo "Skipping pre install due to exisitng image"
+       else
+               echo "# Intial Install manifest">
  ${target_rootfs}/install/initial_install.manifest
+               echo "Installing base dependencies first (base-passwd,
base-files and shadow) since rpm is special"
+ grep /base-passwd-[0-9]
${target_rootfs}/install/total_solution.manifest>>
${target_rootfs}/install/initial_install.manifest || true
+ grep /base-files-[0-9]
${target_rootfs}/install/total_solution.manifest>>
${target_rootfs}/install/initial_install.manifest || true
+ grep /shadow-[0-9]
${target_rootfs}/install/total_solution.manifest>>
${target_rootfs}/install/initial_install.manifest || true
meta/classes/package_rpm.bbclass +2 -0
+
+               # Generate an install solution by doing a --justdb
install, then recreate it with
meta/classes/package_rpm.bbclass +4 -0
+               # an actual package install!
+               mkdir -p ${target_rootfs}/initial
+
+               ${RPM} --predefine "_rpmds_sysinfo_path
${target_rootfs}/etc/rpm/sysinfo" \
+ --predefine "_rpmrc_platform_path
${target_rootfs}/etc/rpm/platform" \
+ -D "_dbpath ${target_rootfs}/initial" -D "`cat
${confbase}-base_archs.macro`" \
meta/classes/package_rpm.bbclass +9 -1
+                       -D "__dbi_txn create nofsync" \
+                       -U --justdb --noscripts --notriggers
--noparentdirs --nolinktos --ignoresize \
+                       ${target_rootfs}/install/initial_install.manifest
+
+               ${RPM} -D "_dbpath ${target_rootfs}/initial" -qa --yaml \
+                       -D "__dbi_txn create nofsync private" \
+                       | grep -i 'Packageorigin' | cut -d : -f 2>
  ${target_rootfs}/install/initial_solution.manifest
+
+               rpm_update_pkg
${target_rootfs}/install/initial_solution.manifest
meta/classes/package_rpm.bbclass +2 -0
+
+               grep -Fv -f
${target_rootfs}/install/initial_solution.manifest ${target_rootfs}/install/total_solution.manifest>
${target_rootfs}/install/total_solution.manifest.new
+ mv ${target_rootfs}/install/total_solution.manifest.new
${target_rootfs}/install/total_solution.manifest
meta/classes/package_rpm.bbclass +5 -0
+
+               rm -rf ${target_rootfs}/initial
+       fi
+
+       echo "Installing main solution manifest
(${target_rootfs}/install/total_solution.manifest)"
Quoted 5 lines
+ rpm_update_pkg ${target_rootfs}/install/total_solution.manifest
}
  python write_specfile () {
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Steve Sakoman

2012-04-12 16:05 UTC #5 lore raw

Quoted 8 lines · Mark Hatle

Quoted 2 lines · Steve Sakoman
On Thu, Apr 12, 2012 at 6:39 AM, Steve Sakoman<[email protected]> wrote:
That is very odd, the system is supposed to identify all of the dependencies needed for the first set of packages... (the three base-passwd, base-files and shadow).

Perhaps the issue is with the bash package -- for some reason rpm doesn't realize that bash provides /bin/sh?

Quoted 7 lines

The end result is a list of 10-12 binaries to be installed that meet those requirements, as well as the original three requested packages.
We also shouldn't specify bash because busybox provides /bin/sh in a lot of configurations.
Which image did you try to build and I'll see what I can replicate here.

The images I built are not standard oe-core/yocto images, they are custom images that don't include busybox. Perhaps that is why I am seeing the issue and others aren't.

If you are really interested in replicating let me know and I can provide you with pointers to my image recipes.

Steve

Richard Purdie

author

2012-04-12 16:17 UTC #6 lore raw

Quoted 19 lines · Steve Sakoman

Quoted 8 lines · Mark Hatle
Quoted 2 lines · Steve Sakoman
On Thu, Apr 12, 2012 at 6:39 AM, Steve Sakoman<[email protected]> wrote:
That is very odd, the system is supposed to identify all of the dependencies needed for the first set of packages... (the three base-passwd, base-files and shadow).
Perhaps the issue is with the bash package -- for some reason rpm doesn't realize that bash provides /bin/sh?
Quoted 6 lines
The end result is a list of 10-12 binaries to be installed that meet those requirements, as well as the original three requested packages.
We also shouldn't specify bash because busybox provides /bin/sh in a lot of configurations.

Quoted 8 lines

Quoted 1 line
Which image did you try to build and I'll see what I can replicate here.
The images I built are not standard oe-core/yocto images, they are custom images that don't include busybox. Perhaps that is why I am seeing the issue and others aren't.
If you are really interested in replicating let me know and I can provide you with pointers to my image recipes.

For what its worth, bash here does Provides: /bin/sh so I'm a little puzzled about what is going on here. If its not doing that, it would certainly cause the error you're reporting...

This is with master? Did you do anything like disable the per file dependencies?

Cheers,

Richard

Steve Sakoman

2012-04-12 17:16 UTC #7 lore raw

Quoted 3 lines · Richard Purdie

For what its worth, bash here does Provides: /bin/sh so I'm a little puzzled about what is going on here. If its not doing that, it would certainly cause the error you're reporting...

Hmm . . . while I see:

Provides: /bin/sh

in bash.spec, it is a different story in the pkgdata/runtime files.

busybox has:

FILERPROVIDES_/bin/busybox_busybox: <snip many filenames> /bin/sh <snip many filenames>

While bash doesn't have a FILERPROVIDES entry at all.

Could this be why rpm doesn't realize bash provides /bin/sh?

Steve

Mark Hatle

2012-04-12 17:20 UTC #8 lore raw

Quoted 13 lines · Steve Sakoman

Quoted 3 lines · Richard Purdie
For what its worth, bash here does Provides: /bin/sh so I'm a little puzzled about what is going on here. If its not doing that, it would certainly cause the error you're reporting...
Hmm . . . while I see:
Provides: /bin/sh
in bash.spec, it is a different story in the pkgdata/runtime files.

Quoted 8 lines

busybox has:
FILERPROVIDES_/bin/busybox_busybox:<snip many filenames> /bin/sh <snip many filenames>
While bash doesn't have a FILERPROVIDES entry at all.
Could this be why rpm doesn't realize bash provides /bin/sh?

They should both be equally valid for the purpose of dependency resolution.

--Mark

Quoted 1 line

Steve

Mark Hatle

2012-04-12 16:47 UTC #9 lore raw

In reply to Steve Sakoman

Quoted 19 lines · Steve Sakoman

Quoted 8 lines · Mark Hatle
Quoted 2 lines · Steve Sakoman
On Thu, Apr 12, 2012 at 6:39 AM, Steve Sakoman<[email protected]> wrote:
That is very odd, the system is supposed to identify all of the dependencies needed for the first set of packages... (the three base-passwd, base-files and shadow).
Perhaps the issue is with the bash package -- for some reason rpm doesn't realize that bash provides /bin/sh?
Quoted 6 lines
The end result is a list of 10-12 binaries to be installed that meet those requirements, as well as the original three requested packages.
We also shouldn't specify bash because busybox provides /bin/sh in a lot of configurations.

Quoted 8 lines

Quoted 1 line
Which image did you try to build and I'll see what I can replicate here.
The images I built are not standard oe-core/yocto images, they are custom images that don't include busybox. Perhaps that is why I am seeing the issue and others aren't.
If you are really interested in replicating let me know and I can provide you with pointers to my image recipes.

Ya, if you could point me to the image recipe that eliminates BB and uses the discrete versions of things I'll look into this.

I can try to do it with hob, but I'm not sure of everything I'd have to select.

--Mark

Quoted 1 line

Steve

Steve Sakoman

2012-04-12 16:57 UTC #10 lore raw

Quoted 5 lines

Ya, if you could point me to the image recipe that eliminates BB and uses the discrete versions of things I'll look into this.
I can try to do it with hob, but I'm not sure of everything I'd have to select.

Here is one of the images that shows the issue:

http://www.sakoman.com/cgi-bin/gitweb.cgi?p=meta-sakoman.git;a=blob_plain;f=recipes-sakoman/images/sakoman-systemd-image.bb;h=d1f1202528af6ed73171a9a26148f18da9ee389c;hb=HEAD

It pulls in a few things from meta-oe and my meta-sakoman layer so this may be more trouble that you want to deal with . . .

Steve