package_manager.py: check the dest before create hard link

PATCH New [email protected] 1 participant 5 messages Oct 24, 2018 – Oct 25, 2018

lore

Share & tools

lore

[email protected]

author

2018-10-24 07:19 UTC #0 lore raw

PATCH 1/1 package_manager.py: check the dest before create hard link

From: Mingli Yu <[email protected]>

After create_packages_dir added in below commit:
85e72e1 package_manager: Filter to only rpms we depend upon

When add below line into conf/local.conf
INHERIT += "packagefeed-stability"

There comes below error when do_rootfs
Exception: FileExistsError: [Errno 17] File exists: '/$Prj/tmp/deploy/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm' -> '/$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm'

Add the check before create hard link to
avoid the above error.

Signed-off-by Mingli Yu <[email protected]>

---
 meta/lib/oe/package_manager.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

meta/lib/oe/package_manager.py +2 -1

index 2cc1c752b3..5d4739d66d 100644
@@ -703,7 +703,8 @@ def create_packages_dir(d, subrepo_dir, deploydir, taskname, filterbydependencie
                     bb.utils.mkdirhier(destdir)
                     seendirs.add(destdir)
                 try:
-                    os.link(l, dest)
+                    if not os.path.exists(dest):
+                        os.link(l, dest)
                 except OSError as err:
                     if err.errno == errno.EXDEV:
                         bb.utils.copyfile(l, dest)

Signature · 1 line

2.18.0

[email protected]

author

2018-10-24 10:36 UTC #1 lore raw

PATCH v2 1/1 package_manager.py: correct the deploydir when packagefeed-stability inherited

From: Mingli Yu <[email protected]>

After create_packages_dir added in below commit:
85e72e1 package_manager: Filter to only rpms we depend upon

When add below line into conf/local.conf
INHERIT += "packagefeed-stability"

In create_packages_dir function, there is a logic
as bb.utils.remove(subrepo_dir, recurse=True) to
clean subrepo_dir which is actually as example is /$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm.

But currently when inherit packagefeed-stability class,
the parameter deploydir passed to create_packages_dir is
still /$Prj/tmp/deploy/rpm as example, it should be
/$Prj/tmp/deploy/rpm-diff. Otherwise, there is a logic
os.link(l, dest) in create_packages_dir function will
result in below logic:
os.link("/$Prj/tmp/deploy/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm", "/$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm")

Actually doesn't clean /$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm-prediff
, need to update the deploydir to make the
logic changed as below to fix the do_rootfs failure
s.link("/$Prj/tmp/deploy/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm", "/$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm/i586/initscripts-1.0-r155.i586.rpm")

Signed-off-by Mingli Yu <[email protected]>

---
 meta/lib/oe/package_manager.py | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)

meta/lib/oe/package_manager.py +12 -4

index 2cc1c752b3..032d0d811b 100644
@@ -733,8 +733,10 @@ class RpmPM(PackageManager):
             self.primary_arch = self.d.getVar('MACHINE_ARCH')
 
         self.rpm_repo_dir = oe.path.join(self.d.getVar('WORKDIR'), rpm_repo_workdir)
-        create_packages_dir(self.d, oe.path.join(self.rpm_repo_dir, "rpm"), d.getVar("DEPLOY_DIR_RPM"), "package_write_rpm", filterbydependencies)
-
+        deploy_prepath = d.getVar("DEPLOY_DIR_RPM")
+        if bb.data.inherits_class('packagefeed-stability', d):
+            deploy_prepath = deploy_prepath + "-prediff"
+        create_packages_dir(self.d, oe.path.join(self.rpm_repo_dir, "rpm"), deploy_prepath, "package_write_rpm", filterbydependencies)
         self.saved_packaging_data = self.d.expand('${T}/saved_packaging_data/%s' % self.task_name)
         if not os.path.exists(self.d.expand('${T}/saved_packaging_data')):
             bb.utils.mkdirhier(self.d.expand('${T}/saved_packaging_data'))

@@ -1147,9 +1149,12 @@ class OpkgPM(OpkgDpkgPM):
         self.opkg_cmd = bb.utils.which(os.getenv('PATH'), "opkg")
         self.opkg_args = "--volatile-cache -f %s -t %s -o %s " % (self.config_file, self.d.expand('${T}/ipktemp/') ,target_rootfs)
         self.opkg_args += self.d.getVar("OPKG_ARGS")
+        deploy_prepath = d.getVar("DEPLOY_DIR_IPK")
+        if bb.data.inherits_class('packagefeed-stability', d):
+            deploy_prepath = deploy_prepath + "-prediff"
 
         if prepare_index:
-            create_packages_dir(self.d, self.deploy_dir, d.getVar("DEPLOY_DIR_IPK"), "package_write_ipk", filterbydependencies)
+            create_packages_dir(self.d, self.deploy_dir, deploy_prepath, "package_write_ipk", filterbydependencies)
 
         opkg_lib_dir = self.d.getVar('OPKGLIBDIR')
         if opkg_lib_dir[0] == "/":

@@ -1526,7 +1531,10 @@ class DpkgPM(OpkgDpkgPM):
         super(DpkgPM, self).__init__(d, target_rootfs)
         self.deploy_dir = oe.path.join(self.d.getVar('WORKDIR'), deb_repo_workdir)
 
-        create_packages_dir(self.d, self.deploy_dir, d.getVar("DEPLOY_DIR_DEB"), "package_write_deb", filterbydependencies)
+        deploy_prepath = d.getVar("DEPLOY_DIR_DEB")
+        if bb.data.inherits_class('packagefeed-stability', d):
+            deploy_prepath = deploy_prepath + "-prediff"
+        create_packages_dir(self.d, self.deploy_dir, deploy_prepath, "package_write_deb", filterbydependencies)
 
         if apt_conf_dir is None:
             self.apt_conf_dir = self.d.expand("${APTCONF_TARGET}/apt")

Signature · 1 line

2.18.0

Yu, Mingli

author

2018-10-24 10:39 UTC #2 lore raw

Re: [PATCH v2] package_manager.py: correct the deploydir when packagefeed-stability inherited

Quoted 24 lines · On 2018年10月24日 18:35, [email protected] wrote:

From: Mingli Yu <[email protected]>
After create_packages_dir added in below commit:
85e72e1 package_manager: Filter to only rpms we depend upon
When add below line into conf/local.conf
INHERIT += "packagefeed-stability"
There comes below error when do_rootfs
Exception: FileExistsError: [Errno 17] File exists: '/$Prj/tmp/deploy/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm' -> '/$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm'
In create_packages_dir function, there is a logic
as bb.utils.remove(subrepo_dir, recurse=True) to
clean subrepo_dir which is actually as example is /$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm.
But currently when inherit packagefeed-stability class,
the parameter deploydir passed to create_packages_dir is
still /$Prj/tmp/deploy/rpm as example, it should be
/$Prj/tmp/deploy/rpm-diff. Otherwise, there is a logic
os.link(l, dest) in create_packages_dir function will
result in below logic:
os.link("/$Prj/tmp/deploy/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm", "/$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm")

Quoted 4 lines

Actually doesn't clean /$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm-prediff
, need to update the deploydir to make the
logic changed as below to fix the do_rootfs failure
s.link("/$Prj/tmp/deploy/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm", "/$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm/i586/initscripts-1.0-r155.i586.rpm")

A typo here, resend out V3.

Thanks,

Quoted 50 lines

Signed-off-by Mingli Yu <[email protected]>

---
  meta/lib/oe/package_manager.py | 16 ++++++++++++----
  1 file changed, 12 insertions(+), 4 deletions(-)

meta/lib/oe/package_manager.py +12 -4

index 2cc1c752b3..032d0d811b 100644
@@ -733,8 +733,10 @@ class RpmPM(PackageManager):
              self.primary_arch = self.d.getVar('MACHINE_ARCH')
 
          self.rpm_repo_dir = oe.path.join(self.d.getVar('WORKDIR'), rpm_repo_workdir)
-        create_packages_dir(self.d, oe.path.join(self.rpm_repo_dir, "rpm"), d.getVar("DEPLOY_DIR_RPM"), "package_write_rpm", filterbydependencies)
-
+        deploy_prepath = d.getVar("DEPLOY_DIR_RPM")
+        if bb.data.inherits_class('packagefeed-stability', d):
+            deploy_prepath = deploy_prepath + "-prediff"
+        create_packages_dir(self.d, oe.path.join(self.rpm_repo_dir, "rpm"), deploy_prepath, "package_write_rpm", filterbydependencies)
          self.saved_packaging_data = self.d.expand('${T}/saved_packaging_data/%s' % self.task_name)
          if not os.path.exists(self.d.expand('${T}/saved_packaging_data')):
              bb.utils.mkdirhier(self.d.expand('${T}/saved_packaging_data'))

@@ -1147,9 +1149,12 @@ class OpkgPM(OpkgDpkgPM):
          self.opkg_cmd = bb.utils.which(os.getenv('PATH'), "opkg")
          self.opkg_args = "--volatile-cache -f %s -t %s -o %s " % (self.config_file, self.d.expand('${T}/ipktemp/') ,target_rootfs)
          self.opkg_args += self.d.getVar("OPKG_ARGS")
+        deploy_prepath = d.getVar("DEPLOY_DIR_IPK")
+        if bb.data.inherits_class('packagefeed-stability', d):
+            deploy_prepath = deploy_prepath + "-prediff"
 
          if prepare_index:
-            create_packages_dir(self.d, self.deploy_dir, d.getVar("DEPLOY_DIR_IPK"), "package_write_ipk", filterbydependencies)
+            create_packages_dir(self.d, self.deploy_dir, deploy_prepath, "package_write_ipk", filterbydependencies)
 
          opkg_lib_dir = self.d.getVar('OPKGLIBDIR')
          if opkg_lib_dir[0] == "/":

@@ -1526,7 +1531,10 @@ class DpkgPM(OpkgDpkgPM):
          super(DpkgPM, self).__init__(d, target_rootfs)
          self.deploy_dir = oe.path.join(self.d.getVar('WORKDIR'), deb_repo_workdir)
 
-        create_packages_dir(self.d, self.deploy_dir, d.getVar("DEPLOY_DIR_DEB"), "package_write_deb", filterbydependencies)
+        deploy_prepath = d.getVar("DEPLOY_DIR_DEB")
+        if bb.data.inherits_class('packagefeed-stability', d):
+            deploy_prepath = deploy_prepath + "-prediff"
+        create_packages_dir(self.d, self.deploy_dir, deploy_prepath, "package_write_deb", filterbydependencies)
 
          if apt_conf_dir is None:
              self.apt_conf_dir = self.d.expand("${APTCONF_TARGET}/apt")

[email protected]

author

2018-10-24 10:38 UTC #3 lore raw

PATCH v3 1/1 package_manager.py: correct the deploydir when packagefeed-stability inherited

In reply to [email protected]

From: Mingli Yu <[email protected]>

After create_packages_dir added in below commit:
85e72e1 package_manager: Filter to only rpms we depend upon

When add below line into conf/local.conf
INHERIT += "packagefeed-stability"

Actually doesn't clean /$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm-prediff
, need to update the deploydir to make the
logic changed as below to fix the do_rootfs failure
os.link("/$Prj/tmp/deploy/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm", "/$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm/i586/initscripts-1.0-r155.i586.rpm")

Signed-off-by Mingli Yu <[email protected]>

---
 meta/lib/oe/package_manager.py | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)

meta/lib/oe/package_manager.py +12 -4

index 2cc1c752b3..032d0d811b 100644
@@ -733,8 +733,10 @@ class RpmPM(PackageManager):
             self.primary_arch = self.d.getVar('MACHINE_ARCH')
 
         self.rpm_repo_dir = oe.path.join(self.d.getVar('WORKDIR'), rpm_repo_workdir)
-        create_packages_dir(self.d, oe.path.join(self.rpm_repo_dir, "rpm"), d.getVar("DEPLOY_DIR_RPM"), "package_write_rpm", filterbydependencies)
-
+        deploy_prepath = d.getVar("DEPLOY_DIR_RPM")
+        if bb.data.inherits_class('packagefeed-stability', d):
+            deploy_prepath = deploy_prepath + "-prediff"
+        create_packages_dir(self.d, oe.path.join(self.rpm_repo_dir, "rpm"), deploy_prepath, "package_write_rpm", filterbydependencies)
         self.saved_packaging_data = self.d.expand('${T}/saved_packaging_data/%s' % self.task_name)
         if not os.path.exists(self.d.expand('${T}/saved_packaging_data')):
             bb.utils.mkdirhier(self.d.expand('${T}/saved_packaging_data'))

@@ -1147,9 +1149,12 @@ class OpkgPM(OpkgDpkgPM):
         self.opkg_cmd = bb.utils.which(os.getenv('PATH'), "opkg")
         self.opkg_args = "--volatile-cache -f %s -t %s -o %s " % (self.config_file, self.d.expand('${T}/ipktemp/') ,target_rootfs)
         self.opkg_args += self.d.getVar("OPKG_ARGS")
+        deploy_prepath = d.getVar("DEPLOY_DIR_IPK")
+        if bb.data.inherits_class('packagefeed-stability', d):
+            deploy_prepath = deploy_prepath + "-prediff"
 
         if prepare_index:
-            create_packages_dir(self.d, self.deploy_dir, d.getVar("DEPLOY_DIR_IPK"), "package_write_ipk", filterbydependencies)
+            create_packages_dir(self.d, self.deploy_dir, deploy_prepath, "package_write_ipk", filterbydependencies)
 
         opkg_lib_dir = self.d.getVar('OPKGLIBDIR')
         if opkg_lib_dir[0] == "/":

@@ -1526,7 +1531,10 @@ class DpkgPM(OpkgDpkgPM):
         super(DpkgPM, self).__init__(d, target_rootfs)
         self.deploy_dir = oe.path.join(self.d.getVar('WORKDIR'), deb_repo_workdir)
 
-        create_packages_dir(self.d, self.deploy_dir, d.getVar("DEPLOY_DIR_DEB"), "package_write_deb", filterbydependencies)
+        deploy_prepath = d.getVar("DEPLOY_DIR_DEB")
+        if bb.data.inherits_class('packagefeed-stability', d):
+            deploy_prepath = deploy_prepath + "-prediff"
+        create_packages_dir(self.d, self.deploy_dir, deploy_prepath, "package_write_deb", filterbydependencies)
 
         if apt_conf_dir is None:
             self.apt_conf_dir = self.d.expand("${APTCONF_TARGET}/apt")

Signature · 1 line

2.18.0

[email protected]

author

2018-10-25 06:19 UTC #4 lore raw

PATCH v4 1/1 package_manager.py: correct the deploydir when packagefeed-stability inherited

In reply to [email protected]

From: Mingli Yu <[email protected]>

After create_packages_dir added in below commit:
85e72e1 package_manager: Filter to only rpms we depend upon

When add below line into conf/local.conf
INHERIT += "packagefeed-stability"

def create_packages_dir(d, subrepo_dir, deploydir, taskname, filterbydependencies):
[snip]
bb.utils.remove(subrepo_dir, recurse=True)
[snip]

But currently when inherit packagefeed-stability class, the deploydir should be /$Prj/tmp/deploy/rpm-prediff, not the default /$Prj/tmp/deploy/rpm.

If use /$Prj/tmp/deploy/rpm, then result in the
logic as below:
os.link("/$Prj/tmp/deploy/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm", "/$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm")

Update to the actual deploydir to guarantee the logic
as below:
os.link("/$Prj/tmp/deploy/rpm-prediff/i586/initscripts-1.0-r155.i586.rpm", "/$Prj/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo/rpm/i586/initscripts-1.0-r155.i586.rpm")

Signed-off-by Mingli Yu <[email protected]>

---
 meta/lib/oe/package_manager.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

meta/lib/oe/package_manager.py +4 -1

index 2cc1c752b3..882e7c429f 100644
@@ -690,7 +690,10 @@ def create_packages_dir(d, subrepo_dir, deploydir, taskname, filterbydependencie
             for l in f:
                 l = l.strip()
                 deploydir = os.path.normpath(deploydir)
-                dest = l.replace(deploydir, "")
+                if bb.data.inherits_class('packagefeed-stability', d):
+                    dest = l.replace(deploydir + "-prediff", "")
+                else:
+                    dest = l.replace(deploydir, "")
                 dest = subrepo_dir + dest
                 if l.endswith("/"):
                     if dest not in seendirs:

Signature · 1 line

2.18.0