package_manager: Run createrepo sequentially

Running multiple createrepo in parallel can cause raice conditions when creating/accessing rpm database. Createrepo fails with the error "package_manager: rpmdb: BDB0060 PANIC: fatal region error detected" in such cases.

Running createrepo sequentially should fix the race as rpm database will not be accessed by multiple createrepo at the same time.

[YOCTO #6571]

Signed-off-by Ed Bartosh <[email protected]>

---
 meta/lib/oe/package_manager.py | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)

meta/lib/oe/package_manager.py +4 -3

index c9a8084..c956116 100644
@@ -108,25 +108,22 @@ class RpmIndexer(Indexer):
         archs = archs.union(set(sdk_pkg_archs))
 
         rpm_createrepo = bb.utils.which(os.getenv('PATH'), "createrepo")
-        index_cmds = []
         rpm_dirs_found = False
         for arch in archs:
             arch_dir = os.path.join(self.deploy_dir, arch)
             if not os.path.isdir(arch_dir):
                 continue
 
-            index_cmds.append("%s --update -q %s" % (rpm_createrepo, arch_dir))
-
             rpm_dirs_found = True
 
+            result = create_index("%s --update -q %s" % (rpm_createrepo, arch_dir))
+            if result:
+                bb.fatal(result)
+
         if not rpm_dirs_found:

Quoted 47 lines · Ed Bartosh

Running multiple createrepo in parallel can cause raice conditions when creating/accessing rpm database. Createrepo fails with the error "package_manager: rpmdb: BDB0060 PANIC: fatal region error detected" in such cases.

Running createrepo sequentially should fix the race as rpm database will not be accessed by multiple createrepo at the same time.

[YOCTO #6571]

Signed-off-by Ed Bartosh <[email protected]>

---
 meta/lib/oe/package_manager.py | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)

meta/lib/oe/package_manager.py +4 -7

index c9a8084..c956116 100644
@@ -108,25 +108,22 @@ class RpmIndexer(Indexer):
         archs = archs.union(set(sdk_pkg_archs))
 
         rpm_createrepo = bb.utils.which(os.getenv('PATH'), "createrepo")
-        index_cmds = []
         rpm_dirs_found = False
         for arch in archs:
             arch_dir = os.path.join(self.deploy_dir, arch)
             if not os.path.isdir(arch_dir):
                 continue
 
-            index_cmds.append("%s --update -q %s" % (rpm_createrepo, arch_dir))
-
             rpm_dirs_found = True
 
+            result = create_index("%s --update -q %s" % (rpm_createrepo, arch_dir))
+            if result:
+                bb.fatal(result)
+
         if not rpm_dirs_found:
             bb.note("There are no packages in %s" % self.deploy_dir)
             return
 
-        result = oe.utils.multiprocess_exec(index_cmds, create_index)
-        if result:
-            bb.fatal('%s' % ('\n'.join(result)))
-
 

meta/lib/oe/package_manager.py class RpmIndexer(Indexer): · line 129

Quoted 2 lines

meta/lib/oe/package_manager.py +0 -0

 class OpkgIndexer(Indexer):
     def write_index(self):

Running these in parallel gives us a significant speed improvement. Could we instead point them at separate directories for the RPM database (within WORKDIR to address the previous issue too)?

Quoted 48 lines · Richard Purdie

Quoted 47 lines · Ed Bartosh

Running multiple createrepo in parallel can cause raice conditions when creating/accessing rpm database. Createrepo fails with the error "package_manager: rpmdb: BDB0060 PANIC: fatal region error detected" in such cases.

Running createrepo sequentially should fix the race as rpm database will not be accessed by multiple createrepo at the same time.

[YOCTO #6571]

Signed-off-by Ed Bartosh <[email protected]>

---
 meta/lib/oe/package_manager.py | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)

meta/lib/oe/package_manager.py +4 -7

index c9a8084..c956116 100644
@@ -108,25 +108,22 @@ class RpmIndexer(Indexer):
         archs = archs.union(set(sdk_pkg_archs))
 
         rpm_createrepo = bb.utils.which(os.getenv('PATH'), "createrepo")
-        index_cmds = []
         rpm_dirs_found = False
         for arch in archs:
             arch_dir = os.path.join(self.deploy_dir, arch)
             if not os.path.isdir(arch_dir):
                 continue
 
-            index_cmds.append("%s --update -q %s" % (rpm_createrepo, arch_dir))
-
             rpm_dirs_found = True
 
+            result = create_index("%s --update -q %s" % (rpm_createrepo, arch_dir))
+            if result:
+                bb.fatal(result)
+
         if not rpm_dirs_found:
             bb.note("There are no packages in %s" % self.deploy_dir)
             return
 
-        result = oe.utils.multiprocess_exec(index_cmds, create_index)
-        if result:
-            bb.fatal('%s' % ('\n'.join(result)))
-
 

Quoted 7 lines

meta/lib/oe/package_manager.py class RpmIndexer(Indexer): · line 129

Quoted 2 lines

meta/lib/oe/package_manager.py +0 -0

 class OpkgIndexer(Indexer):
     def write_index(self):

Running these in parallel gives us a significant speed improvement. Could we instead point them at separate directories for the RPM database (within WORKDIR to address the previous issue too)?

I didn't see any noticeable performance gain at least for minimal image. I measured the time of bitbake build before and after this change. It was practically the same.