rootfs.py: Remove rpm database from staging area
PATCH 1/1 rootfs.py: Remove rpm database from staging area
Rpm database in staging area is used only by createrepo. createrepo fails with the error
"rpmdb: BDB0060 PANIC: fatal region error detected" if rpm database is broken from the previous run of createrepo.
Removing the databae before running createrepo can hopefully prevent this failure to happen.
[YOCTO #6571]
--- meta/lib/oe/rootfs.py | 3 +++ 1 file changed, 3 insertions(+)
meta/lib/oe/rootfs.py +3 -0
@@ -306,6 +306,9 @@ class RpmRootfs(Rootfs): bb.utils.remove(self.image_rootfs, True) else: self.pm.recovery_packaging_data() + dbpath = os.path.join(self.d.getVar('STAGING_DIR_NATIVE', True), + 'var/lib/rpm/*') + bb.utils.remove(dbpath, recurse=True) bb.utils.remove(self.d.getVar('MULTILIB_TEMP_ROOTFS', True), True) self.pm.create_configs()
Signature · 1 line
2.1.4
Quoted 20 lines · Ed Bartosh
Rpm database in staging area is used only by createrepo. createrepo fails with the error
"rpmdb: BDB0060 PANIC: fatal region error detected" if rpm database is broken from the previous run of createrepo.Removing the databae before running createrepo can hopefully prevent this failure to happen.
[YOCTO #6571]
Signed-off-by Ed Bartosh <[email protected]>--- meta/lib/oe/rootfs.py | 3 +++ 1 file changed, 3 insertions(+)meta/lib/oe/rootfs.py +0
-0
meta/lib/oe/rootfs.py class RpmRootfs(Rootfs): · line 314
Quoted 9 lines
meta/lib/oe/rootfs.py +3
-0@@ -306,6 +306,9 @@ class RpmRootfs(Rootfs): bb.utils.remove(self.image_rootfs, True) else: self.pm.recovery_packaging_data() + dbpath = os.path.join(self.d.getVar('STAGING_DIR_NATIVE', True), + 'var/lib/rpm/*') + bb.utils.remove(dbpath, recurse=True) bb.utils.remove(self.d.getVar('MULTILIB_TEMP_ROOTFS', True), True) self.pm.crea
This patch helps me see the problem a lot. I'd never realised there was an rpm database in the native sysroot that was getting corrupted, I'd always assumed it was the target rootfs one.
I'm a little worried about what happens if you have multiple images generating at the same time as this change as above may delete something being worked on by another process.
I'm wondering why is it getting into the sysroot at all? Rather than delete it here, could we either not generate it at all, or place it somewhere in WORKDIR (so that other tasks can't see it or race against it)?
Cheers,
Richard
Quoted 35 lines · Richard Purdie
Quoted 20 lines · Ed Bartosh
Rpm database in staging area is used only by createrepo. createrepo fails with the error
"rpmdb: BDB0060 PANIC: fatal region error detected" if rpm database is broken from the previous run of createrepo.Removing the databae before running createrepo can hopefully prevent this failure to happen.
[YOCTO #6571]
Signed-off-by Ed Bartosh <[email protected]>--- meta/lib/oe/rootfs.py | 3 +++ 1 file changed, 3 insertions(+)meta/lib/oe/rootfs.py +0
-0meta/lib/oe/rootfs.py class RpmRootfs(Rootfs): · line 314
Quoted 9 lines
meta/lib/oe/rootfs.py +3
-0@@ -306,6 +306,9 @@ class RpmRootfs(Rootfs): bb.utils.remove(self.image_rootfs, True) else: self.pm.recovery_packaging_data() + dbpath = os.path.join(self.d.getVar('STAGING_DIR_NATIVE', True), + 'var/lib/rpm/*') + bb.utils.remove(dbpath, recurse=True) bb.utils.remove(self.d.getVar('MULTILIB_TEMP_ROOTFS', True), True) self.pm.creaThis patch helps me see the problem a lot. I'd never realised there was an rpm database in the native sysroot that was getting corrupted, I'd always assumed it was the target rootfs one.
Quoted 8 lines
I'm a little worried about what happens if you have multiple images generating at the same time as this change as above may delete something being worked on by another process.
I'm wondering why is it getting into the sysroot at all? Rather than delete it here, could we either not generate it at all, or place it somewhere in WORKDIR (so that other tasks can't see it or race against it)?
My guess is that either createrepo or other rpm commands are doing a lazy init on the DB. As you mentioned deleting the DB is a little worrying because if it's open in another process you -are- going to cause other failures.
It would be nice if we could find the creation points and redirect them to the current workdir. (Pseudo and it's logging might help us be able to do that.) The other option could be to create an empty DB, and simply set it and the directory containing it to read-only, no write.... (but I'm not sure that the filesystem is actually going to enforce that for the current user.)
--Mark
Quoted 4 lines
Cheers,
Richard
Quoted 35 lines · Richard Purdie
Quoted 20 lines · Ed Bartosh
Rpm database in staging area is used only by createrepo. createrepo fails with the error
"rpmdb: BDB0060 PANIC: fatal region error detected" if rpm database is broken from the previous run of createrepo.Removing the databae before running createrepo can hopefully prevent this failure to happen.
[YOCTO #6571]
Signed-off-by Ed Bartosh <[email protected]>--- meta/lib/oe/rootfs.py | 3 +++ 1 file changed, 3 insertions(+)meta/lib/oe/rootfs.py +0
-0meta/lib/oe/rootfs.py class RpmRootfs(Rootfs): · line 314
Quoted 9 lines
meta/lib/oe/rootfs.py +3
-0@@ -306,6 +306,9 @@ class RpmRootfs(Rootfs): bb.utils.remove(self.image_rootfs, True) else: self.pm.recovery_packaging_data() + dbpath = os.path.join(self.d.getVar('STAGING_DIR_NATIVE', True), + 'var/lib/rpm/*') + bb.utils.remove(dbpath, recurse=True) bb.utils.remove(self.d.getVar('MULTILIB_TEMP_ROOTFS', True), True) self.pm.creaThis patch helps me see the problem a lot. I'd never realised there was an rpm database in the native sysroot that was getting corrupted, I'd always assumed it was the target rootfs one.
Quoted 8 lines
I'm a little worried about what happens if you have multiple images generating at the same time as this change as above may delete something being worked on by another process.
I'm wondering why is it getting into the sysroot at all? Rather than delete it here, could we either not generate it at all, or place it somewhere in WORKDIR (so that other tasks can't see it or race against it)?
I did some quick looking. It appears (at least on first glance) that the call to:
rpm.TransactionSet()
is what is opening the DB. It appears to me that we can change the call slightly, and it should do what is being requested:
{ "TransactionSet", (PyCFunction) rpmts_Create, METH_VARARGS|METH_KEYWORDS,
"rpm.TransactionSet([rootDir, [db]]) -> ts\n\
- Create a transaction set.\n" },
So transaction set takes two arguments, the rootDir of the thing we're working against, and a DB path. It -should- be as simply as setting a rootDir to the WORKDIR. If that doesn't work, then both arguments will be needed.
I'd suggest just adding a "--root" option to genpkgmetadata.py in the createrepo, and maybe a "--dbpath" option as well (second argument). Then only call the function with their values IF they were defined. I.e.
if root:
if dbpath:
self.ts = rpm.TransactionSet(root, dbpath)
else:
self.ts = rpm.TransactionSet(root)
else:
self.ts = rpm.TransactionSet()
(If there is a better way to do that in python, fine.. but passing in "None" or "" won't result in the desired behavior.. since RPM is actually parsing arguments and appears like it will use the values if passed in, whatever they are.)
--Mark
Quoted 4 lines
Cheers,
Richard
Quoted 72 lines · Mark Hatle
Quoted 35 lines · Richard Purdie
Quoted 20 lines · Ed Bartosh
Rpm database in staging area is used only by createrepo. createrepo fails with the error
"rpmdb: BDB0060 PANIC: fatal region error detected" if rpm database is broken from the previous run of createrepo.Removing the databae before running createrepo can hopefully prevent this failure to happen.
[YOCTO #6571]
Signed-off-by Ed Bartosh <[email protected]>--- meta/lib/oe/rootfs.py | 3 +++ 1 file changed, 3 insertions(+)meta/lib/oe/rootfs.py +0
-0meta/lib/oe/rootfs.py class RpmRootfs(Rootfs): · line 314
Quoted 9 lines
meta/lib/oe/rootfs.py +3
-0@@ -306,6 +306,9 @@ class RpmRootfs(Rootfs): bb.utils.remove(self.image_rootfs, True) else: self.pm.recovery_packaging_data() + dbpath = os.path.join(self.d.getVar('STAGING_DIR_NATIVE', True), + 'var/lib/rpm/*') + bb.utils.remove(dbpath, recurse=True) bb.utils.remove(self.d.getVar('MULTILIB_TEMP_ROOTFS', True), True) self.pm.creaThis patch helps me see the problem a lot. I'd never realised there was an rpm database in the native sysroot that was getting corrupted, I'd always assumed it was the target rootfs one.
Quoted 8 lines
I'm a little worried about what happens if you have multiple images generating at the same time as this change as above may delete something being worked on by another process.
I'm wondering why is it getting into the sysroot at all? Rather than delete it here, could we either not generate it at all, or place it somewhere in WORKDIR (so that other tasks can't see it or race against it)?
I did some quick looking. It appears (at least on first glance) that the call to:
rpm.TransactionSet()is what is opening the DB. It appears to me that we can change the call slightly, and it should do what is being requested:
{ "TransactionSet", (PyCFunction)
rpmts_Create, METH_VARARGS|METH_KEYWORDS,
"rpm.TransactionSet([rootDir, [db]])-> ts\n\
- Create a transaction set.\n" },So transaction set takes two arguments, the rootDir of the thing we're working against, and a DB path. It -should- be as simply as setting a rootDir to the WORKDIR. If that doesn't work, then both arguments will be needed.
I'd suggest just adding a "--root" option to
genpkgmetadata.pyin the createrepo, and maybe a "--dbpath" option as well (second argument). Then only call the function with their values IF they were defined. I.e.if root:
if dbpath:
self.ts =rpm.TransactionSet(root, dbpath)
else:
self.ts =rpm.TransactionSet(root)
else:
self.ts =rpm.TransactionSet()
Quoted 4 lines
(If there is a better way to do that in python, fine.. but passing in "None" or "" won't result in the desired behavior.. since RPM is actually parsing arguments and appears like it will use the values if passed in, whatever they are.)
I was going to do something like this:
if dbpath:
rpm.addMacro("_dbpath", dbpath)
self.ts = rpm.TransactionSet()
Brief test showed that this approach works just fine.
Regards,
Ed
Quoted 78 lines · Ed Bartosh
Quoted 72 lines · Mark Hatle
Quoted 35 lines · Richard Purdie
Quoted 20 lines · Ed Bartosh
Rpm database in staging area is used only by createrepo. createrepo fails with the error
"rpmdb: BDB0060 PANIC: fatal region error detected" if rpm database is broken from the previous run of createrepo.Removing the databae before running createrepo can hopefully prevent this failure to happen.
[YOCTO #6571]
Signed-off-by Ed Bartosh <[email protected]>--- meta/lib/oe/rootfs.py | 3 +++ 1 file changed, 3 insertions(+)meta/lib/oe/rootfs.py +0
-0meta/lib/oe/rootfs.py class RpmRootfs(Rootfs): · line 314
Quoted 9 lines
meta/lib/oe/rootfs.py +3
-0@@ -306,6 +306,9 @@ class RpmRootfs(Rootfs): bb.utils.remove(self.image_rootfs, True) else: self.pm.recovery_packaging_data() + dbpath = os.path.join(self.d.getVar('STAGING_DIR_NATIVE', True), + 'var/lib/rpm/*') + bb.utils.remove(dbpath, recurse=True) bb.utils.remove(self.d.getVar('MULTILIB_TEMP_ROOTFS', True), True) self.pm.creaThis patch helps me see the problem a lot. I'd never realised there was an rpm database in the native sysroot that was getting corrupted, I'd always assumed it was the target rootfs one.
Quoted 8 lines
I'm a little worried about what happens if you have multiple images generating at the same time as this change as above may delete something being worked on by another process.
I'm wondering why is it getting into the sysroot at all? Rather than delete it here, could we either not generate it at all, or place it somewhere in WORKDIR (so that other tasks can't see it or race against it)?
I did some quick looking. It appears (at least on first glance) that the call to:
rpm.TransactionSet()is what is opening the DB. It appears to me that we can change the call slightly, and it should do what is being requested:
{ "TransactionSet", (PyCFunction)
rpmts_Create, METH_VARARGS|METH_KEYWORDS,
"rpm.TransactionSet([rootDir, [db]])-> ts\n\
- Create a transaction set.\n" },So transaction set takes two arguments, the rootDir of the thing we're working against, and a DB path. It -should- be as simply as setting a rootDir to the WORKDIR. If that doesn't work, then both arguments will be needed.
I'd suggest just adding a "--root" option to
genpkgmetadata.pyin the createrepo, and maybe a "--dbpath" option as well (second argument). Then only call the function with their values IF they were defined. I.e.if root:
if dbpath:
self.ts =rpm.TransactionSet(root, dbpath)
else:
self.ts =rpm.TransactionSet(root)
else:
self.ts =rpm.TransactionSet()Quoted 4 lines
(If there is a better way to do that in python, fine.. but passing in "None" or "" won't result in the desired behavior.. since RPM is actually parsing arguments and appears like it will use the values if passed in, whatever they are.)
Quoted 8 lines
I was going to do something like this:
if dbpath:
rpm.addMacro("_dbpath", dbpath)self.ts =
rpm.TransactionSet()Brief test showed that this approach works just fine.
Yes, that can work as well.. but the transactionSet actually has arguments for root and dbpath built in. It's a good idea to select the WORKDIR as the 'root', and the dbpath as somewhere "in the WORKDIR".
(This will prevent rpm for intentionally writing anything into the execution sysroot, and restrict it to the WORKDIR.)
--Mark
Quoted 3 lines
Regards,
Ed
Quoted 72 lines · Mark Hatle
Quoted 35 lines · Richard Purdie
Quoted 20 lines · Ed Bartosh
Rpm database in staging area is used only by createrepo. createrepo fails with the error
"rpmdb: BDB0060 PANIC: fatal region error detected" if rpm database is broken from the previous run of createrepo.Removing the databae before running createrepo can hopefully prevent this failure to happen.
[YOCTO #6571]
Signed-off-by Ed Bartosh <[email protected]>--- meta/lib/oe/rootfs.py | 3 +++ 1 file changed, 3 insertions(+)meta/lib/oe/rootfs.py +0
-0meta/lib/oe/rootfs.py class RpmRootfs(Rootfs): · line 314
Quoted 9 lines
meta/lib/oe/rootfs.py +3
-0@@ -306,6 +306,9 @@ class RpmRootfs(Rootfs): bb.utils.remove(self.image_rootfs, True) else: self.pm.recovery_packaging_data() + dbpath = os.path.join(self.d.getVar('STAGING_DIR_NATIVE', True), + 'var/lib/rpm/*') + bb.utils.remove(dbpath, recurse=True) bb.utils.remove(self.d.getVar('MULTILIB_TEMP_ROOTFS', True), True) self.pm.creaThis patch helps me see the problem a lot. I'd never realised there was an rpm database in the native sysroot that was getting corrupted, I'd always assumed it was the target rootfs one.
Quoted 8 lines
I'm a little worried about what happens if you have multiple images generating at the same time as this change as above may delete something being worked on by another process.
I'm wondering why is it getting into the sysroot at all? Rather than delete it here, could we either not generate it at all, or place it somewhere in WORKDIR (so that other tasks can't see it or race against it)?
I did some quick looking. It appears (at least on first glance) that the call to:
rpm.TransactionSet()is what is opening the DB. It appears to me that we can change the call slightly, and it should do what is being requested:
{ "TransactionSet", (PyCFunction)
rpmts_Create, METH_VARARGS|METH_KEYWORDS,
"rpm.TransactionSet([rootDir, [db]])-> ts\n\
- Create a transaction set.\n" },So transaction set takes two arguments, the rootDir of the thing we're working against, and a DB path. It -should- be as simply as setting a rootDir to the WORKDIR. If that doesn't work, then both arguments will be needed.
I'd suggest just adding a "--root" option to
genpkgmetadata.pyin the createrepo, and maybe a "--dbpath" option as well (second argument). Then only call the function with their values IF they were defined. I.e.if root:
if dbpath:
self.ts =rpm.TransactionSet(root, dbpath)
else:
self.ts =rpm.TransactionSet(root)
else:
self.ts =rpm.TransactionSet()
Quoted 4 lines
(If there is a better way to do that in python, fine.. but passing in "None" or "" won't result in the desired behavior.. since RPM is actually parsing arguments and appears like it will use the values if passed in, whatever they are.)
Second parameter is not a path. Integer is required there, so I guess it's a db mode something like that. Defining _dbpath macro looks better from my point of view as it allows to specify full path to the db directory.
Regards,
Ed
PATCH 1/1 createrepo: Implement --dbpath command line option
--dbpath option allows to specify path to the directory with rpm database. By default createrepo uses or creates rpm database in /var/lib/rpm/.
Upstream-Status: Pending
--- .../createrepo/createrepo/createrepo-dbpath.patch | 51 ++++++++++++++++++++++ .../createrepo/createrepo_0.4.11.bb | 1 + 2 files changed, 52 insertions(+) create mode 100644 meta/recipes-support/createrepo/createrepo/createrepo-dbpath.patch
meta/recipes-support/createrepo/createrepo/createrepo-dbpath.patch new file +51 -0
@@ -0,0 +1,51 @@ +--- createrepo-0.4.11.orig/genpkgmetadata.py 2015-03-30 22:18:19.904000000 +0300 ++++ createrepo-0.4.11.orig/genpkgmetadata.py 2015-03-30 22:28:49.208000000 +0300 +@@ -65,6 +65,7 @@ + -p, --pretty = output xml files in pretty format. + --update = update existing metadata (if present) + -d, --database = generate the sqlite databases. ++ --dbpath <dir> = specify path to rpm db directory. + """) + + sys.exit(retval) +@@ -72,10 +73,13 @@ + class MetaDataGenerator: + def __init__(self, cmds): + self.cmds = cmds +- self.ts = rpm.TransactionSet() + self.pkgcount = 0 + self.files = [] + ++ if self.cmds['dbpath']: ++ rpm.addMacro("_dbpath", self.cmds['dbpath']) ++ self.ts = rpm.TransactionSet() ++ + def _os_path_walk(self, top, func, arg): + """Directory tree walk with callback function. + copy of os.path.walk, fixes the link/stating problem +@@ -435,6 +439,7 @@ + cmds['dir-pattern-match'] = ['.*bin\/.*', '^\/etc\/.*'] + cmds['skip-symlinks'] = False + cmds['pkglist'] = [] ++ cmds['dbpath'] = None + + try: + gopts, argsleft = getopt.getopt(args, 'phqVvndg:s:x:u:c:o:CSi:', ['help', 'exclude=', +@@ -442,7 +447,7 @@ + 'baseurl=', 'groupfile=', 'checksum=', + 'version', 'pretty', 'split', 'outputdir=', + 'noepoch', 'checkts', 'database', 'update', +- 'skip-symlinks', 'pkglist=']) ++ 'skip-symlinks', 'pkglist=', 'dbpath=']) + except getopt.error, e: + errorprint(_('Options Error: %s.') % e) + usage() +@@ -516,6 +521,8 @@ + cmds['skip-symlinks'] = True + elif arg in ['-i', '--pkglist']: + cmds['pkglist'] = a ++ elif arg == '--dbpath': ++ cmds['dbpath'] = os.path.realpath(a) + + except ValueError, e: + errorprint(_('Options Error: %s') % e)
meta/recipes-support/createrepo/createrepo_0.4.11.bb +1 -0
@@ -14,6 +14,7 @@ SRC_URI= "http://createrepo.baseurl.org/download/${BP}.tar.gz \ file://python-scripts-should-use-interpreter-from-env.patch \ file://createrepo-rpm549.patch \ file://recommends.patch \ + file://createrepo-dbpath.patch \ file://rpm-createsolvedb.py \ "
Signature · 1 line
2.1.4
PATCH 1/1 package_manager: call createrepo with --dbpath pointing inside WORKDIR
Rpm database in staging area is used only by createrepo. createrepo fails with the error
"rpmdb: BDB0060 PANIC: fatal region error detected" if rpm database is broken during previous run of createrepo.
Made createrepo to create rpm db in $WORKDIR/rpmdb/ from scratch for every build. This should potentially fix the failure.
[YOCTO #6571]
--- meta/lib/oe/package_manager.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
meta/lib/oe/package_manager.py +5 -1
@@ -110,12 +110,16 @@ class RpmIndexer(Indexer): rpm_createrepo = bb.utils.which(os.getenv('PATH'), "createrepo") index_cmds = [] rpm_dirs_found = False + dbpath = os.path.join(self.d.getVar('WORKDIR', True), 'rpmdb') + if os.path.exists(dbpath): + bb.utils.remove(dbpath, True) for arch in archs: arch_dir = os.path.join(self.deploy_dir, arch) if not os.path.isdir(arch_dir): continue - index_cmds.append("%s --update -q %s" % (rpm_createrepo, arch_dir)) + index_cmds.append("%s --dbpath %s --update -q %s" % \ + (rpm_createrepo, dbpath, arch_dir)) rpm_dirs_found = True
Signature · 1 line
2.1.4
Re: [PATCH] package_manager: call createrepo with --dbpath pointing inside WORKDIR
Quoted 26 lines · Ed Bartosh
Rpm database in staging area is used only by createrepo. createrepo fails with the error
"rpmdb: BDB0060 PANIC: fatal region error detected" if rpm database is broken during previous run of createrepo.Made createrepo to create rpm db in $WORKDIR/rpmdb/ from scratch for every build. This should potentially fix the failure.
[YOCTO #6571]
Signed-off-by Ed Bartosh <[email protected]>--- meta/lib/oe/package_manager.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)meta/lib/oe/package_manager.py +3
-0@@ -110,12 +110,16 @@ class RpmIndexer(Indexer): rpm_createrepo = bb.utils.which(os.getenv('PATH'), "createrepo") index_cmds = [] rpm_dirs_found = False + dbpath = os.path.join(self.d.getVar('WORKDIR', True), 'rpmdb') + if os.path.exists(dbpath): + bb.utils.remove(dbpath, True)
meta/lib/oe/package_manager.py class RpmIndexer(Indexer): · line 124
Quoted 10 lines
meta/lib/oe/package_manager.py +2
-1for arch in archs: arch_dir = os.path.join(self.deploy_dir, arch) if not os.path.isdir(arch_dir): continue - index_cmds.append("%s --update -q %s" % (rpm_createrepo, arch_dir)) + index_cmds.append("%s --dbpath %s --update -q %s" % \ + (rpm_createrepo, dbpath, arch_dir)) rpm_dirs_found = True
Do we need to add arch to dbpath here to avoid races between the separate createrepo processes?
Cheers,
Richard
Re: [PATCH] package_manager: call createrepo with --dbpath pointing inside WORKDIR
In reply to Ed BartoshQuoted 24 lines · Ed Bartosh
Rpm database in staging area is used only by createrepo. createrepo fails with the error
"rpmdb: BDB0060 PANIC: fatal region error detected" if rpm database is broken during previous run of createrepo.Made createrepo to create rpm db in $WORKDIR/rpmdb/ from scratch for every build. This should potentially fix the failure.
[YOCTO #6571]
Signed-off-by Ed Bartosh <[email protected]>--- meta/lib/oe/package_manager.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)meta/lib/oe/package_manager.py +1
-0@@ -110,12 +110,16 @@ class RpmIndexer(Indexer): rpm_createrepo = bb.utils.which(os.getenv('PATH'), "createrepo") index_cmds = [] rpm_dirs_found = False + dbpath = os.path.join(self.d.getVar('WORKDIR', True), 'rpmdb')
meta/lib/oe/package_manager.py class RpmIndexer(Indexer): · line 122
Quoted 10 lines
meta/lib/oe/package_manager.py +4
-1+ if os.path.exists(dbpath): + bb.utils.remove(dbpath, True) for arch in archs: arch_dir = os.path.join(self.deploy_dir, arch) if not os.path.isdir(arch_dir): continue - index_cmds.append("%s --update -q %s" % (rpm_createrepo, arch_dir)) + index_cmds.append("%s --dbpath %s --update -q %s" % \ + (rpm_createrepo, dbpath, arch_dir))
I'd suggest that the dbpath be made arch specific.... this would eliminate any serialization that may occur with even simple DB locks.
--dbpath %s/%s ....
--Mark
Quoted 4 lines
meta/lib/oe/package_manager.py +0
-0rpm_dirs_found = True
PATCH 1/1 package_manager: call createrepo with --dbpath pointing inside WORKDIR
Rpm database in staging area is used only by createrepo. createrepo fails with the error
"rpmdb: BDB0060 PANIC: fatal region error detected" if rpm database is broken during previous run of createrepo.
Made createrepo to create rpm db in $WORKDIR/rpmdb/ from scratch for every build and architecture. This should potentially fix the failure as every run of createrepo will be using separate db.
[YOCTO #6571]
--- meta/lib/oe/package_manager.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
meta/lib/oe/package_manager.py +5 -1
@@ -111,11 +111,15 @@ class RpmIndexer(Indexer): index_cmds = [] rpm_dirs_found = False for arch in archs: + dbpath = os.path.join(self.d.getVar('WORKDIR', True), 'rpmdb', arch) + if os.path.exists(dbpath): + bb.utils.remove(dbpath, True) arch_dir = os.path.join(self.deploy_dir, arch) if not os.path.isdir(arch_dir): continue - index_cmds.append("%s --update -q %s" % (rpm_createrepo, arch_dir)) + index_cmds.append("%s --dbpath %s --update -q %s" % \ + (rpm_createrepo, dbpath, arch_dir)) rpm_dirs_found = True
Signature · 1 line
2.1.4
Re: [PATCH] createrepo: Implement --dbpath command line option
In reply to Ed BartoshHi Ed,
Quoted 2 lines · Ed Bartosh
Upstream-Status: Pending
The commit log needs just a Signed-off-by; the patch needs an explanation, Signed-off-by, and Upstream-Status. The goal is that the patch itself is self-explanatory, so should explain what it does and why, whether it went upstream and why (not), and who admits creating it in the first place.
Ross
Re: [PATCH] createrepo: Implement --dbpath command line option
Quoted 4 lines · Ross
Hi Ed,
On 31 March 2015 at 00:49, Ed Bartosh <[email protected]> wrote:
Quoted 8 lines
Quoted 2 lines
Upstream-Status: Pending
The commit log needs just a Signed-off-by; the patch needs an explanation, Signed-off-by, and Upstream-Status. The goal is that the patch itself is self-explanatory, so should explain what it does and why, whether it went upstream and why (not), and who admits creating it in the first place.
Thanks for the explanations. I'll update the patch.
Regards,
Ed
PATCH 1/1 createrepo: Implement --dbpath command line option
In reply to Burton, Ross--dbpath option can be used in cases where users don't want createrepo to use system rpm db to avoid possible collisiouns with other programs.
For bitbake builds it would be possible to specify different databases even for every createrepo run. Considering that rootfs builds can run multiple createrepo in parallel, it can help to avoid race conditions caused by accessing or creating the same rpm database by multiple createrepo instances at the same time.
--- .../createrepo/createrepo/createrepo-dbpath.patch | 51 ++++++++++++++++++++++ .../createrepo/createrepo_0.4.11.bb | 1 + 2 files changed, 52 insertions(+) create mode 100644 meta/recipes-support/createrepo/createrepo/createrepo-dbpath.patch
meta/recipes-support/createrepo/createrepo/createrepo-dbpath.patch new file +51 -0
@@ -0,0 +1,51 @@ +--- createrepo-0.4.11.orig/genpkgmetadata.py 2015-03-30 22:18:19.904000000 +0300 ++++ createrepo-0.4.11.orig/genpkgmetadata.py 2015-03-30 22:28:49.208000000 +0300 +@@ -65,6 +65,7 @@ + -p, --pretty = output xml files in pretty format. + --update = update existing metadata (if present) + -d, --database = generate the sqlite databases. ++ --dbpath <dir> = specify path to rpm db directory. + """) + + sys.exit(retval) +@@ -72,10 +73,13 @@ + class MetaDataGenerator: + def __init__(self, cmds): + self.cmds = cmds +- self.ts = rpm.TransactionSet() + self.pkgcount = 0 + self.files = [] + ++ if self.cmds['dbpath']: ++ rpm.addMacro("_dbpath", self.cmds['dbpath']) ++ self.ts = rpm.TransactionSet() ++ + def _os_path_walk(self, top, func, arg): + """Directory tree walk with callback function. + copy of os.path.walk, fixes the link/stating problem +@@ -435,6 +439,7 @@ + cmds['dir-pattern-match'] = ['.*bin\/.*', '^\/etc\/.*'] + cmds['skip-symlinks'] = False + cmds['pkglist'] = [] ++ cmds['dbpath'] = None + + try: + gopts, argsleft = getopt.getopt(args, 'phqVvndg:s:x:u:c:o:CSi:', ['help', 'exclude=', +@@ -442,7 +447,7 @@ + 'baseurl=', 'groupfile=', 'checksum=', + 'version', 'pretty', 'split', 'outputdir=', + 'noepoch', 'checkts', 'database', 'update', +- 'skip-symlinks', 'pkglist=']) ++ 'skip-symlinks', 'pkglist=', 'dbpath=']) + except getopt.error, e: + errorprint(_('Options Error: %s.') % e) + usage() +@@ -516,6 +521,8 @@ + cmds['skip-symlinks'] = True + elif arg in ['-i', '--pkglist']: + cmds['pkglist'] = a ++ elif arg == '--dbpath': ++ cmds['dbpath'] = os.path.realpath(a) + + except ValueError, e: + errorprint(_('Options Error: %s') % e)
meta/recipes-support/createrepo/createrepo_0.4.11.bb +1 -0
@@ -14,6 +14,7 @@ SRC_URI= "http://createrepo.baseurl.org/download/${BP}.tar.gz \ file://python-scripts-should-use-interpreter-from-env.patch \ file://createrepo-rpm549.patch \ file://recommends.patch \ + file://createrepo-dbpath.patch \ file://rpm-createsolvedb.py \ "
Signature · 1 line
2.1.4
Re: [PATCH] createrepo: Implement --dbpath command line option
Quoted 79 lines · Ed Bartosh
--dbpath option can be used in cases where users don't want createrepo to use system rpm db to avoid possible collisiouns with other programs.
For bitbake builds it would be possible to specify different databases even for every createrepo run. Considering that rootfs builds can run multiple createrepo in parallel, it can help to avoid race conditions caused by accessing or creating the same rpm database by multiple createrepo instances at the same time.
Signed-off-by Ed Bartosh <[email protected]>--- .../createrepo/createrepo/createrepo-dbpath.patch | 51 ++++++++++++++++++++++ .../createrepo/createrepo_0.4.11.bb | 1 + 2 files changed, 52 insertions(+) create mode 100644 meta/recipes-support/createrepo/createrepo/createrepo-dbpath.patchmeta/recipes-support/createrepo/createrepo/createrepo-dbpath.patch new file +51
-0@@ -0,0 +1,51 @@ +--- createrepo-0.4.11.orig/genpkgmetadata.py 2015-03-30 22:18:19.904000000 +0300 ++++ createrepo-0.4.11.orig/genpkgmetadata.py 2015-03-30 22:28:49.208000000 +0300 +@@ -65,6 +65,7 @@ + -p, --pretty = output xml files in pretty format. + --update = update existing metadata (if present) + -d, --database = generate the sqlite databases. ++ --dbpath <dir> = specify path to rpm db directory. + """) + + sys.exit(retval) +@@ -72,10 +73,13 @@ + class MetaDataGenerator: + def __init__(self, cmds): + self.cmds = cmds +- self.ts = rpm.TransactionSet() + self.pkgcount = 0 + self.files = [] + ++ if self.cmds['dbpath']: ++ rpm.addMacro("_dbpath", self.cmds['dbpath']) ++ self.ts = rpm.TransactionSet() ++ + def _os_path_walk(self, top, func, arg): + """Directory tree walk with callback function. + copy of os.path.walk, fixes the link/stating problem +@@ -435,6 +439,7 @@ + cmds['dir-pattern-match'] = ['.*bin\/.*', '^\/etc\/.*'] + cmds['skip-symlinks'] = False + cmds['pkglist'] = [] ++ cmds['dbpath'] = None + + try: + gopts, argsleft = getopt.getopt(args, 'phqVvndg:s:x:u:c:o:CSi:', ['help', 'exclude=', +@@ -442,7 +447,7 @@ + 'baseurl=', 'groupfile=', 'checksum=', + 'version', 'pretty', 'split', 'outputdir=', + 'noepoch', 'checkts', 'database', 'update', +- 'skip-symlinks', 'pkglist=']) ++ 'skip-symlinks', 'pkglist=', 'dbpath=']) + except getopt.error, e: + errorprint(_('Options Error: %s.') % e) + usage() +@@ -516,6 +521,8 @@ + cmds['skip-symlinks'] = True + elif arg in ['-i', '--pkglist']: + cmds['pkglist'] = a ++ elif arg == '--dbpath': ++ cmds['dbpath'] = os.path.realpath(a) + + except ValueError, e: + errorprint(_('Options Error: %s') % e)meta/recipes-support/createrepo/createrepo_0.4.11.bb +0
-0
meta/recipes-support/createrepo/createrepo_0.4.11.bb SRC_URI= "http://createrepo.baseurl.org/download/${BP}.tar.gz \ · line 20
Quoted 7 lines
meta/recipes-support/createrepo/createrepo_0.4.11.bb +1
-0@@ -14,6 +14,7 @@ SRC_URI= "http://createrepo.baseurl.org/download/${BP}.tar.gz \ file://python-scripts-should-use-interpreter-from-env.patch \ file://createrepo-rpm549.patch \ file://recommends.patch \ + file://createrepo-dbpath.patch \ file://rpm-createsolvedb.py \ "
Please, don't apply this patch. I've just sent updated one. Sorry for the confusion.
Signature · 2 lines
Regards, Ed