persist_data: reconnect when DB is locked
cover persist_data: reconnect when DB is locked
Fixed yocto bug #1761.
The timeout value set in sqlite3 connection only applies to the very first time when accessing a locked DB file.
By reconnect to the DB in 'database is locked' exception, we can leverage the timeout value in each retry.
The following changes since commit 684c6ef2d08ad503ce68ac3cc273f1b6a8a56ac7:
Paul Eggleton (1):
bitbake/knotty: print task failure summary
are available in the git repository at:
git://git.pokylinux.org/poky-contrib llu/bug1761
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=llu/bug1761
Lianhao Lu (1):
bitbake/persist_data: Reconnect when DB is locked
bitbake/lib/bb/persist_data.py | 12 +++++++----- 1 files changed, 7 insertions(+), 5 deletions(-)
PATCH 1/1 bitbake/persist_data: Reconnect when DB is locked
[YOCTO #1761]
Reconnect to the backend Sqlite DB in 'database is locked' exception so the timeout can be leveraged in each time retry.
--- bitbake/lib/bb/persist_data.py | 12 +++++++----- 1 files changed, 7 insertions(+), 5 deletions(-)
bitbake/lib/bb/persist_data.py +7 -5
@@ -47,9 +47,10 @@ if hasattr(sqlite3, 'enable_shared_cache'): @total_ordering class SQLTable(collections.MutableMapping): """Object representing a table/domain in the database""" - def __init__(self, cursor, table): - self.cursor = cursor + def __init__(self, cachefile, table): + self.cachefile = cachefile self.table = table + self.cursor = connect(self.cachefile) self._execute("CREATE TABLE IF NOT EXISTS %s(key TEXT, value TEXT);" % table)
@@ -63,6 +64,8 @@ class SQLTable(collections.MutableMapping): except sqlite3.OperationalError as exc: if 'database is locked' in str(exc) and count < 500: count = count + 1 + self.cursor.close() + self.cursor = connect(self.cachefile) continue raise
@@ -188,7 +191,7 @@ class PersistData(object): del self.data[domain][key] def connect(database): - return sqlite3.connect(database, timeout=30, isolation_level=None) + return sqlite3.connect(database, timeout=5, isolation_level=None) def persist(domain, d): """Convenience factory for SQLTable objects based upon metadata"""
@@ -201,5 +204,4 @@ def persist(domain, d): bb.utils.mkdirhier(cachedir) cachefile = os.path.join(cachedir, "bb_persist_data.sqlite3") - connection = connect(cachefile) - return SQLTable(connection, domain) + return SQLTable(cachefile, domain)
Signature · 1 line
1.7.0.4
Richard Purdie
AppliedRe: [PATCH 1/1] bitbake/persist_data: Reconnect when DB is locked
Quoted 8 lines · Lianhao Lu
[YOCTO #1761]
Reconnect to the backend Sqlite DB in 'database is locked' exception so the timeout can be leveraged in each time retry.Signed-off-by Lianhao Lu <[email protected]>--- bitbake/lib/bb/persist_data.py | 12 +++++++----- 1 files changed, 7 insertions(+), 5 deletions(-)
Merged to master, thanks. Good to finally have a handle on why this kept coming up! :)
Cheers,
Richard
Re: [PATCH 1/1] bitbake/persist_data: Reconnect when DB is locked
In reply to Lianhao LuThis needs to be in edison-next. It cherry-pick's cleanly as an added plus (doing more testing now). Seems like it's already in denzil-next
-M
Quoted 59 lines · Lianhao Lu
[YOCTO #1761]
Reconnect to the backend Sqlite DB in 'database is locked' exception so the timeout can be leveraged in each time retry.Signed-off-by Lianhao Lu <[email protected]>---
bitbake/lib/bb/persist_data.py | 12 +++++++-----1 files changed, 7 insertions(+), 5 deletions(-)bitbake/lib/bb/persist_data.py +0
-0@@ -47,9 +47,10 @@ if hasattr(sqlite3, 'enable_shared_cache'):@total_ordering
class SQLTable(collections.MutableMapping):
"""Object representing a table/domain in the database"""
- def__init__(self, cursor, table):
- self.cursor = cursor
+ def__init__(self, cachefile, table):
+ self.cachefile = cachefile
self.table = table
+ self.cursor = connect(self.cachefile)
self._execute("CREATE TABLE IF NOT EXISTS %s(key TEXT, value TEXT);" % table)bitbake/lib/bb/persist_data.py +0
-0@@ -63,6 +64,8 @@ class SQLTable(collections.MutableMapping):except sqlite3.OperationalError as exc:
if 'database is locked' in str(exc) and count < 500:
count = count + 1
+ self.cursor.close()
+ self.cursor = connect(self.cachefile) continue
raisebitbake/lib/bb/persist_data.py +0
-0@@ -188,7 +191,7 @@ class PersistData(object):del self.data[domain][key]
def connect(database):
- returnsqlite3.connect(database, timeout=30, isolation_level=None)
+ returnsqlite3.connect(database, timeout=5, isolation_level=None)def
persist(domain, d):
"""Convenience factory for SQLTable objects based upon metadata"""bitbake/lib/bb/persist_data.py +1
-3@@ -201,5 +204,4 @@ def persist(domain, d): bb.utils.mkdirhier(cachedir) cachefile = os.path.join(cachedir, "bb_persist_data.sqlite3") - connection = connect(cachefile) - return SQLTable(connection, domain) + return SQLTable(cachefile, domain) --1.7.0.4
_______________________________________________
bitbake-devel mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel
Re: [PATCH 1/1] bitbake/persist_data: Reconnect when DB is locked
Quoted 2 lines · On 30/05/12 17:33, McClintock Matthew-B29882 wrote:
This needs to be in edison-next. It cherry-pick's cleanly as an added plus (doing more testing now). Seems like it's already in denzil-next
cherry-picked to my josh/edison branch on poky-contrib - I will try and ensure it makes it into the edison branch of the main repository promptly.
Cheers,
Joshua
Signature · 3 lines
Joshua Lock
Yocto Project
Intel Open Source Technology CentreRe: [PATCH 1/1] bitbake/persist_data: Reconnect when DB is locked
Quoted 8 lines · Joshua Lock
Quoted 3 lines · On 30/05/12 17:33, McClintock Matthew-B29882 wrote:
This needs to be in edison-next. It cherry-pick's cleanly as an added plus (doing more testing now). Seems like it's already in denzil-next
cherry-picked to my josh/edison branch on poky-contrib - I will try and ensure it makes it into the edison branch of the main repository promptly.
This has resolved my issue and I've seen no new issues after quite a bit of building.
-M