Recover from corrupt tdb on reboot
PATCH 1/1 Recover from corrupt tdb on reboot
This patch was created for 3.1.0. Based on inspection, I believe the current version has the same problem, but I did not actually try it. I hope you'll find the patch useful anyway.
Xen cannot work when xenstored's tdb is corrupt. When that happens somehow (and we've seen it happen), even reboot doesn't recover from it. It could: there is no state in tdb that needs to be persisted across reboots.
The appended patch arranges that tdb is removed before xenstored is started, provided it doesn't already run. This is safe, because:
* xenstored cannot be restarted. If it dies, Xen's screwed until reboot.
* /usr/sbin/xend always starts xenstored anyway.
* xenstored locks its pid-file (see write_pidfile() in tools/xenstore/xenstored_core.c), and refuses to start when it can't.
* My patch makes /usr/sbin/xend remove tdb iff it can lock the pid-file. In other words, it removes tdb only when xenstored is not running, and locks it out until it is done.
Bonus fix: it also removes stale copies of the tdb xenstored tends to leave behind when it exits uncleanly.
tools/misc/xend +19 -0
@@ -23,6 +23,8 @@ On Solaris, the daemons are SMF managed, and you should not attempt to start xend by hand. """ +import fcntl +import glob import os import os.path import sys
@@ -76,6 +78,23 @@ def check_user(): raise CheckError("invalid user") def start_xenstored(): + pidfname = "/var/run/xenstore.pid" + try: + f = open(pidfname, "a") + try: + fcntl.lockf(f, fcntl.LOCK_EX | fcntl.LOCK_NB) + rootdir = os.getenv("XENSTORED_ROOTDIR") or "/var/lib/xenstored" + for i in glob.glob(rootdir + "/tdb*"): + try: + os.unlink(i) + except: + pass + os.unlink(pidfname) + except: + pass + f.close() + except: + pass XENSTORED_TRACE = os.getenv("XENSTORED_TRACE") cmd = "xenstored --pid-file /var/run/xenstore.pid" if XENSTORED_TRACE:
Quoted 4 lines · Markus Armbruster
Xen cannot work when xenstored's tdb is corrupt. When that happens somehow (and we've seen it happen), even reboot doesn't recover from it. It could: there is no state in tdb that needs to be persisted across reboots.
We're putting the tdb in a tmpfs (in the short-term), is there a reason you can't do that too? Especially as it's way faster.
Quoted 5 lines
The appended patch arranges that tdb is removed before xenstored is started, provided it doesn't already run. This is safe, because:
* xenstored cannot be restarted. If it dies, Xen's screwed until reboot.
That will hopefully not always be the case. At that point we'll need a different solution (tmpfs or not).
regards
john
Quoted 9 lines · John Levon
Quoted 4 lines · Markus Armbruster
Xen cannot work when xenstored's tdb is corrupt. When that happens somehow (and we've seen it happen), even reboot doesn't recover from it. It could: there is no state in tdb that needs to be persisted across reboots.
We're putting the tdb in a tmpfs (in the short-term), is there a reason you can't do that too? Especially as it's way faster.
That's exactly what we're doing in Fedora 9, but this patch was really for the benefit of any existing deployment who might not want to change their setup for TDB storage. If we want to switch xen-unstable to use tmpfs by default, then this patch would likely be unnecessary....
Regards,
Dan.
Signature · 4 lines
|=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|
Quoted 5 lines · Daniel P. Berrange
Quoted 5 lines
Quoted 4 lines
Xen cannot work when xenstored's tdb is corrupt. When that happens somehow (and we've seen it happen), even reboot doesn't recover from it. It could: there is no state in tdb that needs to be persisted across reboots.
Quoted 7 lines
Quoted 2 lines
We're putting the tdb in a tmpfs (in the short-term), is there a reason you can't do that too? Especially as it's way faster.
That's exactly what we're doing in Fedora 9, but this patch was really for the benefit of any existing deployment who might not want to change their setup for TDB storage. If we want to switch xen-unstable to use tmpfs by default, then this patch would likely be unnecessary....
Keir, others? Given that xenstored restartability is quite some way off?
regards
john
Quoted 6 lines · John Levon
Quoted 4 lines
That's exactly what we're doing in Fedora 9, but this patch was really for the benefit of any existing deployment who might not want to change their setup for TDB storage. If we want to switch xen-unstable to use tmpfs by default, then this patch would likely be unnecessary....
Keir, others? Given that xenstored restartability is quite some way off?
I think this patch is sensible given where we are with xenstored right now. Not everyone might want to run with xenstored's tdb on tmpfs.
-- Keir
Quoted 9 lines · Keir Fraser
Quoted 6 lines
Quoted 4 lines
That's exactly what we're doing in Fedora 9, but this patch was really for the benefit of any existing deployment who might not want to change their setup for TDB storage. If we want to switch xen-unstable to use tmpfs by default, then this patch would likely be unnecessary....
Keir, others? Given that xenstored restartability is quite some way off?
I think this patch is sensible given where we are with xenstored right now. Not everyone might want to run with xenstored's tdb on tmpfs.
Can you think of someone who might not, and what reason they would have?
regards
john
Quoted 6 lines · John Levon
Quoted 4 lines
Quoted 1 line
Keir, others? Given that xenstored restartability is quite some way off?
I think this patch is sensible given where we are with xenstored right now. Not everyone might want to run with xenstored's tdb on tmpfs.
Can you think of someone who might not, and what reason they would have?
The main argument for taking this patch is that it fixes a real user problem that I have personally had emails about, and it doesn't look like xenstored is actually going to be changed to use tmpfs before 3.2.0. So this is a fine patch for 3.2.0.
-- Keir
Quoted 9 lines · Keir Fraser
Quoted 4 lines
Quoted 2 lines
I think this patch is sensible given where we are with xenstored right now. Not everyone might want to run with xenstored's tdb on tmpfs.
Can you think of someone who might not, and what reason they would have?
The main argument for taking this patch is that it fixes a real user problem that I have personally had emails about, and it doesn't look like xenstored is actually going to be changed to use tmpfs before 3.2.0. So this is a fine patch for 3.2.0.
That's fine, I suppose, even though the tmpfs change is much simpler, and improves things above and beyond.
regards
john