oxenstored fixes -- fixes recent pvops kernel hang
PATCH v2 1/1 oxenstored fixes -- fixes recent pvops kernel hang
Currently PVHVM Linux guests after ddacf5ef684a "xen/pv-on-hvm kexec:
add xs_reset_watches to shutdown watches from old kernel" hang when run against oxenstored because it does not handle the unknown XS_RESET_WATCHES operation and does not reply.
The symptom of this issue is a hang during boot at this point:
cpu 1 spinlock event irq 70
CPU 1 irqstacks, hard=dec94000 soft=dec96000
Booting Node 0, Processors #1
smpboot cpu 1: start_ip = 99000
Initializing CPU#1
installing Xen timer for CPU 1
Brought up 2 CPUs
Total of 2 processors activated (9625.99 BogoMIPS).
NET: Registered protocol family 16
<HANG>
This series makes oxenstored handle unknown operations by returning an error indicating that the operation is unknown. I have not actually implemented support for XS_RESET_WATCHES.
I also include a patch which I've been using for some time to enable the use of oxenstored in preference to C xenstored when available.
Also included are two (more) little cleanup patches
Changes since v1:
* First two cleanup patches applied already
* Two more cleanup patches added
* Fixed warning in "handle unknown operations by returning an error to the client"
PATCH v2 1/1 oxenstored: handle unknown operations by returning an error to the client
# HG changeset patch
# User Ian Campbell <[email protected]>
# Date 1323792488 0
# Node ID 06c6ba6641640a2fa0ef6f998405f5f7c43bb258
# Parent 74ab4d0c8ebac8d9d344dab3bf440f510d23377d
oxenstored: handle unknown operations by returning an error to the client
Previous an unknown operation would be decoded as a Not_found exception which would bubble all the way up to the try ... with surrounding the call to main_loop where it would be logged and ignored.
This would leave the guest hanging waiting for a response to the invalid request.
Instead introduce a specific "Invalid" operation. Higher level functionality, such as Process.process_packet, already handles operations which are not understood with an error reply due to the final wildcard entry in Process.function_of_type but explicitly handle Invalid this way to make it clear what is going on.
tools/ocaml/libs/xb/op.ml +3 -3
@@ -19,8 +19,7 @@ type operation = Debug | Directory | Rea Transaction_end | Introduce | Release | Getdomainpath | Write | Mkdir | Rm | Setperms | Watchevent | Error | Isintroduced | - Resume | Set_target - | Restrict + Resume | Set_target | Restrict | Invalid let operation_c_mapping = [| Debug; Directory; Read; Getperms;
@@ -41,7 +40,7 @@ let array_search el a = let of_cval i = if i >= 0 && i < size then operation_c_mapping.(i) - else raise Not_found + else Invalid let to_cval op = array_search op operation_c_mapping
@@ -69,3 +68,4 @@ let to_string ty = | Resume -> "RESUME" | Set_target -> "SET_TARGET" | Restrict -> "RESTRICT" + | Invalid -> "INVALID"
tools/ocaml/libs/xb/xb.mli +1 -0
@@ -23,6 +23,7 @@ module Op : | Resume | Set_target | Restrict + | Invalid (* Not a valid wire operation *) val operation_c_mapping : operation array val size : int val array_search : 'a -> 'a array -> int
tools/ocaml/xenstored/logging.ml +1 -0
@@ -182,6 +182,7 @@ let string_of_access_type = function | Xenbus.Xb.Op.Error -> "error " | Xenbus.Xb.Op.Watchevent -> "w event " + | Xenbus.Xb.Op.Invalid -> "invalid " (* | x -> Xenbus.Xb.Op.to_string x *)
tools/ocaml/xenstored/process.ml +2 -1
@@ -324,7 +324,8 @@ let function_of_type ty = | Xenbus.Xb.Op.Resume -> reply_ack do_resume | Xenbus.Xb.Op.Set_target -> reply_ack do_set_target | Xenbus.Xb.Op.Restrict -> reply_ack do_restrict - | _ -> reply_ack do_error + | Xenbus.Xb.Op.Invalid -> reply_ack do_error + | _ -> reply_ack do_error let input_handle_error ~cons ~doms ~fct ~ty ~con ~t ~rid ~data = let reply_error e =
tools/ocaml/xenstored/xenstored.ml +1 -3
@@ -43,9 +43,7 @@ let process_connection_fds store cons do debug "closing socket connection" in let process_fdset_with fds fct = - List.iter (fun fd -> - try try_fct fct (Connections.find cons fd) - with Not_found -> ()) fds + List.iter (fun fd -> try_fct fct (Connections.find cons fd)) fds in process_fdset_with rset Process.do_input; process_fdset_with wset Process.do_output
PATCH v2 1/1 Linux/xencommons: Use oxenstored by default when available
In reply to Ian Campbell# HG changeset patch
# User Ian Campbell <[email protected]>
# Date 1323792507 0
# Node ID f4586d1a539c869e31fadc8b3c3223bd118b37fc # Parent 06c6ba6641640a2fa0ef6f998405f5f7c43bb258 Linux/xencommons: Use oxenstored by default when available
oxenstored is an ocaml implementation of the xenstore daemon. It is faster, more scalable and more reliable than the C xenstored.
In particular the transaction model in oxenstored does not involve taking a complete copy of the database and aborting on any (even non-conflicting) other change.
There is a paper on the design and implementation of oxenstored at
http://gazagnaire.org/pub/GH09.pdf which includes a performance evaluation and comparison with the C daemon etc.
tools/hotplug/Linux/init.d/sysconfig.xencommons +3 -0
@@ -1,6 +1,9 @@ # Log xenconsoled messages (cf xl dmesg) XENCONSOLED_TRACE=guest +# Select xenstored implementation +#XENSTORED=[oxenstored|xenstored] + # Log xenstored messages #XENSTORED_TRACE=[yes|on|1]
tools/hotplug/Linux/init.d/xencommons +13 -2
@@ -70,8 +70,19 @@ do_start () { rm -f "$XENSTORED_ROOTDIR"/tdb* &>/dev/null test -z "$XENSTORED_TRACE" || XENSTORED_ARGS=" -T /var/log/xen/xenstored-trace.log" - echo -n Starting xenstored... - xenstored --pid-file=/var/run/xenstored.pid $XENSTORED_ARGS + if [ -n "$XENSTORED" ] ; then + echo -n Starting $XENSTORED... + $XENSTORED --pid-file /var/run/xenstored.pid $XENSTORED_ARGS + elif [ -x /usr/sbin/oxenstored ] ; then + echo -n Starting oxenstored... + /usr/sbin/oxenstored --pid-file /var/run/xenstored.pid $XENSTORED_ARGS + elif [ -x /usr/sbin/xenstored ] ; then + echo -n Starting C xenstored... + /usr/sbin/xenstored --pid-file /var/run/xenstored.pid $XENSTORED_ARGS + else + echo "No xenstored found" + exit 1 + fi # Wait for xenstored to actually come up, timing out after 30 seconds while [ $time -lt $timeout ] && ! `xenstore-read -s / >/dev/null 2>&1` ; do
Re: [PATCH 2 of 4 V2] Linux/xencommons: Use oxenstored by default when available
Ian Campbell writes ("[Xen-devel] [PATCH 2 of 4 V2] Linux/xencommons: Use oxenstored by default when available"):
Quoted 1 line
Linux/xencommons: Use oxenstored by default when available
I've applied #1, #3, and #4. I'll wait with #2 until my test system gets a pass and push on the change to collect the oxenstored logs.
Ian.
Re: [PATCH 2 of 4 V2] Linux/xencommons: Use oxenstored by default when available
Quoted 5 lines · Ian Jackson
Ian Campbell writes ("[Xen-devel] [PATCH 2 of 4 V2] Linux/xencommons: Use oxenstored by default when available"):
Quoted 1 line
Linux/xencommons: Use oxenstored by default when available
I've applied #1, #3, and #4. I'll wait with #2 until my test system gets a pass and push on the change to collect the oxenstored logs.
Sounds good. Thanks.
Ian.
Re: [PATCH 2 of 4 V2] Linux/xencommons: Use oxenstored by default when available
Quoted 8 lines · Ian Campbell
Quoted 5 lines · Ian Jackson
Ian Campbell writes ("[Xen-devel] [PATCH 2 of 4 V2] Linux/xencommons: Use oxenstored by default when available"):
Quoted 1 line
Linux/xencommons: Use oxenstored by default when available
I've applied #1, #3, and #4. I'll wait with #2 until my test system gets a pass and push on the change to collect the oxenstored logs.
Sounds good. Thanks.
But did you see my response re: /var/log/xenstored.conf vs /var/log/xenstored.log in <[email protected]> ?
Ian.
Re: [PATCH 2 of 4 V2] Linux/xencommons: Use oxenstored by default when available
Ian Campbell writes ("Re: [Xen-devel] [PATCH 2 of 4 V2] Linux/xencommons: Use oxenstored by default when available"):
Quoted 10 lines · Ian Campbell
Quoted 5 lines · Ian Jackson
Quoted 2 lines
I've applied #1, #3, and #4. I'll wait with #2 until my test system gets a pass and push on the change to collect the oxenstored logs.
Sounds good. Thanks.
But did you see my response re:
/var/log/xenstored.confvs/var/log/xenstored.login <[email protected]> ?
Yes :-).
Ian.
Re: [PATCH 2 of 4 V2] Linux/xencommons: Use oxenstored by default when available
In reply to Ian CampbellIan Campbell writes ("[Xen-devel] [PATCH 2 of 4 V2] Linux/xencommons: Use oxenstored by default when available"):
Quoted 1 line
Linux/xencommons: Use oxenstored by default when available
PATCH v2 1/1 oxenstored: install configuration file
In reply to Ian Campbell# HG changeset patch
# User Ian Campbell <[email protected]> # Date 1323792537 0
# Node ID fd321c47d4b8e7095758e008ad3801e5d16461a7 # Parent f4586d1a539c869e31fadc8b3c3223bd118b37fc
oxenstored: install configuration file
First though:
- Move it to /etc/xen/oxenstored.conf.
- Use /var/run/xenstored.pid as default pid file
- Disable test-eagain "Randomly failed a transaction with EAGAIN. Used for testing Xs user". Doesn't sound fun by default...
tools/ocaml/xenstored/Makefile +2 -0
@@ -52,5 +52,7 @@ bins: $(PROGRAMS) install: all $(INSTALL_DIR) $(DESTDIR)$(SBINDIR) $(INSTALL_PROG) oxenstored $(DESTDIR)$(SBINDIR) + $(INSTALL_DIR) $(DESTDIR)$(XEN_CONFIG_DIR) + $(INSTALL_DATA) oxenstored.conf $(DESTDIR)$(XEN_CONFIG_DIR) include $(OCAML_TOPLEVEL)/Makefile.rules
tools/ocaml/xenstored/define.ml +1 -1
@@ -23,7 +23,7 @@ let xenstored_proc_port = "/proc/xen/xsd let xs_daemon_socket = "/var/run/xenstored/socket" let xs_daemon_socket_ro = "/var/run/xenstored/socket_ro" -let default_config_dir = "/etc/xensource" +let default_config_dir = "/etc/xen" let maxwatch = ref (50) let maxtransaction = ref (20)
tools/ocaml/xenstored/oxenstored.conf new file +35 -0
@@ -0,0 +1,35 @@ +# default xenstored config + +# Where the pid file is stored +pid-file = /var/run/xenstored.pid + +# Randomly failed a transaction with EAGAIN. Used for testing Xs user +test-eagain = false + +# Activate transaction merge support +merge-activate = true + +# Activate node permission system +perms-activate = true + +# Activate quota +quota-activate = true +quota-maxentity = 1000 +quota-maxsize = 2048 +quota-maxwatch = 100 +quota-transaction = 10 + +# Activate filed base backend +persistant = false + +# Xenstored logs +# xenstored-log-file = /var/log/xenstored.log +# xenstored-log-level = null +# xenstored-log-nb-files = 10 + +# Xenstored access logs +# access-log-file = /var/log/xenstored-access.log +# access-log-nb-lines = 13215 +# acesss-log-nb-chars = 180 +# access-log-special-ops = false +
tools/ocaml/xenstored/xenstored.conf deleted +0 -35
@@ -1,35 +0,0 @@ -# default xenstored config - -# Where the pid file is stored -pid-file = /var/run/xensource/xenstored.pid - -# Randomly failed a transaction with EAGAIN. Used for testing Xs user -test-eagain = true - -# Activate transaction merge support -merge-activate = true - -# Activate node permission system -perms-activate = true - -# Activate quota -quota-activate = true -quota-maxentity = 1000 -quota-maxsize = 2048 -quota-maxwatch = 100 -quota-transaction = 10 - -# Activate filed base backend -persistant = false - -# Xenstored logs -# xenstored-log-file = /var/log/xenstored.log -# xenstored-log-level = null -# xenstored-log-nb-files = 10 - -# Xenstored access logs -# access-log-file = /var/log/xenstored-access.log -# access-log-nb-lines = 13215 -# acesss-log-nb-chars = 180 -# access-log-special-ops = false -
tools/ocaml/xenstored/xenstored.ml +1 -1
@@ -71,7 +71,7 @@ let sighup_handler _ = let config_filename cf = match cf.config_file with | Some name -> name - | None -> Define.default_config_dir ^ "/xenstored.conf" + | None -> Define.default_config_dir ^ "/oxenstored.conf" let default_pidfile = "/var/run/xenstored.pid"
PATCH v2 1/1 oxenstored: Always log something at start of day (if logging enabled at all)
In reply to Ian Campbell# HG changeset patch
# User Ian Campbell <[email protected]>
# Date 1323792620 0
# Node ID f2e14742b2a2c1f40bc36063a1485b4072865fd1
# Parent fd321c47d4b8e7095758e008ad3801e5d16461a7
oxenstored: Always log something at start of day (if logging enabled at all)
Otherwise at the default level we rarely log anything at all.
A completely empty log file is a good sign, but only if you know you are looking in the right place...
tools/ocaml/xenstored/logging.ml +3 -0
@@ -117,6 +117,9 @@ let init_xenstored_log () = make_logger !xenstored_log_file !xenstored_log_nb_files !xenstored_log_nb_lines !xenstored_log_nb_chars ignore in + let date = string_of_date() in + logger.write ("[%s|%5s|%s] Xen Storage Daemon, version %d.%d") date "" "startup" + Define.xenstored_major Define.xenstored_minor; xenstored_logger := Some logger let xenstored_logging level key (fmt: (_,_,_,_) format4) =
tools/ocaml/xenstored/xenstored.ml +0 -3
@@ -284,9 +284,6 @@ let _ = Logging.init_access_log post_rotate end; - info "Xen Storage Daemon, version %d.%d" - Define.xenstored_major Define.xenstored_minor; - let spec_fds = (match rw_sock with None -> [] | Some x -> [ x ]) @ (match ro_sock with None -> [] | Some x -> [ x ]) @
Ian,
Quoted 2 lines · Ian Campbell
I also include a patch which I've been using for some time to enable the use of oxenstored in preference to C xenstored when available.
Can you also arrange for the test system to collect /etc/xen/oxenstored.conf, /var/log/xenstored.conf* and /var/log/xenstored-access.log* please.
Ian.
Ian Campbell writes ("Re: [Xen-devel] [PATCH 0 of 4 V2] oxenstored fixes -- fixes recent pvops kernel hang"):
Quoted 7 lines · Ian Campbell
Quoted 2 lines
I also include a patch which I've been using for some time to enable the use of oxenstored in preference to C xenstored when available.
Can you also arrange for the test system to collect
/etc/xen/oxenstored.conf,/var/log/xenstored.conf*and/var/log/xenstored-access.log*please.
I hadn't spotted these filenames in the source for oxenstored. "/var/log/xenstored.conf*" ?! omg wtf bbq
Surely these should all be in /var/log/xen.
But I have added them to the list of files to be collected.
Ian.
Quoted 11 lines · Ian Jackson
Ian Campbell writes ("Re: [Xen-devel] [PATCH 0 of 4 V2] oxenstored fixes -- fixes recent pvops kernel hang"):
Quoted 7 lines · Ian Campbell
Quoted 2 lines
I also include a patch which I've been using for some time to enable the use of oxenstored in preference to C xenstored when available.
Can you also arrange for the test system to collect
/etc/xen/oxenstored.conf,/var/log/xenstored.conf*and/var/log/xenstored-access.log*please.I hadn't spotted these filenames in the source for oxenstored. "
/var/log/xenstored.conf*" ?! omg wtf bbq
Sorry, that's my typo, I meant /var/log/xenstored.log*. The .conf name under /etc/xen is correct.
Quoted 1 line
Surely these should all be in
/var/log/xen.
Could move them I guess. I'd like to hear from Jon and the XCP chaps though. Patch would be the following, I think.
Ian.
tools/ocaml/xenstored/logging.ml +2 -2
@@ -104,7 +104,7 @@ let string_of_date () = tm.Unix.tm_hour tm.Unix.tm_min tm.Unix.tm_sec (int_of_float (1000.0 *. msec)) -let xenstored_log_file = ref "/var/log/xenstored.log" +let xenstored_log_file = ref "/var/log/xen/xenstored.log" let xenstored_log_level = ref Warn let xenstored_log_nb_files = ref 10 let xenstored_log_nb_lines = ref 13215
@@ -200,7 +200,7 @@ let sanitize_data data = String.escaped data let activate_access_log = ref true -let access_log_file = ref "/var/log/xenstored-access.log" +let access_log_file = ref "/var/log/xen/xenstored-access.log" let access_log_nb_files = ref 20 let access_log_nb_lines = ref 13215 let access_log_nb_chars = ref 180
tools/ocaml/xenstored/oxenstored.conf +2 -2
@@ -23,12 +23,12 @@ quota-transaction = 10 persistant = false # Xenstored logs -# xenstored-log-file = /var/log/xenstored.log +# xenstored-log-file = /var/log/xen/xenstored.log # xenstored-log-level = null # xenstored-log-nb-files = 10 # Xenstored access logs -# access-log-file = /var/log/xenstored-access.log +# access-log-file = /var/log/xen/xenstored-access.log # access-log-nb-lines = 13215 # acesss-log-nb-chars = 180 # access-log-special-ops = false