race in xen-block-hotplug
PATCH 1/1 race in xen-block-hotplug
Hello,
we encountered a strange race condition in tools/hotplug/Linux/block, which only shows very rarely and only once for the first domain started after the host server is started. The complete details are in our Bugzilla at <https://forge.univention.org/bugzilla/show_bug.cgi?id=20481>.
In our case its Xen-4.1.3 (yes I know it's ancient, but the code is still the same in current Xen-git) and it only happens for file-backed files (in our case a ISO-image as CD-ROM).
Quoted 8 lines
# Avoid a race with the remove if the path has been deleted, or
»···# otherwise changed from "InitWait" state e.g. due to a timeoutxenbus_state=$(xenstore_read_default "$XENBUS_PATH/state" 'unknown') if [ "$xenbus_state" != '2' ] then release_lock "block" fatal "Path closed or removed during hotplug add: $XENBUS_PATH state: $xenbus_state" fi
The problem is that sometimes the other end (kernel/qemu?) is too slow and the device is still in in state 1=Initializing. If that happens, the domU stat is aborted and destroyed.
If the same VM is then started again, it works flawlessly.
That code block was added in
<http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=92e6cb5673b37bd883bdef0d0e83faf000edf61d>
by Keir Fraser.
My work-around is to delay for one more second and retry again if the state is 1=Initializing. The printout confirmed that that case actually happened.
BYtE
Philipp Hahn
Signature · 15 lines
Philipp Hahn Open Source Software Engineer Univention GmbH be open. Mary-Somerville-Str. 1 D-28359 Bremen Tel.: +49 421 22232-0 Fax : +49 421 22232-99 [email protected] http://www.univention.de/ Geschäftsführer: Peter H. Ganten HRB 20755 Amtsgericht Bremen Steuer-Nr.: 71-597-02876
# Bug #20481: Fix race during device hotplug
# Sometimes qemu/the kernel takes too long for the first domain to start. The
# hotplug script then finds the device still in state 1=Initialising and aborts. # Add an artifical delay of 1 seconds and try again.
tools/hotplug/Linux/block +8 -4
@@ -255,12 +255,16 @@ case "$command" in # Avoid a race with the remove if the path has been deleted, or # otherwise changed from "InitWait" state e.g. due to a timeout - xenbus_state=$(xenstore_read_default "$XENBUS_PATH/state" 'unknown') - if [ "$xenbus_state" != '2' ] - then + while true + do + xenbus_state=$(xenstore_read_default "$XENBUS_PATH/state" 'unknown') + case "$xenbus_state" in + 1) log notice "Path still initializing: $XENBUS_PATH" ; sleep 1 ; continue ;; + 2) break ;; + esac release_lock "block" fatal "Path closed or removed during hotplug add: $XENBUS_PATH state: $xenbus_state" - fi + done if [ "$mode" = 'w' ] && ! stat "$file" -c %A | grep -q w then
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel
Quoted 20 lines · Philipp Hahn
Hello,
we encountered a strange race condition in
tools/hotplug/Linux/block, which only shows very rarely and only once for the first domain started after the host server is started. The complete details are in our Bugzilla at <https://forge.univention.org/bugzilla/show_bug.cgi?id=20481>.In our case its Xen-4.1.3 (yes I know it's ancient, but the code is still the same in current Xen-git) and it only happens for file-backed files (in our case a ISO-image as CD-ROM).
Quoted 8 lines
# Avoid a race with the remove if the path has been deleted, or
»···# otherwise changed from "InitWait" state e.g. due to a timeoutxenbus_state=$(xenstore_read_default "$XENBUS_PATH/state" 'unknown') if [ "$xenbus_state" != '2' ] then release_lock "block" fatal "Path closed or removed during hotplug add: $XENBUS_PATH state: $xenbus_state" fi
Quoted 4 lines
The problem is that sometimes the other end (kernel/qemu?) is too slow and the device is still in in state 1=Initializing. If that happens, the domU stat is aborted and destroyed.
If the same VM is then started again, it works flawlessly.
This problem only manifests itself with xend and xl from Xen versions < 4.2, since 4.2 onwards libxl waits for the backend to switch to state 2 before launching hotplug scripts.
I would like to avoid having this kind of infinite loop in the block hotplug script, is there anyway this can be fixed in xend? (which is the only toolstack that has this problem in upstream versions).
Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel
Quoted 26 lines · Roger Pau Monné
Quoted 20 lines · Philipp Hahn
Hello,
we encountered a strange race condition in
tools/hotplug/Linux/block, which only shows very rarely and only once for the first domain started after the host server is started. The complete details are in our Bugzilla at <https://forge.univention.org/bugzilla/show_bug.cgi?id=20481>.In our case its Xen-4.1.3 (yes I know it's ancient, but the code is still the same in current Xen-git) and it only happens for file-backed files (in our case a ISO-image as CD-ROM).
Quoted 8 lines
# Avoid a race with the remove if the path has been deleted, or
»···# otherwise changed from "InitWait" state e.g. due to a timeoutxenbus_state=$(xenstore_read_default "$XENBUS_PATH/state" 'unknown') if [ "$xenbus_state" != '2' ] then release_lock "block" fatal "Path closed or removed during hotplug add: $XENBUS_PATH state: $xenbus_state" fiQuoted 4 lines
The problem is that sometimes the other end (kernel/qemu?) is too slow and the device is still in in state 1=Initializing. If that happens, the domU stat is aborted and destroyed.
If the same VM is then started again, it works flawlessly.
Quoted 7 lines
This problem only manifests itself with xend and xl from Xen versions < 4.2, since 4.2 onwards libxl waits for the backend to switch to state 2 before launching hotplug scripts.
I would like to avoid having this kind of infinite loop in the block hotplug script, is there anyway this can be fixed in xend? (which is the only toolstack that has this problem in upstream versions).
xend relies on udev running the scripts based on the kernel firing the events, so I don't think xend can fix it, at least not without quite a big change, I don't know how the xend maintainers would view that.
Does blkback fire the uevent before it has moved to state 2 though -- sounds a bit iffy to me. Perhaps there is an underlying kernel bug here?
Ian.
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel