xend: do not polling vcpus info if guest state is not RUNNING or PAUSED
PATCH 1/1 xend: do not polling vcpus info if guest state is not RUNNING or PAUSED
When created new guest on NUMA server, xend tried to get the best node by calculated all vcpus info, the race is if other geust is rebooting, the guest in the list when entered find_relaxed_node(), but when call getVCPUInfo() the guest be terminated, then getVCPUInfo() will fail with below error:
[2013-09-04 20:01:26 6254] ERROR (XendDomainInfo:496) VM start failed Traceback (most recent call last):
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 482, in start
XendTask.log_progress(31, 60, self._initDomain)
File "/usr/lib64/python2.4/site-packages/xen/xend/XendTask.py", line 209, in log_progress
retval = func(*args, **kwds)
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 2918, in _initDomain
node = self._setCPUAffinity()
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 2835, in _setCPUAffinity
best_node = find_relaxed_node(candidate_node_list)[0]
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 2803, in find_relaxed_node
cpuinfo = dom.getVCPUInfo()
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1600, in getVCPUInfo
raise XendError(str(exn))
XendError: (3, 'No such process')
This patch will let find_relaxed_node() only polling the RUNNING or PAUSED guest vpus info to avoid the race.
--- tools/python/xen/xend/XendDomainInfo.py | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-)
tools/python/xen/xend/XendDomainInfo.py +2 -0
@@ -2734,6 +2734,8 @@ class XendDomainInfo: from xen.xend import XendDomain doms = XendDomain.instance().list('all') for dom in filter (lambda d: d.domid != self.domid, doms): + if dom._stateGet() not in (DOM_STATE_RUNNING,DOM_STATE_PAUSED): + continue cpuinfo = dom.getVCPUInfo() for vcpu in sxp.children(cpuinfo, 'vcpu'): if sxp.child_value(vcpu, 'online') == 0: continue
Signature · 1 line
1.7.1
Quoted 22 lines · Joe Jin
When created new guest on NUMA server, xend tried to get the best node by calculated all vcpus info, the race is if other geust is rebooting, the guest in the list when entered
find_relaxed_node(), but when callgetVCPUInfo()the guest be terminated, thengetVCPUInfo()will fail with below error:[2013-09-04 20:01:26 6254] ERROR (XendDomainInfo:496) VM start failed Traceback (most recent call last):
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 482, in startXendTask.log_progress(31, 60, self._initDomain)File "
/usr/lib64/python2.4/site-packages/xen/xend/XendTask.py", line 209, inlog_progress
retval =func(*args, **kwds)
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 2918, in_initDomain
node =self._setCPUAffinity()
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 2835, in_setCPUAffinity
best_node= find_relaxed_node(candidate_node_list)[0]
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 2803, infind_relaxed_node
cpuinfo =dom.getVCPUInfo()
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1600, in getVCPUInfo
raise XendError(str(exn))
XendError: (3, 'No such process')
Quoted 5 lines
This patch will let
find_relaxed_node()only polling the RUNNING or PAUSED guest vpus info to avoid the race.Signed-off-by Joe Jin <[email protected]>
The idea looks ok. Unfortunately, I know nothing of xend, thus I really don't feel comfortable enough to provide a formal Ack.
Basically, I don't know whether this patch is the best way to fix the issue, if there are other ways, etc, but the problem certainly exist and the solution sounds sound. :-)
Dario
Signature · 4 lines
<<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel
Quoted 35 lines · Joe Jin
When created new guest on NUMA server, xend tried to get the best node by calculated all vcpus info, the race is if other geust is rebooting, the guest in the list when entered
find_relaxed_node(), but when callgetVCPUInfo()the guest be terminated, thengetVCPUInfo()will fail with below error:[2013-09-04 20:01:26 6254] ERROR (XendDomainInfo:496) VM start failed Traceback (most recent call last):
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 482, in startXendTask.log_progress(31, 60, self._initDomain)File "
/usr/lib64/python2.4/site-packages/xen/xend/XendTask.py", line 209, inlog_progress
retval =func(*args, **kwds)
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 2918, in_initDomain
node =self._setCPUAffinity()
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 2835, in_setCPUAffinity
best_node= find_relaxed_node(candidate_node_list)[0]
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 2803, infind_relaxed_node
cpuinfo =dom.getVCPUInfo()
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1600, in getVCPUInfo
raise XendError(str(exn))
XendError: (3, 'No such process')This patch will let
find_relaxed_node()only polling the RUNNING or PAUSED guest vpus info to avoid the race.Signed-off-by Joe Jin <[email protected]>--- tools/python/xen/xend/XendDomainInfo.py | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-)tools/python/xen/xend/XendDomainInfo.py +0
-0
tools/python/xen/xend/XendDomainInfo.py class XendDomainInfo: · line 2738
Quoted 5 lines
tools/python/xen/xend/XendDomainInfo.py +2
-0@@ -2734,6 +2734,8 @@ class XendDomainInfo: from xen.xend import XendDomain doms = XendDomain.instance().list('all') for dom in filter (lambda d: d.domid != self.domid, doms): + if dom._stateGet() not in (DOM_STATE_RUNNING,DOM_STATE_PAUSED): + continue
Isn't it possible that the domain has rebooted and is no longer there between this two calls?
IMHO it's very unlikely, but there's still a window where getVCPUInfo could fail.
Quoted 4 lines
tools/python/xen/xend/XendDomainInfo.py +0
-0cpuinfo = dom.getVCPUInfo() for vcpu in sxp.children(cpuinfo, 'vcpu'): if sxp.child_value(vcpu, 'online') == 0: continue
Quoted 45 lines · Roger Pau Monné
Quoted 35 lines · Joe Jin
When created new guest on NUMA server, xend tried to get the best node by calculated all vcpus info, the race is if other geust is rebooting, the guest in the list when entered
find_relaxed_node(), but when callgetVCPUInfo()the guest be terminated, thengetVCPUInfo()will fail with below error:[2013-09-04 20:01:26 6254] ERROR (XendDomainInfo:496) VM start failed Traceback (most recent call last):
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 482, in startXendTask.log_progress(31, 60, self._initDomain)File "
/usr/lib64/python2.4/site-packages/xen/xend/XendTask.py", line 209, inlog_progress
retval =func(*args, **kwds)
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 2918, in_initDomain
node =self._setCPUAffinity()
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 2835, in_setCPUAffinity
best_node= find_relaxed_node(candidate_node_list)[0]
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 2803, infind_relaxed_node
cpuinfo =dom.getVCPUInfo()
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1600, in getVCPUInfo
raise XendError(str(exn))
XendError: (3, 'No such process')This patch will let
find_relaxed_node()only polling the RUNNING or PAUSED guest vpus info to avoid the race.Signed-off-by Joe Jin <[email protected]>--- tools/python/xen/xend/XendDomainInfo.py | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-)tools/python/xen/xend/XendDomainInfo.py +0
-0tools/python/xen/xend/XendDomainInfo.py class XendDomainInfo: · line 2738
Quoted 5 lines
tools/python/xen/xend/XendDomainInfo.py +2
-0@@ -2734,6 +2734,8 @@ class XendDomainInfo: from xen.xend import XendDomain doms = XendDomain.instance().list('all') for dom in filter (lambda d: d.domid != self.domid, doms): + if dom._stateGet() not in (DOM_STATE_RUNNING,DOM_STATE_PAUSED): + continueIsn't it possible that the domain has rebooted and is no longer there between this two calls?
Quoted 3 lines
IMHO it's very unlikely, but there's still a window where getVCPUInfo could fail.
Yes your right, this patch just reduce the window. I created a new patch for this, please comment!
[PATCH] xend: getVCPUInfo should handle died domain
When created new guest on NUMA server, xend tried to get the best node by calculated all vcpus info, the race is if other geust is rebooting, the guest in the list when entered find_relaxed_node(), but when call getVCPUInfo() the guest already be terminated, then getVCPUInfo() will fail with below error:
[2013-09-04 20:01:26 6254] ERROR (XendDomainInfo:496) VM start failed Traceback (most recent call last):
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 482, in start
XendTask.log_progress(31, 60, self._initDomain)
File "/usr/lib64/python2.4/site-packages/xen/xend/XendTask.py", line 209, in log_progress
retval = func(*args, **kwds)
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 2918, in _initDomain
node = self._setCPUAffinity()
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 2835, in _setCPUAffinity
best_node = find_relaxed_node(candidate_node_list)[0]
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 2803, in find_relaxed_node
cpuinfo = dom.getVCPUInfo()
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1600, in getVCPUInfo
raise XendError(str(exn))
XendError: (3, 'No such process')
This patch will handle the situation.
--- tools/python/xen/xend/XendDomainInfo.py | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-)
tools/python/xen/xend/XendDomainInfo.py +4 -0
@@ -34,6 +34,7 @@ import os import stat import shutil import traceback +import errno from types import StringTypes import xen.lowlevel.xc
@@ -1541,6 +1542,9 @@ class XendDomainInfo: return sxpr except RuntimeError, exn: + # Domain already died. + if exn.args[0] == errno.ESRCH: + return sxpr raise XendError(str(exn))
Signature · 1 line
1.7.1
Quoted 97 lines · Joe Jin
Quoted 45 lines · Roger Pau Monné
Quoted 35 lines · Joe Jin
When created new guest on NUMA server, xend tried to get the best node by calculated all vcpus info, the race is if other geust is rebooting, the guest in the list when entered
find_relaxed_node(), but when callgetVCPUInfo()the guest be terminated, thengetVCPUInfo()will fail with below error:[2013-09-04 20:01:26 6254] ERROR (XendDomainInfo:496) VM start failed Traceback (most recent call last):
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 482, in startXendTask.log_progress(31, 60, self._initDomain)File "
/usr/lib64/python2.4/site-packages/xen/xend/XendTask.py", line 209, inlog_progress
retval =func(*args, **kwds)
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 2918, in_initDomain
node =self._setCPUAffinity()
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 2835, in_setCPUAffinity
best_node= find_relaxed_node(candidate_node_list)[0]
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 2803, infind_relaxed_node
cpuinfo =dom.getVCPUInfo()
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1600, in getVCPUInfo
raise XendError(str(exn))
XendError: (3, 'No such process')This patch will let
find_relaxed_node()only polling the RUNNING or PAUSED guest vpus info to avoid the race.Signed-off-by Joe Jin <[email protected]>--- tools/python/xen/xend/XendDomainInfo.py | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-)tools/python/xen/xend/XendDomainInfo.py +0
-0tools/python/xen/xend/XendDomainInfo.py class XendDomainInfo: · line 2738
Quoted 5 lines
tools/python/xen/xend/XendDomainInfo.py +2
-0@@ -2734,6 +2734,8 @@ class XendDomainInfo: from xen.xend import XendDomain doms = XendDomain.instance().list('all') for dom in filter (lambda d: d.domid != self.domid, doms): + if dom._stateGet() not in (DOM_STATE_RUNNING,DOM_STATE_PAUSED): + continueIsn't it possible that the domain has rebooted and is no longer there between this two calls?
Quoted 3 lines
IMHO it's very unlikely, but there's still a window where getVCPUInfo could fail.
Yes your right, this patch just reduce the window. I created a new patch for this, please comment!
[PATCH] xend: getVCPUInfo should handle died domain
When created new guest on NUMA server, xend tried to get the best node by calculated all vcpus info, the race is if other geust is rebooting, the guest in the list when entered
find_relaxed_node(), but when callgetVCPUInfo()the guest already be terminated, thengetVCPUInfo()will fail with below error:[2013-09-04 20:01:26 6254] ERROR (XendDomainInfo:496) VM start failed Traceback (most recent call last):
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 482, in startXendTask.log_progress(31, 60, self._initDomain)File "
/usr/lib64/python2.4/site-packages/xen/xend/XendTask.py", line 209, inlog_progress
retval =func(*args, **kwds)
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 2918, in_initDomain
node =self._setCPUAffinity()
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 2835, in_setCPUAffinity
best_node= find_relaxed_node(candidate_node_list)[0]
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 2803, infind_relaxed_node
cpuinfo =dom.getVCPUInfo()
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1600, in getVCPUInfo
raise XendError(str(exn))
XendError: (3, 'No such process')This patch will handle the situation.
Signed-off-by Joe Jin <[email protected]>--- tools/python/xen/xend/XendDomainInfo.py | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-)tools/python/xen/xend/XendDomainInfo.py +1
-0@@ -34,6 +34,7 @@ import os import stat import shutil import traceback +import errno from types import StringTypes import xen.lowlevel.xc
tools/python/xen/xend/XendDomainInfo.py class XendDomainInfo: · line 1550
Quoted 9 lines
tools/python/xen/xend/XendDomainInfo.py +3
-0@@ -1541,6 +1542,9 @@ class XendDomainInfo: return sxpr except RuntimeError, exn: + # Domain already died. + if exn.args[0] == errno.ESRCH: + return sxpr raise XendError(str(exn))
Adding Matt as he has stepped up to be the bug-fix maintainer of Xend (I think? Is that correct - should that be reflected in the MAINTAINERS file?)
Quoted 4 lines
Signature · 1 line
1.7.1
Quoted 1 line · Konrad Rzeszutek Wilk
On Tue, Nov 19, 2013 at 06:41:37PM +0800, Joe Jin wrote:
[...]
Quoted 50 lines
Quoted 50 lines
Yes your right, this patch just reduce the window. I created a new patch for this, please comment!
[PATCH] xend: getVCPUInfo should handle died domain
When created new guest on NUMA server, xend tried to get the best node by calculated all vcpus info, the race is if other geust is rebooting, the guest in the list when entered
find_relaxed_node(), but when callgetVCPUInfo()the guest already be terminated, thengetVCPUInfo()will fail with below error:[2013-09-04 20:01:26 6254] ERROR (XendDomainInfo:496) VM start failed Traceback (most recent call last):
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 482, in startXendTask.log_progress(31, 60, self._initDomain)File "
/usr/lib64/python2.4/site-packages/xen/xend/XendTask.py", line 209, inlog_progress
retval =func(*args, **kwds)
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 2918, in_initDomain
node =self._setCPUAffinity()
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 2835, in_setCPUAffinity
best_node= find_relaxed_node(candidate_node_list)[0]
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 2803, infind_relaxed_node
cpuinfo =dom.getVCPUInfo()
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1600, in getVCPUInfo
raise XendError(str(exn))
XendError: (3, 'No such process')This patch will handle the situation.
Signed-off-by Joe Jin <[email protected]>--- tools/python/xen/xend/XendDomainInfo.py | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-)tools/python/xen/xend/XendDomainInfo.py +1
-0@@ -34,6 +34,7 @@ import os import stat import shutil import traceback +import errno from types import StringTypes import xen.lowlevel.xc@@ -1541,6 +1542,9 @@ class XendDomainInfo: return sxpr
Quoted 10 lines
tools/python/xen/xend/XendDomainInfo.py class XendDomainInfo: · line 1550
Quoted 7 lines
tools/python/xen/xend/XendDomainInfo.py +3
-0except RuntimeError, exn: + # Domain already died. + if exn.args[0] == errno.ESRCH: + return sxpr raise XendError(str(exn))Adding Matt as he has stepped up to be the bug-fix maintainer of Xend (I think? Is that correct - should that be reflected in the
MAINTAINERSfile?)
This should probably be handling xen.lowlevel.xc.Error. There's no guarantee that a RuntimeError will have arguments, though xen.lowlevel.xc.Error seems to always be constructed with arguments.
--msw
Quoted 64 lines · Matt Wilson
Quoted 1 line · Konrad Rzeszutek Wilk
On Tue, Nov 19, 2013 at 06:41:37PM +0800, Joe Jin wrote:
[...]
Quoted 50 lines
Quoted 50 lines
Yes your right, this patch just reduce the window. I created a new patch for this, please comment!
[PATCH] xend: getVCPUInfo should handle died domain
When created new guest on NUMA server, xend tried to get the best node by calculated all vcpus info, the race is if other geust is rebooting, the guest in the list when entered
find_relaxed_node(), but when callgetVCPUInfo()the guest already be terminated, thengetVCPUInfo()will fail with below error:[2013-09-04 20:01:26 6254] ERROR (XendDomainInfo:496) VM start failed Traceback (most recent call last):
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 482, in startXendTask.log_progress(31, 60, self._initDomain)File "
/usr/lib64/python2.4/site-packages/xen/xend/XendTask.py", line 209, inlog_progress
retval =func(*args, **kwds)
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 2918, in_initDomain
node =self._setCPUAffinity()
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 2835, in_setCPUAffinity
best_node= find_relaxed_node(candidate_node_list)[0]
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 2803, infind_relaxed_node
cpuinfo =dom.getVCPUInfo()
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1600, in getVCPUInfo
raise XendError(str(exn))
XendError: (3, 'No such process')This patch will handle the situation.
Signed-off-by Joe Jin <[email protected]>--- tools/python/xen/xend/XendDomainInfo.py | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-)tools/python/xen/xend/XendDomainInfo.py +1
-0@@ -34,6 +34,7 @@ import os import stat import shutil import traceback +import errno from types import StringTypes import xen.lowlevel.xc@@ -1541,6 +1542,9 @@ class XendDomainInfo: return sxprQuoted 10 lines
tools/python/xen/xend/XendDomainInfo.py class XendDomainInfo: · line 1550
Quoted 7 lines
tools/python/xen/xend/XendDomainInfo.py +3
-0except RuntimeError, exn: + # Domain already died. + if exn.args[0] == errno.ESRCH: + return sxpr raise XendError(str(exn))Adding Matt as he has stepped up to be the bug-fix maintainer of Xend (I think? Is that correct - should that be reflected in the
MAINTAINERSfile?)
Quoted 4 lines
This should probably be handling xen.lowlevel.xc.Error. There's no guarantee that a RuntimeError will have arguments, though xen.lowlevel.xc.Error seems to always be constructed with arguments.
Do you means when ESRCH returned to xc, generate fake vcpu info rather than raise exception?
I created a patch for this, can you please review?
Subject: xc: build fake vcpu info when domain already died
--- tools/python/xen/lowlevel/xc/xc.c | 26 ++++++++++++++++++++++---- 1 files changed, 22 insertions(+), 4 deletions(-)
tools/python/xen/lowlevel/xc/xc.c +22 -4
@@ -384,6 +384,7 @@ static PyObject *pyxc_vcpu_getinfo(XcObject *self,int rc, i;xc_cpumap_t cpumap;int nr_cpus; +int died =0 ;static char *kwd_list[] = {"domid" ,"vcpu" ,NULL };
@@ -396,8 +397,13 @@ static PyObject *pyxc_vcpu_getinfo(XcObject *self,return pyxc_error_to_exception (self->xc_handle); rc =xc_vcpu_getinfo (self->xc_handle, dom, vcpu, &info); -if ( rc <0 ) -return pyxc_error_to_exception (self->xc_handle); +if ( rc <0 ) + { +if (errno ==ESRCH ) + died =1 ; +else +return pyxc_error_to_exception (self->xc_handle); + } cpumap =xc_cpumap_alloc (self->xc_handle);if (cpumap ==NULL )
@@ -406,8 +412,20 @@ static PyObject *pyxc_vcpu_getinfo(XcObject *self, rc =xc_vcpu_getaffinity (self->xc_handle, dom, vcpu, cpumap);if ( rc <0 ) { -free (cpumap); -return pyxc_error_to_exception (self->xc_handle); +if (errno ==ESRCH ) + died =1 ; +else + { +free (cpumap); +return pyxc_error_to_exception (self->xc_handle); + } + } + +if (died) + { +memset (&info,0 ,sizeof (info)); + info.cpu_time =0.0 ; + info.cpu =-1 ; } info_dict = Py_BuildValue("{s:i,s:i,s:i,s:L,s:i}",
Signature · 1 line
1.7.1