Skip to contentSkip to conversation

bugzilla #197 fast create/destroy BUG_ON()

PATCH In review Ryan Harper 3 participants 6 messages Sep 9, 2005 – Sep 13, 2005

lore
Share & tools
lore
#0 lore raw

I haven't made any further progress on this [1]bug, even with adding in some extra tracing.

When we put xm create following by xm destroy in a loop, eventually xen hits a hard reboot dumping out a BUG_ON()

(XEN) BUG at domain.c:1054
(XEN) CPU: 0
(XEN) EIP: e008:[<ff12a6b6>] domain_relinquish_resources+0x43/0x1c8
(XEN) EFLAGS: 00010282 CONTEXT: hypervisor

(XEN) eax: ff187fb8   ebx: ffbf1080   ecx: ffbf4000   edx: 00000000
(XEN) esi: 00000007   edi: ff103fac   ebp: ff103b0c   esp: ff103af4
(XEN) cr0: 8005003b   cr3: db629000
(XEN) ds: e010   es: e010   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from esp=ff103af4:
(XEN)    ff17ba49 ff17bb67 0000041e ff11cfa9 ffbd9094 00000000 ff103b2c ff109027
(XEN)    ffbda080 ffbda308 00000001 ffbda080 ffbda080 00000000 ff103f8c ff107f88
(XEN)    ffbda080 ff103dbc 00000000 ffbda310 00000001 00000001 ff103b5c ff13f087
(XEN)    00000003 00000000 ffbc7080 ffbda080 ff103b74 00000020 ff103b8c ff10c368
(XEN)    00000000 ffbf10d4 ff103b9c ff130211 00000001 00000001 00000001 00000004
(XEN)    ffbf1e00 ff103cc8 ff111a6c ff103c2c 00000001 00000009 ff103bbc 00000004
(XEN)    00000004 ff103fb4 ff103bcc ff1233a3 00000004 80000003 80000003 00000004
(XEN)    80000002 80000003 fd8e671c ff12fb9a ff103be8 80000004 ffbf2080 ffbf2080
(XEN)    fd750ef4 ff103fac ff103c5c 80000003 00ef0000 80000003 80000003 80000003
(XEN)    80000002 ff103c48 ff1352eb fd8e6710 ff103c34 00000020 ff103c4c 00000001
(XEN)    80000003 00000001 ffbf2080 ffbf2080 ffbf2080 80000003 80000004 80000003
(XEN)    ffbf2080 00000000 ff103c8c ff1350ed fd70c770 ffbf2080 ffbf2080 00000540
(XEN)    c9e72063 00000000 00000001 00000001 00000000 000001d1 ff103cbc ff135407
(XEN)    fedd1000 ffbf2080 00000400 00000001 00000004 ffbf1e00 ff103ddc ff111a6c
(XEN)    ff103d40 00000001 00000008 ff135de0 fd6edab0 00000001 ff103cdc ff13b552
(XEN)    20000000 00000000 0000b400 000004c4 c4b40000 00000004 00008000 00009b42
(XEN)    000007dc 00000000 f0000000 04c4b400 00000004 00000001 33ef003c 33ef003c
(XEN)    00000000 00000000 00000000 00000000 000007dc 8000003e 8000003f 8000003e
(XEN)    ffbf2080 00000000 00000000 00009b42 00008000 33ef003c 00000000 33ef003c
(XEN)    00000000 00000000 00000000 0000755a 00000000 23ef0000 ff103d7c ff12315c
(XEN) Xen call trace:
(XEN)    [<ff12a6b6>] domain_relinquish_resources+0x43/0x1c8
(XEN)    [<ff109027>] domain_kill+0x62/0x9e
(XEN)    [<ff107f88>] do_dom0_op+0x54d/0x103b
(XEN)    [<ff155d8f>] hypercall+0x8f/0xaf

the line in question is:

   BUG_ON(!cpus_empty(d->cpumask));

This says to me, bail if the domain's cpumask is NOT empty. AFAICT, the only places that cpumask for a domain are modified are:

1. startup_cpu_idle_loop() in xen/arch/x86/domain.c

   cpu_set(smp_processor_id(),v->domain->cpumask);

2. __context_switch() in xen/arch/x86/domain.c

   if ( p->domain != n->domain )
      cpu_set(cpu, n->domain->cpumask);

...

   if ( p->domain != n->domain )
      cpu_clear(cpu, p->domain->cpumask);

If we are hitting that assert, then one or more of the vcpus in the domain are still running? Any help on pointing out where I've misunderstood whats happening or good places to insert some debugging would be of great help.

1. http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=197

Signature · 5 lines
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
[email protected]
#1 lore raw

PATCH 1/1 workaround for bug#197

I don't think this is the right fix, but it does highlight the issue. While killing a domain, the vcpus are descheduled, but every now and then, one of the cpus is still running one of the vcpus, which
means d->cpumask is not empty. This triggers the BUG_ON() in
xen/arch/x86/domain.c:domain_relinquish_resources(). The patch puts in some printks and a cpu_relax() loop till the cpumask is empty before
calling domain_relinquish_resources(). With this patch, I've gone
through several thousand iterations of create/destroy without crashing.

--
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253 T/L: 678-9253
[email protected]

diffstat output:

 domain.c |    8 ++++++++
 1 files changed, 8 insertions(+)

---

xen/common/domain.c +8 -0
diff -r 413c911e5780 xen/common/domain.c
@@ -112,6 +112,14 @@
     {
         for_each_vcpu(d, v)
             sched_rem_domain(v);
+        if(!cpus_empty(d->cpumask)) {
+            printk("ACK! DOM%d still running, waiting before dying\n",
+                   d->domain_id);
+            while(!cpus_empty(d->cpumask))
+                cpu_relax();
+            printk("DOM%d cpumask clear, relinquishing resources\n", 
+                   d->domain_id);
+        }                
         domain_relinquish_resources(d);
         put_domain(d);
#2 lore raw

Re: [PATCH] workaround for bug#197

RH> With this patch, I've gone through several thousand iterations of RH> create/destroy without crashing.

The test 10_create_fastdestroy in the xm-test suite pokes this bug every time for me. After applying this patch, I get the following on a serial console while running the test:

(XEN) ACK! DOM1 still running, waiting before dying

After that, the system is completely hung.

Signature · 4 lines
Dan Smith
IBM Linux Technology Center
Open Hypervisor Team
email: [email protected]
#3 lore raw

PATCH 1/1 workaround for bug#197: second try

In reply to Ryan Harper

Thanks for trying out the last patch. While the previous workaround worked for me, it did not for others. Looking into domain_pause(), we can see that vcpu_sleep_sync() is called on each vcpu:

    /*
     * We can be sure that the VCPU is finally descheduled after the running
     * flag is cleared and the scheduler lock is released.
     */
    while ( test_bit(_VCPUF_running, &v->vcpu_flags)
            && !domain_runnable(v)
            && spin_is_locked(&schedule_data[v->processor].schedule_lock) )
       cpu_relax();

If we are to believe the comment, (which makes sense), then the while loop code is broken. That is, this function will spin until *any* of the three tests returns false rather than waiting until *all* tests are false. This patch switches the &&s to ||s and inverts the domain_runnable() check. I believe we want to spin while 1) vcpu_running flag is up 2) the domain is runnable and 3) the scheduler lock is held.

Signature · 34 lines
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
[email protected]


diffstat output:
 schedule.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

---
#
# fix logic to match comments.  ie. we want to spin until
# 1) the running flag is down, 
# 2) the domain isnt runnable (pausecnt > 0)
# 3) the scheduler lock isnt held
#
# Signed-off-by: Ryan Harper <[email protected]>
#
diff -r 413c911e5780 xen/common/schedule.c
--- a/xen/common/schedule.c	Mon Sep 12 12:48:33 2005
+++ b/xen/common/schedule.c	Tue Sep 13 09:46:36 2005
@@ -214,8 +214,8 @@
      * flag is cleared and the scheduler lock is released.
      */
     while ( test_bit(_VCPUF_running, &v->vcpu_flags)
-            && !domain_runnable(v)
-            && spin_is_locked(&schedule_data[v->processor].schedule_lock) )
+            || domain_runnable(v)
+            || spin_is_locked(&schedule_data[v->processor].schedule_lock) )
         cpu_relax();
 
     sync_vcpu_execstate(v);
#4 lore raw

Re: [PATCH] workaround for bug#197: second try

Quoted 7 lines · Ryan Harper

If we are to believe the comment, (which makes sense), then the while loop code is broken. That is, this function will spin until *any* of the three tests returns false rather than waiting until *all* tests are false. This patch switches the &&s to ||s and inverts the domain_runnable() check. I believe we want to spin while 1) vcpu_running flag is up 2) the domain is runnable and 3) the scheduler lock is held.

Good catch. Actually we want to spin only while the domain is non-runnable (just to catch the very unlikely event that someone else wakes the domain while we spin). I updated the comment and the loop predicate and checked the patch in. Thanks!

-- Keir

#5 lore raw

Re: [PATCH] workaround for bug#197: second try

In reply to Ryan Harper

RH> This patch switches the &&s to ||s and inverts the RH> domain_runnable() check. I believe we want to spin while 1) RH> vcpu_running flag is up 2) the domain is runnable and 3) the RH> scheduler lock is held.

This fixed it for me as well... Thanks Ryan!

Signature · 4 lines
Dan Smith
IBM Linux Technology Center
Open Hypervisor Team
email: [email protected]