New: KVM guest boot crashed

https://bugzilla.kernel.org/show_bug.cgi?id=38672

Summary: KVM guest boot crashed
Product: Virtualization
Version: unspecified
Kernel Version: 3.0.0-rc5+
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: high
Priority: P1
Component: kvm
AssignedTo: [email protected]
ReportedBy: [email protected]
Regression: Yes

Windows Server 2008 R2 KVM guest crashed during boot process. This situation also occur on other linux based guests.

Bug is in: git://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git
not in: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git.

Qemu-kvm repository checkouted before 9 days works well.

Problem could be in this:

author Jan Kiszka <[email protected]>

    Mon, 27 Jun 2011 10:23:35 +0000 (12:23 +0200)
committer    Avi Kivity <[email protected]>    
    Tue, 28 Jun 2011 08:20:08 +0000 (11:20 +0300)

commit bcd4f22796ebda2934a980060ea704ebedb46173
tree f16fe58d2d4120c7b94f23ddef3afb61beb30dfc tree | snapshot
parent 59539c913383fdd3350681301b44f02fa7ee2757 commit | diff

author Jan Kiszka <[email protected]>

    Mon, 27 Jun 2011 10:22:28 +0000 (12:22 +0200)
committer    Avi Kivity <[email protected]>    
    Tue, 28 Jun 2011 08:18:58 +0000 (11:18 +0300)

commit 59539c913383fdd3350681301b44f02fa7ee2757
tree bfdf23a13004d08d04589d02d3c4c754da3dd076 tree | snapshot
parent b7496707af10ce2827d0803c9e46ca8ddc543716 commit | diff

Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

What other Linux guests precisely? None of the Windows and Linux guest I have around expose this problem.

What is your qemu command line?

Can you bisect which change precisely introduced the regression?

Thanks,
Jan

https://bugzilla.kernel.org/show_bug.cgi?id=38672

--- Comment #1 from Anonymous Emailer <[email protected]> 2011-07-02 08:24:18 ---
Reply-To: [email protected]

What other Linux guests precisely? None of the Windows and Linux guest I have around expose this problem.

What is your qemu command line?

Can you bisect which change precisely introduced the regression?

Thanks,
Jan

Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=38672

--- Comment #2 from Steve <[email protected]> 2011-07-02 09:04:56 --- (In reply to comment #1)

I have more guests typeson the same server:
MS Windows Server 2008 R2 (latest updates) Ubuntu 11.04 (2.6.38-10-virtual)
Debian wheezy/sid (2.6.38-rc4-git4-vs2.3.0.37-rc4) Gentoo (2.6.38-gentoo-r7)

Example of guest - OS Gentoo Linux:

/usr/bin/qemu-system-x86_64 --enable-kvm -name vps-25-gentoo -chroot /vservers1 -runas kvm -pidfile /var/run/kvm/vps-25-gentoo.pid -vnc a.b.c.d:0 -vga std --full-screen -smp 2 -m 12G -cpu host -mem-path /hugepages -mem-prealloc -kvm-shadow-memory 12G -daemonize -tdf -localtime -balloon virtio -net nic,model=virtio,vlan=0,macaddr=XX:XX:XX:XX:XX:XX -net tap,vhost=on,vlan=0,ifname=qtap0,script=/etc/kvm/kvm-ifup,downscript=/etc/kvm/kvm-ifdown -drive aio=native,index=0,media=disk,cache=writeback,if=virtio,boot=on,file=/vservers1/vps-25-gentoo.img -boot c

Yes, of course, i'm working on this now.

Thank you for your time.

Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=38672

--- Comment #3 from Steve <[email protected]> 2011-07-03 00:48:13 --- Here is result:

6506e4f995967b1a48cc34418c77b318df92ce35 is the first bad commit commit 6506e4f995967b1a48cc34418c77b318df92ce35

Author: Stefano Stabellini <[email protected]>
Date:   Thu May 19 18:35:44 2011 +0100

xen: remove xen_map_block and xen_unmap_block

Replace xen_map_block with qemu_map_cache with the appropriate locking and size parameters.
Replace xen_unmap_block with qemu_invalidate_entry.

:100644 100644 01f33bb2bca8ca69ffa03ca5170d1fce3ffd2fb4
e11c1dd97a62669255a35d1628f24fc4adf538fb M exec.c
:100644 100644 60f712b229b63f63f2fe9e8bf5c867cf4f031d71
8a2380a151978a8735c797529be871b338958b05 M xen-mapcache-stub.c
:100644 100644 57fe24de86b372775b2a0d4d7537f231626d594e
fac47cd9be72bf1201f21745498625fec44c4515 M xen-mapcache.c
:100644 100644 b89b8f9653a5f58e0ea710ae0db095a7355c9eb6
6216cc3be7eb68d6c53d21c96a950abcc565a1ba M xen-mapcache.h

Please could you look at it ?

Thank you for your time.

Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=38672

--- Comment #4 from Steve <[email protected]> 2011-07-03 01:00:44 --- You should have KVM guest with more than 4 GB memory.

Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=38672

--- Comment #5 from Steve <[email protected]> 2011-07-05 23:00:47 --- I tested reported bug on more servers also with kernel 3.0.0-rc6+ -> same result.

git bisect result:

6506e4f995967b1a48cc34418c77b318df92ce35 is the first bad commit commit 6506e4f995967b1a48cc34418c77b318df92ce35

Author: Stefano Stabellini <[email protected]>
Date:   Thu May 19 18:35:44 2011 +0100

xen: remove xen_map_block and xen_unmap_block

Replace xen_map_block with qemu_map_cache with the appropriate locking and size parameters.
Replace xen_unmap_block with qemu_invalidate_entry.

:100644 100644 01f33bb2bca8ca69ffa03ca5170d1fce3ffd2fb4
e11c1dd97a62669255a35d1628f24fc4adf538fb M exec.c
:100644 100644 60f712b229b63f63f2fe9e8bf5c867cf4f031d71
8a2380a151978a8735c797529be871b338958b05 M xen-mapcache-stub.c
:100644 100644 57fe24de86b372775b2a0d4d7537f231626d594e
fac47cd9be72bf1201f21745498625fec44c4515 M xen-mapcache.c
:100644 100644 b89b8f9653a5f58e0ea710ae0db095a7355c9eb6
6216cc3be7eb68d6c53d21c96a950abcc565a1ba M xen-mapcache.h

complete git bisect log:

git bisect start
# good: [d32e8d0b8d9e0ef7cf7ab2e74548982972789dfc] Merge commit 'v0.14.1' into stable-0.14
git bisect good d32e8d0b8d9e0ef7cf7ab2e74548982972789dfc
# bad: [525e3df73e40290e95743d4c8f8b64d8d9cbe021] Merge branch 'master' of
git://git.qemu.org/qemu into next
git bisect bad 525e3df73e40290e95743d4c8f8b64d8d9cbe021
# good: [2d2339f995d7176dcb2de10d162aed323a1ffbf3] Merge commit 'f487d6278f75f84378833b8c3a67443346d639dc' into upstream-merge
git bisect good 2d2339f995d7176dcb2de10d162aed323a1ffbf3
# good: [0e192fae3c79e7d2830f8b1fa694cd8e128084cf] Update version for 0.14.0-rc0
git bisect good 0e192fae3c79e7d2830f8b1fa694cd8e128084cf
# good: [075360945860ad9bdd491921954b383bf762b0e5] spice: don't call displaystate callbacks from spice server context.
git bisect good 075360945860ad9bdd491921954b383bf762b0e5
# good: [9047c0b40654ce3578c148f6754f878218569252] usb-ehci: move device/vendor/class id to qdev
git bisect good 9047c0b40654ce3578c148f6754f878218569252
# bad: [75ef849696830fc2ddeff8bb90eea5887ff50df6] esp: correctly fill bus id with requested lun
git bisect bad 75ef849696830fc2ddeff8bb90eea5887ff50df6
# bad: [d6034a3a61235042a0d79dcc1dfed0fbf461fb18] Merge remote-tracking branch 'qemu-kvm/uq/master' into staging
git bisect bad d6034a3a61235042a0d79dcc1dfed0fbf461fb18
# good: [b45a9b185120a10455859341d8035cce9b441fc8] Merge remote-tracking branch 'qemu-kvm/uq/master' into staging
git bisect good b45a9b185120a10455859341d8035cce9b441fc8
# bad: [ebed85058b6e89a5202112e9aa2abab3aa3804c3] xen: only track the linear framebuffer
git bisect bad ebed85058b6e89a5202112e9aa2abab3aa3804c3
# good: [22e1e729600dad1639329185614d094243409359] Merge branch 'cocoa-for-upstream' of git://repo.or.cz/qemu/afaerber
git bisect good 22e1e729600dad1639329185614d094243409359
# good: [c13390cd384a9564e6dded127d01ef0627b6b1c5] xen: fix qemu_map_cache with size != MCACHE_BUCKET_SIZE
git bisect good c13390cd384a9564e6dded127d01ef0627b6b1c5
# bad: [38bee5dc94ee355640b030d28f311b03ee2f13d1] exec.c: refactor cpu_physical_memory_map
git bisect bad 38bee5dc94ee355640b030d28f311b03ee2f13d1
# bad: [6506e4f995967b1a48cc34418c77b318df92ce35] xen: remove xen_map_block and xen_unmap_block
git bisect bad 6506e4f995967b1a48cc34418c77b318df92ce35
# good: [cd306087e5a9ea4091071a0a41c0ea99fac60ab0] xen: remove qemu_map_cache_unlock
git bisect good cd306087e5a9ea4091071a0a41c0ea99fac60ab0

Please could you look at ?

Thank you for your time.

Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=38672

--- Comment #6 from Steve <[email protected]> 2011-07-05 23:44:45 --- Here is bug (xen-mapcache.c):

void qemu_map_cache_init(void)
{
-> mapcache->entry = qemu_mallocz(size);

should be:

      mapcache->entry = qemu_mallocz(size*sizeof(MapCacheEntry));

}

Should somebody commit this fix ?

Thank you for your time.

Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=38672

--- Comment #7 from Steve <[email protected]> 2011-07-05 23:54:31 --- After applying above simple fix all tested guests started & running correctly.

Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=38672

--- Comment #8 from Steve <[email protected]> 2011-07-06 11:45:19 --- Should someone commit fix from comment #6 solving reported bug ?

Thank you for your time.

Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=38672

--- Comment #9 from Avi Kivity <[email protected]> 2011-07-06 16:39:39 --- Please post the fix on [email protected], with a signed-off-by line.

Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=38672

Rafael J. Wysocki <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------

                 CC|                            |[email protected],
                   |                            |[email protected],
                   |                            |[email protected]
             Blocks|                            |36912

Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=38672

Rafael J. Wysocki <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------

             Status|NEW                         |RESOLVED
         Resolution|                            |INVALID

--- Comment #10 from Rafael J. Wysocki <[email protected]> 2011-07-10 10:26:41 --- First-Bad-Commit : 6506e4f995967b1a48cc34418c77b318df92ce35

--- Comment #11 from Rafael J. Wysocki <[email protected]> 2011-07-10 11:07:58 --- Sorry, this is a quemu bug, it appears. Closing.

Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=38672

Rafael J. Wysocki <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|RESOLVED |CLOSED

Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.