Skip to contentSkip to conversation

guest crash on 2.6.20-rc4

DISCUSSION Roland Dreier 2 participants 6 messages Jan 8, 2007 – Jan 11, 2007 KVM

lore
Share & tools
lore
#0 lore raw

I'm running a 64-bit Fedora 6 install as a guest on a host running 2.6.20-rc4 with the kvm-10 userspace release. The CPU is a Xeon 5160 and I have 6 GB of RAM. The guest is given 512 MB of memory. I left the guest idle overnight, and the makewhatis cron job seems to have triggered this:

    Unable to handle kernel paging request at ffff81000ba04000 RIP:
     [<ffffffff8025f402>] clear_page+0x16/0x44
    PGD 8063 PUD 9063 PMD 800000000ba001e3 PTE aad8a7d881d984d9
    Oops: 0003 [1] SMP
    last sysfs file: /block/hda/removable
    CPU 0
    Modules linked in: autofs4 hidp nfs lockd fscache nfs_acl rfcomm l2cap bluetooth sunrpc ip_conntrack_netbios_ns ipt_REJECT xt_state ip_conntrack nfnetlink iptable_filter ip_tables ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables dm_multipath video sbs i2c_ec i2c_core button battery asus_acpi ac ipv6 parport_pc lp parport floppy pcspkr ne2k_pci 8390 serio_raw ide_cd cdrom dm_snapshot dm_zero dm_mirror dm_mod ext3 jbd ehci_hcd ohci_hcd uhci_hcd
    Pid: 4687, comm: makewhatis Not tainted 2.6.18-1.2869.fc6 #1
    RIP: 0010:[<ffffffff8025f402>]  [<ffffffff8025f402>] clear_page+0x16/0x44
    RSP: 0018:ffff810003e85c40  EFLAGS: 00010216
    RAX: 0000000000000000 RBX: ffff8100012e9140 RCX: 000000000000003f
    RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff81000ba04000
    RBP: 0000000000000001 R08: ffff81001fdc4d8e R09: 00000000000021e6
    R10: 0000000000000000 R11: 0000000000000001 R12: ffff8100012e9100
    R13: ffff81000000b500 R14: ffff81000000c400 R15: 0000000000000001
    FS:  00002aaaaaac6db0(0000) GS:ffffffff805e4000(0000) knlGS:0000000000000000
    CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
    CR2: ffff81000ba04000 CR3: 0000000003c05000 CR4: 00000000000006e0
    Process makewhatis (pid: 4687, threadinfo ffff810003e84000, task ffff81001f3a77f0)
    Stack:  ffffffff8020a632 ffff81000000c400 0000004400000000 ffff81000000c400
     000284d000000000 0000000000000001 0000000000000001 00000000000084d0
     ffff81000000c400 00000000000084d0 ffff81000000c400 ffff81001f3a77f0
    Call Trace:
    Inexact backtrace:
     [<ffffffff8020a632>] get_page_from_freelist+0x351/0x3c4
     [<ffffffff8020f00b>] __alloc_pages+0x76/0x2c3
     [<ffffffff8022b9d3>] get_zeroed_page+0x21/0x74
     [<ffffffff8022e5be>] __pud_alloc+0x14/0x90
     [<ffffffff80208091>] copy_page_range+0x122/0x6c1
     [<ffffffff802224de>] dup_fd+0x208/0x2a8
     [<ffffffff8021f5ec>] copy_process+0xd28/0x159d
     [<ffffffff80230fd9>] do_fork+0x69/0x163
     [<ffffffff802623a4>] _spin_lock_irqsave+0x9/0xe
     [<ffffffff8025bcce>] system_call+0x7e/0x83
     [<ffffffff8025bfdb>] ptregscall_common+0x67/0xac
    
    
    Code: 48 89 07 48 89 47 08 48 89 47 10 48 89 47 18 48 89 47 20 48
    RIP  [<ffffffff8025f402>] clear_page+0x16/0x44
     RSP <ffff810003e85c40>
    CR2: ffff81000ba04000

I just let yum update the guest to the 2.6.18-1.2869.fc6 kernel, but I'm more suspicious of the MMU changes to kvm...

I don't see anything come up in the host logs when this happens.

Let me know if there is other debugging info that would be helpful.

- R.

#1 lore raw
Quoted 10 lines · Roland Dreier

I'm running a 64-bit Fedora 6 install as a guest on a host running 2.6.20-rc4 with the kvm-10 userspace release. The CPU is a Xeon 5160 and I have 6 GB of RAM. The guest is given 512 MB of memory. I left the guest idle overnight, and the makewhatis cron job seems to have triggered this:

    Unable to handle kernel paging request at ffff81000ba04000 RIP:
     [<ffffffff8025f402>] clear_page+0x16/0x44
    PGD 8063 PUD 9063 PMD 800000000ba001e3 PTE aad8a7d881d984d9

The pgd/pud/pmd entries are all correct, so it's clear the mmu is confused.

Quoted 4 lines

I just let yum update the guest to the 2.6.18-1.2869.fc6 kernel, but I'm more suspicious of the MMU changes to kvm...

Yes.

Quoted 4 lines

I don't see anything come up in the host logs when this happens.

Let me know if there is other debugging info that would be helpful.

A way to reproduce this would be nice, though I realize it's asking much.

Signature · 8 lines
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
#2 lore raw
In reply to Roland Dreier
Quoted 9 lines · Roland Dreier

I'm running a 64-bit Fedora 6 install as a guest on a host running 2.6.20-rc4 with the kvm-10 userspace release. The CPU is a Xeon 5160 and I have 6 GB of RAM. The guest is given 512 MB of memory. I left the guest idle overnight, and the makewhatis cron job seems to have triggered this:

    Unable to handle kernel paging request at ffff81000ba04000 RIP:
     [<ffffffff8025f402>] clear_page+0x16/0x44

I've managed to reproduce a bug with similar characteristics: a write fault into a present, writable kernel page. The attached patch should fix it.

Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
drivers/kvm/paging_tmpl.h +1 -1
Index: b/drivers/kvm/paging_tmpl.h
===================================================================
@@ -274,7 +274,7 @@
 	struct kvm_mmu_page *page;
 
 	if (is_writeble_pte(*shadow_ent))
-		return 0;
+		return 1;
 
 	writable_shadow = *shadow_ent & PT_SHADOW_WRITABLE_MASK;
 	if (user) {
#3 lore raw
Quoted 3 lines

I've managed to reproduce a bug with similar characteristics: a write fault into a present, writable kernel page. The attached patch should fix it.

Sorry for the delay in continuing this thread. Anyway, the oops seems to be pretty reproducible by running the makewhatis and locate db update scripts in a loop. I've applied your patch and kicked off a test run; I'll let you know if I can still get the bug to happen.

Thanks

#4 lore raw
In reply to Avi Kivity

drivers/kvm/paging_tmpl.h line 277

Quoted 3 lines
drivers/kvm/paging_tmpl.h +1 -1
 	if (is_writeble_pte(*shadow_ent))
-		return 0;
+		return 1;

With this patch, it looks like my guest is surviving the load that triggered the oops before. So I think this fixes the issue I saw as well. I assume you'll send this in for 2.6.20?

- R.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

#5 lore raw
Quoted 8 lines · Roland Dreier

drivers/kvm/paging_tmpl.h line 277

Quoted 3 lines
drivers/kvm/paging_tmpl.h +1 -1
 	if (is_writeble_pte(*shadow_ent))
-		return 0;
+		return 1;

With this patch, it looks like my guest is surviving the load that triggered the oops before. So I think this fixes the issue I saw as well. I assume you'll send this in for 2.6.20?

The patch actually replaces one bug (guest pagefaults on writable dirty ptes, under certain conditions) with another, rarer one (spinning on a user-mode pagefault on writable dirty kernel ptes). I'll do it right and re-test, then send for .20 along with a few friends.

Signature · 8 lines
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV