Skip to contentSkip to conversation

2.6.38.1 general protection fault

DISCUSSION Tomasz Chmielewski 5 participants 13 messages Mar 25, 2011 – Apr 20, 2011 KVM

lore
Share & tools
lore
#0 lore raw

I got this on a 2.6.38.1 system which (I think) had some problem accessing guest image on a btrfs filesystem.

general protection fault: 0000 [#1] SMP 
last sysfs file: /sys/kernel/uevent_seqnum
CPU 0 
Modules linked in: ipt_MASQUERADE vhost_net kvm_intel kvm iptable_filter xt_tcpudp iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables bridge stp btrfs zlib_deflate crc32c libcrc32c coretemp f71882fg snd_pcm snd_timer snd soundcore i2c_i801 snd_page_alloc tpm_tis tpm tpm_bios pcspkr i7core_edac edac_core r8169 mii raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov raid6_pq async_tx raid1 raid0 ahci libahci sata_nv sata_sil sata_via 3w_9xxx 3w_xxxx [last unloaded: scsi_wait_scan]

Pid: 10199, comm: kvm Not tainted 2.6.38.1 #1 MSI MS-7522/MSI X58 Pro-E (MS-7522)

RIP: 0010:[<ffffffffa02cae20>]  [<ffffffffa02cae20>] kvm_unmap_rmapp+0x20/0x70 [kvm]
RSP: 0018:ffff880508ee9bf0  EFLAGS: 00010202
RAX: 00008805d6b087f8 RBX: ffff8805b7b10000 RCX: 0000000000000050
RDX: 0000000000000000 RSI: 00008805d6b087f8 RDI: ffff8805b7b10000
RBP: ffff880508ee9c10 R08: ffff8801061d4000 R09: ffffc9001f19aff0
R10: 0000000000000030 R11: 0000000000000000 R12: 0000000000000000
R13: ffffc9001f19aff8 R14: 0000000000000060 R15: ffff8801061d4000
FS:  00007f7ca25d6730(0000) GS:ffff8800bf400000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000462b10 CR3: 00000003ac47f000 CR4: 00000000000026e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kvm (pid: 10199, threadinfo ffff880508ee8000, task ffff88001b5a5b00)
Stack:
 ffffffffffffffcf 00000000000220ff 0000000000000001 ffff8801061d4050
 ffff880508ee9c80 ffffffffa02c8a54 0000000000000030 ffffffffa02cae00
 0000000000000000 00007f7c80a2b000 ffff8805b7b10000 0000000000000001
Call Trace:
 [<ffffffffa02c8a54>] kvm_handle_hva+0xb4/0x170 [kvm]
 [<ffffffffa02cae00>] ? kvm_unmap_rmapp+0x0/0x70 [kvm]
 [<ffffffffa02c8b27>] kvm_unmap_hva+0x17/0x20 [kvm]
 [<ffffffffa02b1e72>] kvm_mmu_notifier_invalidate_range_start+0x62/0xb0 [kvm]
 [<ffffffff8113ea11>] __mmu_notifier_invalidate_range_start+0x51/0x70
 [<ffffffff8111e2c1>] copy_page_range+0x3b1/0x460
 [<ffffffff812c5628>] ? rb_insert_color+0x98/0x140
 [<ffffffff81060cdc>] dup_mm+0x2fc/0x500
 [<ffffffff810617fe>] copy_process+0x8be/0x11b0
 [<ffffffff81062165>] do_fork+0x75/0x350
 [<ffffffff81177bcd>] ? mntput+0x1d/0x40
 [<ffffffff8115b095>] ? fput+0x1e5/0x270
 [<ffffffff815aa7f5>] ? _raw_spin_lock_irq+0x15/0x20
 [<ffffffff81075141>] ? sigprocmask+0x91/0x110
 [<ffffffff81014ab8>] sys_clone+0x28/0x30
 [<ffffffff8100c3e3>] stub_clone+0x13/0x20
 [<ffffffff8100c0c2>] ? system_call_fastpath+0x16/0x1b
Code: 49 89 01 eb 91 66 0f 1f 44 00 00 55 48 89 e5 41 55 41 54 53 48 83 ec 08 0f 1f 44 00 00 45 31 e4 48 89 fb 49 89 f5 eb 1d 0f 1f 00 <f6> 06 01 74 38 48 8b 15 a4 66 02 00 48 89 df 41 bc 01 00 00 00 
RIP  [<ffffffffa02cae20>] kvm_unmap_rmapp+0x20/0x70 [kvm]
 RSP <ffff880508ee9bf0>
---[ end trace 85201a339b7635fc ]---
Signature · 2 lines
Tomasz Chmielewski
http://wpkg.org
#1 lore raw
Quoted 40 lines · Tomasz Chmielewski

I got this on a 2.6.38.1 system which (I think) had some problem accessing guest image on a btrfs filesystem.

general protection fault: 0000 [#1] SMP
last sysfs file: /sys/kernel/uevent_seqnum
CPU 0
Modules linked in: ipt_MASQUERADE vhost_net kvm_intel kvm iptable_filter xt_tcpudp iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables x_tables bridge stp btrfs zlib_deflate crc32c libcrc32c coretemp f71882fg snd_pcm snd_timer snd soundcore i2c_i801 snd_page_alloc tpm_tis tpm tpm_bios pcspkr i7core_edac edac_core r8169 mii raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov raid6_pq async_tx raid1 raid0 ahci libahci sata_nv sata_sil sata_via 3w_9xxx 3w_xxxx [last unloaded: scsi_wait_scan]

Pid: 10199, comm: kvm Not tainted 2.6.38.1 #1 MSI MS-7522/MSI X58 Pro-E (MS-7522)

RIP: 0010:[<ffffffffa02cae20>]  [<ffffffffa02cae20>] kvm_unmap_rmapp+0x20/0x70 [kvm]
RSP: 0018:ffff880508ee9bf0  EFLAGS: 00010202
RAX: 00008805d6b087f8 RBX: ffff8805b7b10000 RCX: 0000000000000050
RDX: 0000000000000000 RSI: 00008805d6b087f8 RDI: ffff8805b7b10000
RBP: ffff880508ee9c10 R08: ffff8801061d4000 R09: ffffc9001f19aff0
R10: 0000000000000030 R11: 0000000000000000 R12: 0000000000000000
R13: ffffc9001f19aff8 R14: 0000000000000060 R15: ffff8801061d4000
FS:  00007f7ca25d6730(0000) GS:ffff8800bf400000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000462b10 CR3: 00000003ac47f000 CR4: 00000000000026e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kvm (pid: 10199, threadinfo ffff880508ee8000, task ffff88001b5a5b00)
Stack:
  ffffffffffffffcf 00000000000220ff 0000000000000001 ffff8801061d4050
  ffff880508ee9c80 ffffffffa02c8a54 0000000000000030 ffffffffa02cae00
  0000000000000000 00007f7c80a2b000 ffff8805b7b10000 0000000000000001
Call Trace:
  [<ffffffffa02c8a54>] kvm_handle_hva+0xb4/0x170 [kvm]
  [<ffffffffa02cae00>] ? kvm_unmap_rmapp+0x0/0x70 [kvm]
  [<ffffffffa02c8b27>] kvm_unmap_hva+0x17/0x20 [kvm]
  [<ffffffffa02b1e72>] kvm_mmu_notifier_invalidate_range_start+0x62/0xb0 [kvm]
  [<ffffffff8113ea11>] __mmu_notifier_invalidate_range_start+0x51/0x70
  [<ffffffff8111e2c1>] copy_page_range+0x3b1/0x460
  [<ffffffff812c5628>] ? rb_insert_color+0x98/0x140
  [<ffffffff81060cdc>] dup_mm+0x2fc/0x500
  [<ffffffff810617fe>] copy_process+0x8be/0x11b0
  [<ffffffff81062165>] do_fork+0x75/0x350
  [<ffffffff81177bcd>] ? mntput+0x1d/0x40
  [<ffffffff8115b095>] ? fput+0x1e5/0x270
  [<ffffffff815aa7f5>] ? _raw_spin_lock_irq+0x15/0x20
Quoted 11 lines
  [<ffffffff81075141>] ? sigprocmask+0x91/0x110
  [<ffffffff81014ab8>] sys_clone+0x28/0x30
  [<ffffffff8100c3e3>] stub_clone+0x13/0x20
  [<ffffffff8100c0c2>] ? system_call_fastpath+0x16/0x1b
Code: 49 89 01 eb 91 66 0f 1f 44 00 00 55 48 89 e5 41 55 41 54 53 48 83 ec 08 0f 1f 44 00 00 45 31 e4 48 89 fb 49 89 f5 eb 1d 0f 1f 00<f6>  06 01 74 38 48 8b 15 a4 66 02 00 48 89 df 41 bc 01 00 00 00
RIP  [<ffffffffa02cae20>] kvm_unmap_rmapp+0x20/0x70 [kvm]
  RSP<ffff880508ee9bf0>
---[ end trace 85201a339b7635fc ]---
   0:    55                       push   %rbp
   1:    48 89 e5                 mov    %rsp,%rbp
   4:    41 55                    push   %r13
   6:    41 54                    push   %r12
   8:    53                       push   %rbx
   9:    48 83 ec 08              sub    $0x8,%rsp
   d:    0f 1f 44 00 00           nopl   0x0(%rax,%rax,1)
  12:    45 31 e4                 xor    %r12d,%r12d
  15:    48 89 fb                 mov    %rdi,%rbx
  18:    49 89 f5                 mov    %rsi,%r13
  1b:    eb 1d                    jmp    0x3a
  1d:    0f 1f 00                 nopl   (%rax)
  20:    f6 06 01                 testb  $0x1,(%rsi)

Looks like the top 16 bits of %rsi are flipped.

Also wierd to see a fork(). What's your qemu command line?

Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.
#2 lore raw
Quoted 6 lines · Avi Kivity
Quoted 5 lines · Tomasz Chmielewski

I got this on a 2.6.38.1 system which (I think) had some problem accessing guest image on a btrfs filesystem.

general protection fault: 0000 [#1] SMP

(...)

Quoted 15 lines

0: 55 push %rbp
1: 48 89 e5 mov %rsp,%rbp
4: 41 55 push %r13
6: 41 54 push %r12
8: 53 push %rbx
9: 48 83 ec 08 sub $0x8,%rsp
d: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
12: 45 31 e4 xor %r12d,%r12d
15: 48 89 fb mov %rdi,%rbx
18: 49 89 f5 mov %rsi,%r13
1b: eb 1d jmp 0x3a
1d: 0f 1f 00 nopl (%rax)
20: f6 06 01 testb $0x1,(%rsi)

Quoted 3 lines

Looks like the top 16 bits of %rsi are flipped.

Also wierd to see a fork(). What's your qemu command line?

/usr/bin/kvm -monitor unix:/var/run/qemu-server/113.mon,server,nowait -vnc unix:/var/run/qemu-server/113.vnc,password -pidfile /var/run/qemu-server/113.pid -daemonize -usbdevice tablet -name swcache -smp sockets=1,cores=1 -nodefaults -boot menu=on -vga cirrus -tdf -k de -drive file=/var/lib/vz/template/iso/systemrescuecd-x86-2.0.0.iso,if=ide,index=2,media=cdrom -drive file=/var/lib/vz/images/113/vm-113-disk-1.raw,if=scsi,index=0,cache=none,boot=on -m 1024 -netdev type=tap,id=vlan0d0,ifname=tap113i0d0,script=/var/lib/qemu-server/bridge-vlan,vhost=on -device virtio-net-pci,mac=DE:42:48:50:D8:69,netdev=vlan0d0 -netdev type=tap,id=vlan100d0,ifname=tap113i100d0,script=/var/lib/qemu-server/bridge-vlan,vhost=on -device virtio-net-pci,mac=72:D2:6E:8E:07:4D,netdev=vlan100d0

Signature · 2 lines
Tomasz Chmielewski
http://wpkg.org
#3 lore raw
Quoted 25 lines · Tomasz Chmielewski
Quoted 6 lines · Avi Kivity
Quoted 5 lines · Tomasz Chmielewski

I got this on a 2.6.38.1 system which (I think) had some problem accessing guest image on a btrfs filesystem.

 general protection fault: 0000 [#1] SMP

(...)

Quoted 15 lines

0: 55 push %rbp
1: 48 89 e5 mov %rsp,%rbp
4: 41 55 push %r13
6: 41 54 push %r12
8: 53 push %rbx
9: 48 83 ec 08 sub $0x8,%rsp
d: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
12: 45 31 e4 xor %r12d,%r12d
15: 48 89 fb mov %rdi,%rbx
18: 49 89 f5 mov %rsi,%r13
1b: eb 1d jmp 0x3a
1d: 0f 1f 00 nopl (%rax)
20: f6 06 01 testb $0x1,(%rsi)

Quoted 7 lines
Quoted 3 lines

Looks like the top 16 bits of %rsi are flipped.

Also wierd to see a fork(). What's your qemu command line?

/usr/bin/kvm -monitor unix:/var/run/qemu-server/113.mon,server,nowait -vnc unix:/var/run/qemu-server/113.vnc,password -pidfile /var/run/qemu-server/113.pid -daemonize -usbdevice tablet -name swcache -smp sockets=1,cores=1 -nodefaults -boot menu=on -vga cirrus -tdf -k de -drive file=/var/lib/vz/template/iso/systemrescuecd-x86-2.0.0.iso,if=ide,index=2,media=cdrom -drive file=/var/lib/vz/images/113/vm-113-disk-1.raw,if=scsi,index=0,cache=none,boot=on -m 1024 -netdev type=tap,id=vlan0d0,ifname=tap113i0d0,script=/var/lib/qemu-server/bridge-vlan,vhost=on -device virtio-net-pci,mac=DE:42:48:50:D8:69,netdev=vlan0d0 -netdev type=tap,id=vlan100d0,ifname=tap113i100d0,script=/var/lib/qemu-server/bridge-vlan,vhost=on -device virtio-net-pci,mac=72:D2:6E:8E:07:4D,netdev=vlan100d0

Okay, the fork came from the ,script=.

The issue with %rsi looks like a use-after-free, however kvm_mmu_notifier_invalidate_range_start appears to be properly srcu protected.

Signature · 1 line
error compiling committee.c: too many arguments to function
#4 lore raw

On 27.03.2011 11:42, Avi Kivity wrote:

(...)

Quoted 5 lines

Okay, the fork came from the ,script=.

The issue with %rsi looks like a use-after-free, however kvm_mmu_notifier_invalidate_range_start appears to be properly srcu protected.

FYI, I saw this one as well:

http://www.virtall.com/files/temp/kvm.txt

If you need to look at the config, it's available here:

http://www.virtall.com/files/temp/config-2.6.38.1

Signature · 2 lines
Tomasz Chmielewski
http://wpkg.org
#5 lore raw
Quoted 6 lines · Tomasz Chmielewski

On 27.03.2011 11:42, Avi Kivity wrote:

(...)

Quoted 2 lines

Okay, the fork came from the ,script=.

Quoted 7 lines
Quoted 3 lines

The issue with %rsi looks like a use-after-free, however kvm_mmu_notifier_invalidate_range_start appears to be properly srcu protected.

FYI, I saw this one as well:

http://www.virtall.com/files/temp/kvm.txt

Similar pattern - top 16 bits of %rsi are flipped.

Marcelo, what was the option to enable padding for allocations and overrun detection? Also use-after-free?

Signature · 1 line
error compiling committee.c: too many arguments to function
#6 lore raw

Hello everyone,

Quoted 11 lines · Avi Kivity
Quoted 10 lines · Tomasz Chmielewski

On 27.03.2011 11:42, Avi Kivity wrote:

(...)

Quoted 5 lines

Okay, the fork came from the ,script=.

The issue with %rsi looks like a use-after-free, however kvm_mmu_notifier_invalidate_range_start appears to be properly srcu protected.

Quoted 8 lines
Quoted 3 lines

FYI, I saw this one as well:

http://www.virtall.com/files/temp/kvm.txt

Similar pattern - top 16 bits of %rsi are flipped.

Marcelo, what was the option to enable padding for allocations and overrun detection? Also use-after-free?

BTW, is it genuine that a protection fault is generated instead of a page fault while dereferencing address 0x00008805d6b087f8? I would normally except a page fault from a memory dereference that doesn't alter processor state/segments.

The other GFP happened in pmdp_clear_flush_notify inside collapse_huge_page.

#7 lore raw
Quoted 4 lines · Andrea Arcangeli

BTW, is it genuine that a protection fault is generated instead of a page fault while dereferencing address 0x00008805d6b087f8? I would normally except a page fault from a memory dereference that doesn't alter processor state/segments.

Yes. Bits 48-63 of the address must be equal to bit 47, or a #GP is generated (non-canonical address).

Signature · 1 line
error compiling committee.c: too many arguments to function
#8 lore raw
Quoted 8 lines · Avi Kivity
Quoted 4 lines · Andrea Arcangeli

BTW, is it genuine that a protection fault is generated instead of a page fault while dereferencing address 0x00008805d6b087f8? I would normally except a page fault from a memory dereference that doesn't alter processor state/segments.

Yes. Bits 48-63 of the address must be equal to bit 47, or a #GP is generated (non-canonical address).

Ok, when you said 16 bit reversed I didn't match it to bit 48 and max 128TB of user address space. I thought it was good idea to check because in the past I've seen GFP that were hardware issues triggering on normal memory dereference but this is probably not the case.

Tomasz, how easily can you reproduce? Could you upload to the site the output of objdump -dr arch/x86/kvm/mmu.o too? (my assembly is vastly different than the one shown so far, I may find more info in the oops if I get the assembly of the caller too and of the iteration of the loop that runs in that function before the GFP)

khugepaged is present in your second trace (and khugepaged is mangling over some memslot range with guest gfn mapped or kvm_unmap_rmapp wouldn't be called in the first place, hope the memslot are all ok) but probably you didn't get the right alignment so likely the THP are mapped as 4k pages in the guest, which must work fine too. I wonder if that might be related to that (my qemu-kvm I keep it patched with the patch below which isn't yet polished enough to be digestible for qemu, wrong alignments, x86 4M alignment not handled yet, and not sure if the DONTFORK fix to prevent OOM with hotplug/migrate is acceptable in that position).

Can you try to "echo 0 >/sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs" and then run "cat /proc/pgrep qemu/smaps >/dev/null" once per minute (or find the right pid by hand if you've more than one qemu process running).
This debug trick will only work for 2.6.38.1, as 2.6.39 has a native
THP handling in the smaps file, but in 2.6.38.1 it should flush all
sptes mapped on THP just like fork (this might help to reproduce).

I'm also surprised this happened during fork that initialize the tap interface, shouldn't that fork run before any sptes is established? (we're running the spte invalidate with mmu notifier in the parent before wrprotecting the ptes during fork)

I also wonder if it's a memslot race of some kind, I don't see anything wrong in the rmapp handling at the moment.

This isn't a patch to try, I'm only showing it here for reference as I guess I suspect it might hide the bug. I'm now going to reverse it and see if I can reproduce, in case having large sptes (instead of 4k sptes) always mapped on host THP changes something.

Thanks!

exec.c +19 -1
index bb0c1be..f60e5fe 100644
@@ -2856,6 +2856,18 @@ static ram_addr_t last_ram_offset(void)
     return last;
 }
 
+#if defined(__linux__) && defined(__x86_64__)
+/*
+ * Align on the max transparent hugepage size so that
+ * "(gfn ^ pfn) & (HPAGE_SIZE-1) == 0" to allow KVM to
+ * take advantage of hugepages with NPT/EPT or to
+ * ensure the first 2M of the guest physical ram will
+ * be mapped by the same hugetlb for QEMU (it is worth
+ * it even without NPT/EPT).
+ */
+#define PREFERRED_RAM_ALIGN (2*1024*1024)
+#endif
+
 ram_addr_t qemu_ram_alloc_from_ptr(DeviceState *dev, const char *name,
                                    ram_addr_t size, void *host)
 {
@@ -2902,9 +2914,15 @@ ram_addr_t qemu_ram_alloc_from_ptr(DeviceState *dev, const char *name,
                                    PROT_EXEC|PROT_READ|PROT_WRITE,
                                    MAP_SHARED | MAP_ANONYMOUS, -1, 0);
 #else
-            new_block->host = qemu_vmalloc(size);
+#ifdef PREFERRED_RAM_ALIGN
+	    if (size >= PREFERRED_RAM_ALIGN)
+		    new_block->host = qemu_memalign(PREFERRED_RAM_ALIGN, size);
+	    else
+#endif
+		    new_block->host = qemu_vmalloc(size);
 #endif
             qemu_madvise(new_block->host, size, QEMU_MADV_MERGEABLE);
+            qemu_madvise(new_block->host, size, QEMU_MADV_DONTFORK);
         }
     }
 
#9 lore raw
Quoted 1 line · Andrea Arcangeli

Tomasz, how easily can you reproduce?

Well, this server runs 10 VMs or so, and it happens after 1-2 days of uptime.

I reverted now to a 2.6.35.x, as it had enough downtime with 2.6.38 already ;) so I'd rather not experiment anymore for some time with a kernel known to cause problems.

Quoted 2 lines

Could you upload to the site the
output of objdump -dr arch/x86/kvm/mmu.o too?

http://virtall.com/files/temp/mmu-objdump.txt

Signature · 2 lines
Tomasz Chmielewski
http://wpkg.org
#12 lore raw
In reply to Avi Kivity
Quoted 11 lines · Avi Kivity
Quoted 10 lines · Tomasz Chmielewski

On 27.03.2011 11:42, Avi Kivity wrote:

(...)

Quoted 5 lines

Okay, the fork came from the ,script=.

The issue with %rsi looks like a use-after-free, however kvm_mmu_notifier_invalidate_range_start appears to be properly srcu protected.

Quoted 8 lines
Quoted 3 lines

FYI, I saw this one as well:

http://www.virtall.com/files/temp/kvm.txt

Similar pattern - top 16 bits of %rsi are flipped.

Marcelo, what was the option to enable padding for allocations and overrun detection? Also use-after-free?

slub_debug=ZFPU boot kernel parameter.

Documentation/vm/slub.txt:

Possible debug options are
F Sanity checks on (enables SLAB_DEBUG_FREE. Sorry
SLAB legacy issues)
Z Red zoning
P Poisoning (object and padding)
U User tracking (free and alloc)

#10 lore raw
In reply to Tomasz Chmielewski
Quoted 10 lines · Tomasz Chmielewski
Quoted 1 line · Andrea Arcangeli

Tomasz, how easily can you reproduce?

Well, this server runs 10 VMs or so, and it happens after 1-2 days of uptime.

I reverted now to a 2.6.35.x, as it had enough downtime with 2.6.38 already ;) so I'd rather not experiment anymore for some time with a kernel known to cause problems.

Tomasz, to which exact kernel version (host+guests) did you switch and is it now stable?

thanks, -t

#11 lore raw
Quoted 8 lines · Thomas Treutner
Quoted 7 lines · Tomasz Chmielewski
Quoted 1 line · Andrea Arcangeli

Tomasz, how easily can you reproduce?

Well, this server runs 10 VMs or so, and it happens after 1-2 days of uptime.

Quoted 6 lines
Quoted 3 lines

I reverted now to a 2.6.35.x, as it had enough downtime with 2.6.38 already ;) so I'd rather not experiment anymore for some time with a kernel known to cause problems.

Tomasz, to which exact kernel version (host+guests) did you switch and is it now stable?

I've switched the host to the latest 2.6.35.x and it's stable.

Guest kernel doesn't seem to make a difference here, but majority of them are running 2.6.38.x kernel (had some weird issues with "events/0", taking 100% CPU on guests when I used 2.6.35, which made the guests crawling slow).

Signature · 2 lines
Tomasz Chmielewski
http://wpkg.org