syscall rmdir hangs with autofs
Hi *,
we are encountering occasional problems with autofs running inside an KVM guest.
[1387441.969106] INFO: task automount:26560 blocked for more than 120 seconds. [1387441.969110] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [1387441.969112] automount D e8510198 0 26560 2702 0x00000000 [1387441.969117] db0a1ef4 00000082 80000000 e8510198 0004ed69 c8266000 f6e85a40 00000000 [1387441.969123] c08455e0 c08455e0 f41157f0 f4115a88 c55315e0 00000000 c0207c0a db0a1ef0 [1387441.969128] f4115a88 f7222bbc f7222bb8 ffffffff db0a1f20 c05976ae db0a1f14 f41157f0 [1387441.969133] Call Trace: [1387441.969140] [<c0207c0a>] ? mntput_no_expire+0x1a/0xd0 [1387441.969146] [<c05976ae>] __mutex_lock_slowpath+0xbe/0x120 [1387441.969149] [<c05975d0>] mutex_lock+0x20/0x40 [1387441.969152] [<c01fbc82>] do_rmdir+0x52/0xe0 [1387441.969155] [<c059ae47>] ? do_page_fault+0x1d7/0x3a0 [1387441.969158] [<c01fbd70>] sys_rmdir+0x10/0x20 [1387441.969161] [<c01033cc>] syscall_call+0x7/0xb
The block always occurs in sys_rmdir when automount tries to remove the mountpoint right after umounting the filesystem. There is an successful lstat() on the mountpoint directly precceeding the rmdir call.
It looks like we are triggering some sort of race condition here.
We are currently using 2.6.31-20-generic-pae ubuntu kernel in the 6 CPU guest, 2.6.34 vanilla and qemu-kvm-0.12.4 in the host. But the problem existed long before with all different combinations of guest/host/qemu versions. The virtual HD is if=ide,format=host_device,cache=none on an DRBD container on top of an LVM device. FS is ext3.
Unfortunately, the problem is not easy reproduceable. It occurs every one or two weeks. But since the hanging system call blocks the whole filesystem we have to reboot the guest to get it into an useable state again.
Any ideas what's going wrong here?
Best regards,
Sebastian
Quoted 32 lines · Sebastian Hetze
Hi *,
we are encountering occasional problems with autofs running inside an KVM guest.
[1387441.969106] INFO: task automount:26560 blocked for more than 120 seconds. [1387441.969110] "echo 0> /proc/sys/kernel/hung_task_timeout_secs" disables this message. [1387441.969112] automount D e8510198 0 26560 2702 0x00000000 [1387441.969117] db0a1ef4 00000082 80000000 e8510198 0004ed69 c8266000 f6e85a40 00000000 [1387441.969123] c08455e0 c08455e0 f41157f0 f4115a88 c55315e0 00000000 c0207c0a db0a1ef0 [1387441.969128] f4115a88 f7222bbc f7222bb8 ffffffff db0a1f20 c05976ae db0a1f14 f41157f0 [1387441.969133] Call Trace: [1387441.969140] [<c0207c0a>] ? mntput_no_expire+0x1a/0xd0 [1387441.969146] [<c05976ae>] __mutex_lock_slowpath+0xbe/0x120 [1387441.969149] [<c05975d0>] mutex_lock+0x20/0x40 [1387441.969152] [<c01fbc82>] do_rmdir+0x52/0xe0 [1387441.969155] [<c059ae47>] ? do_page_fault+0x1d7/0x3a0 [1387441.969158] [<c01fbd70>] sys_rmdir+0x10/0x20 [1387441.969161] [<c01033cc>] syscall_call+0x7/0xbThe block always occurs in
sys_rmdirwhen automount tries to remove the mountpoint right after umounting the filesystem. There is an successfullstat()on the mountpoint directly precceeding the rmdir call.It looks like we are triggering some sort of race condition here.
We are currently using 2.6.31-20-generic-pae ubuntu kernel in the 6 CPU guest, 2.6.34 vanilla and qemu-kvm-0.12.4 in the host. But the problem existed long before with all different combinations of guest/host/qemu versions. The virtual HD is if=ide,format=host_device,cache=none on an DRBD container on top of an LVM device. FS is ext3.
Quoted 7 lines
Unfortunately, the problem is not easy reproduceable. It occurs every one or two weeks. But since the hanging system call blocks the whole filesystem we have to reboot the guest to get it into an useable state again.
Any ideas what's going wrong here?
Is there substantial I/O going on?
If not, it may be an autofs bug unrelated to kvm.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 37 lines · Avi Kivity
Quoted 36 lines · Sebastian Hetze
Hi *,
we are encountering occasional problems with autofs running inside an KVM guest.
[1387441.969106] INFO: task automount:26560 blocked for more than 120 seconds. [1387441.969110] "echo 0> /proc/sys/kernel/hung_task_timeout_secs" disables this message. [1387441.969112] automount D e8510198 0 26560 2702 0x00000000 [1387441.969117] db0a1ef4 00000082 80000000 e8510198 0004ed69 c8266000 f6e85a40 00000000 [1387441.969123] c08455e0 c08455e0 f41157f0 f4115a88 c55315e0 00000000 c0207c0a db0a1ef0 [1387441.969128] f4115a88 f7222bbc f7222bb8 ffffffff db0a1f20 c05976ae db0a1f14 f41157f0 [1387441.969133] Call Trace: [1387441.969140] [<c0207c0a>] ? mntput_no_expire+0x1a/0xd0 [1387441.969146] [<c05976ae>] __mutex_lock_slowpath+0xbe/0x120 [1387441.969149] [<c05975d0>] mutex_lock+0x20/0x40 [1387441.969152] [<c01fbc82>] do_rmdir+0x52/0xe0 [1387441.969155] [<c059ae47>] ? do_page_fault+0x1d7/0x3a0 [1387441.969158] [<c01fbd70>] sys_rmdir+0x10/0x20 [1387441.969161] [<c01033cc>] syscall_call+0x7/0xbThe block always occurs in
sys_rmdirwhen automount tries to remove the mountpoint right after umounting the filesystem. There is an successfullstat()on the mountpoint directly precceeding the rmdir call.It looks like we are triggering some sort of race condition here.
We are currently using 2.6.31-20-generic-pae ubuntu kernel in the 6 CPU guest, 2.6.34 vanilla and qemu-kvm-0.12.4 in the host. But the problem existed long before with all different combinations of guest/host/qemu versions. The virtual HD is if=ide,format=host_device,cache=none on an DRBD container on top of an LVM device. FS is ext3.
Unfortunately, the problem is not easy reproduceable. It occurs every one or two weeks. But since the hanging system call blocks the whole filesystem we have to reboot the guest to get it into an useable state again.
Quoted 7 lines
Quoted 3 lines
Any ideas what's going wrong here?
Is there substantial I/O going on?
If not, it may be an autofs bug unrelated to kvm.
the autofs expire event occured at 01:15:01
sar shows
00:00:01 CPU %user %nice %system %iowait %steal %idle 00:35:02 all 0,31 1,90 3,88 29,78 0,00 64,12 00:45:02 all 0,72 1,99 3,56 23,93 0,00 69,80 00:55:01 all 1,35 1,49 4,13 23,76 0,00 69,27 01:05:01 all 0,77 1,84 4,43 28,34 0,00 64,62 01:15:01 all 0,29 1,46 3,41 44,07 0,00 50,77 01:25:02 all 0,22 1,25 2,63 45,34 0,00 50,56 01:35:02 all 0,34 1,33 2,87 46,74 0,00 48,72 01:45:02 all 0,30 0,90 2,57 40,03 0,00 56,20 01:55:02 all 0,26 0,43 2,29 9,79 0,00 87,23
00:00:01 tps rtps wtps bread/s bwrtn/s
00:35:02 461,69 407,75 53,94 35196,06 32673,83 00:45:02 298,29 238,30 59,99 38553,34 33062,97 00:55:01 294,81 241,08 53,73 35469,66 25948,30 01:05:01 338,62 279,97 58,66 36164,27 31109,18 01:15:01 462,22 406,24 55,97 28428,26 25725,05 01:25:02 366,88 331,82 35,07 24160,53 22284,83 01:35:02 394,73 358,21 36,52 25770,79 23516,81 01:45:02 409,83 379,66 30,17 17874,79 15608,74 01:55:02 453,18 448,62 4,56 3754,82 79,47
so, yes there is substantion I/O going on.
Quoted 74 lines · Sebastian Hetze
Quoted 38 lines · Avi Kivity
Quoted 36 lines · Sebastian Hetze
Hi *,
we are encountering occasional problems with autofs running inside an KVM guest.
[1387441.969106] INFO: task automount:26560 blocked for more than 120 seconds. [1387441.969110] "echo 0> /proc/sys/kernel/hung_task_timeout_secs" disables this message. [1387441.969112] automount D e8510198 0 26560 2702 0x00000000 [1387441.969117] db0a1ef4 00000082 80000000 e8510198 0004ed69 c8266000 f6e85a40 00000000 [1387441.969123] c08455e0 c08455e0 f41157f0 f4115a88 c55315e0 00000000 c0207c0a db0a1ef0 [1387441.969128] f4115a88 f7222bbc f7222bb8 ffffffff db0a1f20 c05976ae db0a1f14 f41157f0 [1387441.969133] Call Trace: [1387441.969140] [<c0207c0a>] ? mntput_no_expire+0x1a/0xd0 [1387441.969146] [<c05976ae>] __mutex_lock_slowpath+0xbe/0x120 [1387441.969149] [<c05975d0>] mutex_lock+0x20/0x40 [1387441.969152] [<c01fbc82>] do_rmdir+0x52/0xe0 [1387441.969155] [<c059ae47>] ? do_page_fault+0x1d7/0x3a0 [1387441.969158] [<c01fbd70>] sys_rmdir+0x10/0x20 [1387441.969161] [<c01033cc>] syscall_call+0x7/0xbThe block always occurs in
sys_rmdirwhen automount tries to remove the mountpoint right after umounting the filesystem. There is an successfullstat()on the mountpoint directly precceeding the rmdir call.It looks like we are triggering some sort of race condition here.
We are currently using 2.6.31-20-generic-pae ubuntu kernel in the 6 CPU guest, 2.6.34 vanilla and qemu-kvm-0.12.4 in the host. But the problem existed long before with all different combinations of guest/host/qemu versions. The virtual HD is if=ide,format=host_device,cache=none on an DRBD container on top of an LVM device. FS is ext3.
Unfortunately, the problem is not easy reproduceable. It occurs every one or two weeks. But since the hanging system call blocks the whole filesystem we have to reboot the guest to get it into an useable state again.
Quoted 8 lines
Quoted 4 lines
Any ideas what's going wrong here?
Is there substantial I/O going on?
If not, it may be an autofs bug unrelated to kvm.
the autofs expire event occured at 01:15:01
sar shows
00:00:01 CPU %user %nice %system %iowait %steal %idle 00:35:02 all 0,31 1,90 3,88 29,78 0,00 64,12 00:45:02 all 0,72 1,99 3,56 23,93 0,00 69,80 00:55:01 all 1,35 1,49 4,13 23,76 0,00 69,27 01:05:01 all 0,77 1,84 4,43 28,34 0,00 64,62 01:15:01 all 0,29 1,46 3,41 44,07 0,00 50,77 01:25:02 all 0,22 1,25 2,63 45,34 0,00 50,56 01:35:02 all 0,34 1,33 2,87 46,74 0,00 48,72 01:45:02 all 0,30 0,90 2,57 40,03 0,00 56,20 01:55:02 all 0,26 0,43 2,29 9,79 0,00 87,2300:00:01 tps rtps wtps bread/s bwrtn/s
00:35:02 461,69 407,75 53,94 35196,06 32673,83 00:45:02 298,29 238,30 59,99 38553,34 33062,97 00:55:01 294,81 241,08 53,73 35469,66 25948,30 01:05:01 338,62 279,97 58,66 36164,27 31109,18 01:15:01 462,22 406,24 55,97 28428,26 25725,05 01:25:02 366,88 331,82 35,07 24160,53 22284,83 01:35:02 394,73 358,21 36,52 25770,79 23516,81 01:45:02 409,83 379,66 30,17 17874,79 15608,74 01:55:02 453,18 448,62 4,56 3754,82 79,47
Quoted 2 lines
so, yes there is substantion I/O going on.
Looks like a false alarm then. The rmdir is waiting for the mount to flush everything to disk, which is slow and takes a while.
Does it return eventually?
Perhaps it should do a lazy unmount.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 75 lines · Avi Kivity
Quoted 74 lines · Sebastian Hetze
Quoted 38 lines · Avi Kivity
Quoted 36 lines · Sebastian Hetze
Hi *,
we are encountering occasional problems with autofs running inside an KVM guest.
[1387441.969106] INFO: task automount:26560 blocked for more than 120 seconds. [1387441.969110] "echo 0> /proc/sys/kernel/hung_task_timeout_secs" disables this message. [1387441.969112] automount D e8510198 0 26560 2702 0x00000000 [1387441.969117] db0a1ef4 00000082 80000000 e8510198 0004ed69 c8266000 f6e85a40 00000000 [1387441.969123] c08455e0 c08455e0 f41157f0 f4115a88 c55315e0 00000000 c0207c0a db0a1ef0 [1387441.969128] f4115a88 f7222bbc f7222bb8 ffffffff db0a1f20 c05976ae db0a1f14 f41157f0 [1387441.969133] Call Trace: [1387441.969140] [<c0207c0a>] ? mntput_no_expire+0x1a/0xd0 [1387441.969146] [<c05976ae>] __mutex_lock_slowpath+0xbe/0x120 [1387441.969149] [<c05975d0>] mutex_lock+0x20/0x40 [1387441.969152] [<c01fbc82>] do_rmdir+0x52/0xe0 [1387441.969155] [<c059ae47>] ? do_page_fault+0x1d7/0x3a0 [1387441.969158] [<c01fbd70>] sys_rmdir+0x10/0x20 [1387441.969161] [<c01033cc>] syscall_call+0x7/0xbThe block always occurs in
sys_rmdirwhen automount tries to remove the mountpoint right after umounting the filesystem. There is an successfullstat()on the mountpoint directly precceeding the rmdir call.It looks like we are triggering some sort of race condition here.
We are currently using 2.6.31-20-generic-pae ubuntu kernel in the 6 CPU guest, 2.6.34 vanilla and qemu-kvm-0.12.4 in the host. But the problem existed long before with all different combinations of guest/host/qemu versions. The virtual HD is if=ide,format=host_device,cache=none on an DRBD container on top of an LVM device. FS is ext3.
Unfortunately, the problem is not easy reproduceable. It occurs every one or two weeks. But since the hanging system call blocks the whole filesystem we have to reboot the guest to get it into an useable state again.
Quoted 8 lines
Quoted 4 lines
Any ideas what's going wrong here?
Is there substantial I/O going on?
If not, it may be an autofs bug unrelated to kvm.
the autofs expire event occured at 01:15:01
sar shows
00:00:01 CPU %user %nice %system %iowait %steal %idle 00:35:02 all 0,31 1,90 3,88 29,78 0,00 64,12 00:45:02 all 0,72 1,99 3,56 23,93 0,00 69,80 00:55:01 all 1,35 1,49 4,13 23,76 0,00 69,27 01:05:01 all 0,77 1,84 4,43 28,34 0,00 64,62 01:15:01 all 0,29 1,46 3,41 44,07 0,00 50,77 01:25:02 all 0,22 1,25 2,63 45,34 0,00 50,56 01:35:02 all 0,34 1,33 2,87 46,74 0,00 48,72 01:45:02 all 0,30 0,90 2,57 40,03 0,00 56,20 01:55:02 all 0,26 0,43 2,29 9,79 0,00 87,2300:00:01 tps rtps wtps bread/s bwrtn/s
00:35:02 461,69 407,75 53,94 35196,06 32673,83 00:45:02 298,29 238,30 59,99 38553,34 33062,97 00:55:01 294,81 241,08 53,73 35469,66 25948,30 01:05:01 338,62 279,97 58,66 36164,27 31109,18 01:15:01 462,22 406,24 55,97 28428,26 25725,05 01:25:02 366,88 331,82 35,07 24160,53 22284,83 01:35:02 394,73 358,21 36,52 25770,79 23516,81 01:45:02 409,83 379,66 30,17 17874,79 15608,74 01:55:02 453,18 448,62 4,56 3754,82 79,47
Quoted 7 lines
Quoted 2 lines
so, yes there is substantion I/O going on.
Looks like a false alarm then. The rmdir is waiting for the mount to flush everything to disk, which is slow and takes a while.
Does it return eventually?
No, it does not return within hours (>10). And the problem occurs only once in a while although the system is busy every day (and night) and automount is mounting/expiring frequently.
I would expect the "/bin/umount dir" process (which is forked by automount if I read the code correctly) to return only after the flush is complete. So I expect the rmdir being called afterwards on an plain empty directory.
BTW: the mount is just a bind mount, so no flush should be necessary anyway.
Quoted 10 lines · Sebastian Hetze
Quoted 6 lines
Looks like a false alarm then. The rmdir is waiting for the mount to flush everything to disk, which is slow and takes a while.
Does it return eventually?
No, it does not return within hours (>10). And the problem occurs only once in a while although the system is busy every day (and night) and automount is mounting/expiring frequently.
Quoted 8 lines
I would expect the "
/bin/umountdir" process (which is forked by automount if I read the code correctly) to return only after the flush is complete. So I expect the rmdir being called afterwards on an plain empty directory.BTW: the mount is just a bind mount, so no flush should be necessary anyway.
It looks like VFS breakage. Does this happen with older guest kernels?
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 16 lines · Avi Kivity
Quoted 15 lines · Sebastian Hetze
Quoted 6 lines
Looks like a false alarm then. The rmdir is waiting for the mount to flush everything to disk, which is slow and takes a while.
Does it return eventually?
No, it does not return within hours (>10). And the problem occurs only once in a while although the system is busy every day (and night) and automount is mounting/expiring frequently.
I would expect the "
/bin/umountdir" process (which is forked by automount if I read the code correctly) to return only after the flush is complete. So I expect the rmdir being called afterwards on an plain empty directory.
Quoted 5 lines
Quoted 3 lines
BTW: the mount is just a bind mount, so no flush should be necessary anyway.
It looks like VFS breakage. Does this happen with older guest kernels?
currently I can confirm this problem with 2.6.31-22-generic-pae back to 2.6.31-16-generic-pae. What guest kernel version would you suggest?
Quoted 6 lines · Sebastian Hetze
Quoted 3 lines
It looks like VFS breakage. Does this happen with older guest kernels?
currently I can confirm this problem with 2.6.31-22-generic-pae back to 2.6.31-16-generic-pae. What guest kernel version would you suggest?
Can you try it on a non-virtualized system?
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 9 lines · Avi Kivity
Quoted 6 lines · Sebastian Hetze
Quoted 3 lines
It looks like VFS breakage. Does this happen with older guest kernels?
currently I can confirm this problem with 2.6.31-22-generic-pae back to 2.6.31-16-generic-pae. What guest kernel version would you suggest?
Can you try it on a non-virtualized system?
This automount setup has been running non-virtualized for quite a while without problems until last year. Never seen or heared about similar hangs with automount before.
We have not been able to reproduce this error in laboratory conditions. The productive system has 2TB data and the problem occurs totally unpredictable with real live workload.
And there is no easy way to switch back to non-virtualized unless we declare this whole project a failure.
Quoted 14 lines · Sebastian Hetze
Quoted 12 lines · Avi Kivity
Quoted 8 lines · Sebastian Hetze
Quoted 4 lines
It looks like VFS breakage. Does this happen with older guest kernels?
currently I can confirm this problem with 2.6.31-22-generic-pae back to 2.6.31-16-generic-pae. What guest kernel version would you suggest?
Can you try it on a non-virtualized system?
Quoted 9 lines
This automount setup has been running non-virtualized for quite a while without problems until last year. Never seen or heared about similar hangs with automount before.
We have not been able to reproduce this error in laboratory conditions. The productive system has 2TB data and the problem occurs totally unpredictable with real live workload.
And there is no easy way to switch back to non-virtualized unless we declare this whole project a failure.
Ahem.
What's your hardware platform? EPT/NPT capable? Host kernel version?
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 25 lines · Avi Kivity
Quoted 14 lines · Sebastian Hetze
Quoted 12 lines · Avi Kivity
Quoted 8 lines · Sebastian Hetze
Quoted 4 lines
It looks like VFS breakage. Does this happen with older guest kernels?
currently I can confirm this problem with 2.6.31-22-generic-pae back to 2.6.31-16-generic-pae. What guest kernel version would you suggest?
Can you try it on a non-virtualized system?
Quoted 9 lines
This automount setup has been running non-virtualized for quite a while without problems until last year. Never seen or heared about similar hangs with automount before.
We have not been able to reproduce this error in laboratory conditions. The productive system has 2TB data and the problem occurs totally unpredictable with real live workload.
And there is no easy way to switch back to non-virtualized unless we declare this whole project a failure.
Quoted 3 lines
Ahem.
What's your hardware platform? EPT/NPT capable? Host kernel version?
Intel S5520HC Board with 2 Xeon CPU E5520, HT enabled
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
(so yes, EPT is available)
host kernel is vanilla 2.6.34
Quoted 11 lines · Sebastian Hetze
Quoted 3 lines
What's your hardware platform? EPT/NPT capable? Host kernel version?
Intel S5520HC Board with 2 Xeon CPU E5520, HT enabled
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lmconstant_tscarch_perfmonpebs btsrep_goodxtopologynonstop_tscaperfmperf pni dtes64 monitords_cplvmx est tm2 ssse3 cx16 xtpr pdcm dcasse4_1sse4_2popcntlahf_lmidatpr_shadowvnmi flexpriority ept vpid
Quoted 5 lines
(so yes, EPT is available)
host kernel is vanilla 2.6.34
Well, EPT makes it unlikely that there's an mmu bug involved.
Is the guest smp? Can you try UP?
Can you try 2.6.32.latest in the guest?
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 14 lines · Avi Kivity
Quoted 13 lines · Sebastian Hetze
Quoted 3 lines
What's your hardware platform? EPT/NPT capable? Host kernel version?
Intel S5520HC Board with 2 Xeon CPU E5520, HT enabled
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lmconstant_tscarch_perfmonpebs btsrep_goodxtopologynonstop_tscaperfmperf pni dtes64 monitords_cplvmx est tm2 ssse3 cx16 xtpr pdcm dcasse4_1sse4_2popcntlahf_lmidatpr_shadowvnmi flexpriority ept vpid(so yes, EPT is available)
Quoted 7 lines
Quoted 3 lines
host kernel is vanilla 2.6.34
Well, EPT makes it unlikely that there's an mmu bug involved.
Is the guest smp? Can you try UP?
yes, the guest starts with -smp 6
Since the system is quite busy and the problem occurs only ever two weeks or so, UP is no option.
Quoted 2 lines
Can you try 2.6.32.latest in the guest?
This we can certainly do. Is 2.6.32.16 your recommendation or will any newer kernel do also?
Quoted 10 lines · Sebastian Hetze
Quoted 5 lines
Well, EPT makes it unlikely that there's an mmu bug involved.
Is the guest smp? Can you try UP?
yes, the guest starts with -smp 6
Since the system is quite busy and the problem occurs only ever two weeks or so, UP is no option.
I see.
Quoted 6 lines
Quoted 2 lines
Can you try 2.6.32.latest in the guest?
This we can certainly do. Is 2.6.32.16 your recommendation or will any newer kernel do also?
2.6.32.16 is probably the stablest out there. Note that there's a problem exposed with certain versions of gcc, I recommend compiling with CONFIG_DEBUG_RODATA=n, or the kernel may fail to boot.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.