Help debugging a regression in KVM Module
Hi,
some time a go I stumbled across a regression in the KVM Module that has been introduced somewhere between 3.17 and 3.19.
I have a rather old openSUSE guest with an XFS filesystem which realiably crashes after some live migrations. I originally believed that the issue might be related to my setup with a 3.12 host kernel and kvm-kmod 3.19, but I now found that it is also still present with a 3.19 host kernel with included 3.19 kvm module.
My idea was to continue testing on a 3.12 host kernel and then bisect all commits to the kvm related parts.
Now my question is how to best bisect only kvm related changes (those that go into kvm-kmod)?
Thanks,
Peter
Paolo Bonzini
maintainer----- Original Message -----
Quoted 19 lines
From: "Peter Lieven" <[email protected]>
To: [email protected], [email protected]
Cc: "Paolo Bonzini" <[email protected]>
Sent: Friday, August 14, 2015 1:11:34 PM
Subject: Help debugging a regression in KVM ModuleHi,
some time a go I stumbled across a regression in the KVM Module that has been introduced somewhere
between 3.17 and 3.19.I have a rather old openSUSE guest with an XFS filesystem which realiably crashes after some live migrations.
I originally believed that the issue might be related to my setup with a 3.12 host kernel and kvm-kmod 3.19,
but I now found that it is also still present with a 3.19 host kernel with included 3.19 kvm module.
Quoted 5 lines
My idea was to continue testing on a 3.12 host kernel and then bisect all commits to the kvm related parts.
Now my question is how to best bisect only kvm related changes (those that go into kvm-kmod)?
I haven't forgotten this. Sorry. :(
Unfortunately I'll be away for three weeks, but I'll make it a priority when I'm back.
Paolo
Am 14.08.2015 um 15:01 schrieb Paolo Bonzini:
Quoted 24 lines
----- Original Message -----
Quoted 22 lines
From: "Peter Lieven" <[email protected]>
To: [email protected], [email protected]
Cc: "Paolo Bonzini" <[email protected]>
Sent: Friday, August 14, 2015 1:11:34 PM
Subject: Help debugging a regression in KVM ModuleHi,
some time a go I stumbled across a regression in the KVM Module that has been introduced somewhere
between 3.17 and 3.19.I have a rather old openSUSE guest with an XFS filesystem which realiably crashes after some live migrations.
I originally believed that the issue might be related to my setup with a 3.12 host kernel and kvm-kmod 3.19,
but I now found that it is also still present with a 3.19 host kernel with included 3.19 kvm module.My idea was to continue testing on a 3.12 host kernel and then bisect all commits to the kvm related parts.
Quoted 6 lines
Quoted 2 lines
Now my question is how to best bisect only kvm related changes (those that go into kvm-kmod)?
I haven't forgotten this. Sorry. :(
Unfortunately I'll be away for three weeks, but I'll make it a priority when I'm back.
Its not time critical, but I think its worth investigating as it might affect other systems as well - and maybe XFS is only very sensitive.
I suppose you are going on vacation. Enjoy!
Peter
Quoted 5 lines · Peter Lieven
Hi,
some time a go I stumbled across a regression in the KVM Module that has been introduced somewhere between 3.17 and 3.19.
Quoted 8 lines
I have a rather old openSUSE guest with an XFS filesystem which realiably crashes after some live migrations. I originally believed that the issue might be related to my setup with a 3.12 host kernel and kvm-kmod 3.19, but I now found that it is also still present with a 3.19 host kernel with included 3.19 kvm module.
My idea was to continue testing on a 3.12 host kernel and then bisect all commits to the kvm related parts.
Now my question is how to best bisect only kvm related changes (those that go into kvm-kmod)?
In general I don't bother. As it is a bisection you eliminate half the commits at a time you get their fairly quickly anyway. However you can tell bisect which parts of the tree you car about:
git bisect start -- arch/arm64/kvm include/linux/kvm* include/uapi/linux/kvm* virt/kvm/
Quoted 3 lines
Thanks,
Peter
Signature · 1 line
Alex Bennée
Am 14.08.2015 um 22:01 schrieb Alex Bennée:
Quoted 13 lines
Quoted 11 lines · Peter Lieven
Hi,
some time a go I stumbled across a regression in the KVM Module that has been introduced somewhere between 3.17 and 3.19.
I have a rather old openSUSE guest with an XFS filesystem which realiably crashes after some live migrations. I originally believed that the issue might be related to my setup with a 3.12 host kernel and kvm-kmod 3.19, but I now found that it is also still present with a 3.19 host kernel with included 3.19 kvm module.
My idea was to continue testing on a 3.12 host kernel and then bisect all commits to the kvm related parts.
Quoted 7 lines
Quoted 2 lines
Now my question is how to best bisect only kvm related changes (those that go into kvm-kmod)?
In general I don't bother. As it is a bisection you eliminate half the commits at a time you get their fairly quickly anyway. However you can tell bisect which parts of the tree you car about:
git bisect start --
arch/arm64/kvminclude/linux/kvm*include/uapi/linux/kvm*virt/kvm/
Yes, I just have to find out how exactly that works out if I want to bisect the linux submodule of the kvm-kmod repository. But thanks for the pointer on how to limit the directories.
Thanks,
Peter
Am 14.08.2015 um 22:01 schrieb Alex Bennée:
Quoted 13 lines
Quoted 11 lines · Peter Lieven
Hi,
some time a go I stumbled across a regression in the KVM Module that has been introduced somewhere between 3.17 and 3.19.
I have a rather old openSUSE guest with an XFS filesystem which realiably crashes after some live migrations. I originally believed that the issue might be related to my setup with a 3.12 host kernel and kvm-kmod 3.19, but I now found that it is also still present with a 3.19 host kernel with included 3.19 kvm module.
My idea was to continue testing on a 3.12 host kernel and then bisect all commits to the kvm related parts.
Quoted 7 lines
Quoted 2 lines
Now my question is how to best bisect only kvm related changes (those that go into kvm-kmod)?
In general I don't bother. As it is a bisection you eliminate half the commits at a time you get their fairly quickly anyway. However you can tell bisect which parts of the tree you car about:
git bisect start --
arch/arm64/kvminclude/linux/kvm*include/uapi/linux/kvm*virt/kvm/
After some experiments I was able to find out the bad commit that introduced the regression:
commit f30ebc312ca9def25650b4e1d01cdb425c310dca
Author: Radim Krčmář <[email protected]> Date: Thu Oct 30 15:06:47 2014 +0100
It seems that this optimisation is not working reliabliy after live migration. I can't reproduce if I take a 3.19 kernel and revert this single commit.
Peter
2015-08-18 16:54+0200, Peter Lieven:
Quoted 8 lines
After some experiments I was able to find out the bad commit that introduced the regression:
commit
f30ebc312ca9def25650b4e1d01cdb425c310dcaAuthor: Radim Krčmář <[email protected]> Date: Thu Oct 30 15:06:47 2014 +0100It seems that this optimisation is not working reliabliy after live migration. I can't reproduce if I take a 3.19 kernel and revert this single commit.
Hello, this bug has gone unnoticed for a long time so it is fixed only since v4.1 (and v3.19.stable was dead at that point).
commit b6ac069532218027f2991cba01d7a72a200688b0
Author: Radim Krčmář <[email protected]> Date: Fri Jun 5 20:57:41 2015 +0200
KVM: x86: fix lapic.timer_mode on restore
lapic.timer_mode was not properly initialized after migration, which broke few useful things, like login, by making every sleep eternal.
Fix this by calling apic_update_lvtt in kvm_apic_post_state_restore.
There are other slowpaths that update lvtt, so this patch makes sure something similar doesn't happen again by calling apic_update_lvtt after every modification.
Quoted 11 lines
Am 18.08.2015 um 17:25 schrieb Radim Krčmář <[email protected]>:
2015-08-18 16:54+0200, Peter Lieven:
Quoted 6 lines
After some experiments I was able to find out the bad commit that introduced the regression:
commit
f30ebc312ca9def25650b4e1d01cdb425c310dcaAuthor: Radim Krčmář <[email protected]> Date: Thu Oct 30 15:06:47 2014 +0100
Quoted 5 lines
Quoted 2 lines
It seems that this optimisation is not working reliabliy after live migration. I can't reproduce if I take a 3.19 kernel and revert this single commit.
Hello, this bug has gone unnoticed for a long time so it is fixed only since v4.1 (and v3.19.stable was dead at that point).
thanks for the pointer. i noticed the regression some time ago, but never found the time to debug. some distros rely on 3.19 e.g. Ubuntu LTS 14.04.2. I will try to ping the maintainer.
Peter
Quoted 20 lines
commit
b6ac069532218027f2991cba01d7a72a200688b0Author: Radim Krčmář <[email protected]> Date: Fri Jun 5 20:57:41 2015 +0200KVM: x86: fix lapic.timer_mode on restore
lapic.timer_mode was not properly initialized after migration, which broke few useful things, like login, by making every sleep eternal.
Fix this by calling
apic_update_lvttinkvm_apic_post_state_restore.There are other slowpaths that update lvtt, so this patch makes sure something similar doesn't happen again by calling
apic_update_lvttafter every modification.Fixes f30ebc312ca9 ("KVM: x86: optimize some accesses to LVTT and SPIV")Signed-off-by Radim Krčmář <[email protected]>Signed-off-by Marcelo Tosatti <[email protected]>