Debugging an inconsistent shadow page table

DISCUSSION Jan Kiszka 3 participants 9 messages Apr 25, 2009 – Apr 26, 2009 KVM

Share & tools

Jan Kiszka

author

2009-04-25 10:36 UTC #0 lore raw

Hi,

turning on MMU_DEBUG and AUDIT in arch/x86/kvm/mmu.c (and fixing a build error, patch will follow) I got this (and then a #GP :( - patch will follow):

...
kvm_mmu_get_page: looking gfn 0 role f0120
kvm_mmu_get_page: found
kvm_mmu_get_page: looking gfn 0 role f0220
kvm_mmu_get_page: found
kvm_mmu_get_page: looking gfn 0 role f0320
kvm_mmu_get_page: found
kvm_mmu_get_page: looking gfn e1f role e0044
kvm_mmu_get_page: adding gfn e1f role e0044
rmap_write_protect: spte ffff8100660a60f8 7ca98067
paging64_page_fault: addr 100105 err 19
audit_write_protection: (pre page fault) shadow page has writable mappings: gfn e1f role e0044
audit: (pre page fault) nontrapping pte in nonleaf level: levels 4 gva 8000000000 level 4 pte 0

Is the last message indicating a problem? I get it very early during guest boot. oos_shadow is disabled.

I'm currently trying to understand an obvious inconsistency in the pte describing a page of the virtio-net rx ring. On some guests with some qemu (upstream) command lines I can trigger this with '-smb /some/path' and then doing smbclient -L in the guest. Once the inconsistency slipped in, host and guest see different page contents and virtio-net stops to work. Very strange, but fortunately easily reproducible here. Any hints or debugging suggestions welcome!

Jan

Avi Kivity

2009-04-26 10:32 UTC #1 lore raw

Quoted 20 lines · Jan Kiszka

Hi,
turning on MMU_DEBUG and AUDIT in arch/x86/kvm/mmu.c (and fixing a build error, patch will follow) I got this (and then a #GP :( - patch will follow):
...
kvm_mmu_get_page: looking gfn 0 role f0120
kvm_mmu_get_page: found
kvm_mmu_get_page: looking gfn 0 role f0220
kvm_mmu_get_page: found
kvm_mmu_get_page: looking gfn 0 role f0320
kvm_mmu_get_page: found
kvm_mmu_get_page: looking gfn e1f role e0044
kvm_mmu_get_page: adding gfn e1f role e0044
rmap_write_protect: spte ffff8100660a60f8 7ca98067
paging64_page_fault: addr 100105 err 19
audit_write_protection: (pre page fault) shadow page has writable mappings: gfn e1f role e0044
audit: (pre page fault) nontrapping pte in nonleaf level: levels 4 gva 8000000000 level 4 pte 0

Quoted 3 lines

Is the last message indicating a problem? I get it very early during guest boot. oos_shadow is disabled.

Yes. It means the guest will receive a page fault if is accesses anything this pte points to. Theoretically we could have made this work, but we never did.

But the message is self-contradictory. Level 4 PTEs map 0.5TB each, and the gva mentioned isn't 0.5TB aligned.

Quoted 8 lines

I'm currently trying to understand an obvious inconsistency in the pte describing a page of the virtio-net rx ring. On some guests with some qemu (upstream) command lines I can trigger this with '-smb /some/path' and then doing smbclient -L in the guest. Once the inconsistency slipped in, host and guest see different page contents and virtio-net stops to work. Very strange, but fortunately easily reproducible here. Any hints or debugging suggestions welcome!

What type of inconsistency? pfn or flags?

Signature · 1 line

error compiling committee.c: too many arguments to function

Jan Kiszka

author

2009-04-26 11:11 UTC #2 lore raw

Quoted 43 lines · Avi Kivity

Quoted 22 lines · Jan Kiszka
Hi,
turning on MMU_DEBUG and AUDIT in arch/x86/kvm/mmu.c (and fixing a build error, patch will follow) I got this (and then a #GP :( - patch will follow):
...
kvm_mmu_get_page: looking gfn 0 role f0120
kvm_mmu_get_page: found
kvm_mmu_get_page: looking gfn 0 role f0220
kvm_mmu_get_page: found
kvm_mmu_get_page: looking gfn 0 role f0320
kvm_mmu_get_page: found
kvm_mmu_get_page: looking gfn e1f role e0044
kvm_mmu_get_page: adding gfn e1f role e0044
rmap_write_protect: spte ffff8100660a60f8 7ca98067
paging64_page_fault: addr 100105 err 19
audit_write_protection: (pre page fault) shadow page has writable
mappings: gfn e1f role e0044
audit: (pre page fault) nontrapping pte in nonleaf level: levels 4 gva 8000000000 level 4 pte 0
Quoted 3 lines
Is the last message indicating a problem? I get it very early during guest boot. oos_shadow is disabled.
Yes. It means the guest will receive a page fault if is accesses anything this pte points to. Theoretically we could have made this work, but we never did.
But the message is self-contradictory. Level 4 PTEs map 0.5TB each, and the gva mentioned isn't 0.5TB aligned.
Quoted 8 lines
I'm currently trying to understand an obvious inconsistency in the pte describing a page of the virtio-net rx ring. On some guests with some qemu (upstream) command lines I can trigger this with '-smb /some/path' and then doing smbclient -L in the guest. Once the inconsistency slipped in, host and guest see different page contents and virtio-net stops to work. Very strange, but fortunately easily reproducible here. Any hints or debugging suggestions welcome!

Quoted 2 lines

What type of inconsistency? pfn or flags?

The former. Here is a before-after walk of the shadow and the host page table (format: <entry address> (<entry value>) ):

[good]
gva=ffff88001ef22000: 000000002fc57000 -> 000000002fc57880 (000000003d119027) -> 000000003d119000 (000000005d9bf027) -> 000000005d9bf7b8 (000000003d159027) -> 000000003d159910 (000000002fbce063) -> 000000002fbce000 = 01 00 07 00 hva=00007f4665cac000: 00000000701f4000 -> 00000000701f47f0 (000000005d925067) -> 000000005d9258c8 (000000005d8c5067) -> 000000005d8c5970 (000000003d2b4067) -> 000000003d2b4560 (800000002fbce067) -> 800000002fbce000 = 01 00 07 00

[bad]
gva=ffff88001ef22000: 000000002fc57000 -> 000000002fc57880 (000000003d119027) -> 000000003d119000 (000000005d9bf027) -> 000000005d9bf7b8 (000000003d159027) -> 000000003d159910 (000000002fbce063) -> 000000002fbce000 = 01 00 0a 00 hva=00007f4665cac000: 00000000701f4000 -> 00000000701f47f0 (000000005d925067) -> 000000005d9258c8 (000000005d8c5067) -> 000000005d8c5970 (000000003d2b4067) -> 000000003d2b4560 (800000006576d067) -> 800000006576d000 = 01 00 10 00

That raise a question for a kvm-mmu newbie like me:

If a page of the qemu process gets pushed around (here likely due to fork()->exec(smbd)->COW), how will kvm's shadow table catch up? Via MMU_NOTIFIER?

I'm on a 2.6.25 kernel, and that means without CONFIG_MMU_NOTIFIER. So far I assumed that kernels without this feature do not work optimally, but they won't break my guests...

Jan

Gleb Natapov

2009-04-26 11:27 UTC #3 lore raw

Quoted 10 lines · Jan Kiszka

That raise a question for a kvm-mmu newbie like me:
If a page of the qemu process gets pushed around (here likely due to fork()->exec(smbd)->COW), how will kvm's shadow table catch up? Via MMU_NOTIFIER?
I'm on a 2.6.25 kernel, and that means without CONFIG_MMU_NOTIFIER. So far I assumed that kernels without this feature do not work optimally, but they won't break my guests...

Guest memory is not COWed on fork (madvise(MADV_DONTFORK))

--
Gleb.

Avi Kivity

2009-04-26 11:35 UTC #4 lore raw

Quoted 8 lines · Gleb Natapov

Quoted 6 lines · Jan Kiszka
That raise a question for a kvm-mmu newbie like me:
If a page of the qemu process gets pushed around (here likely due to fork()->exec(smbd)->COW), how will kvm's shadow table catch up? Via MMU_NOTIFIER?

Quoted 8 lines

Quoted 5 lines
I'm on a 2.6.25 kernel, and that means without CONFIG_MMU_NOTIFIER. So far I assumed that kernels without this feature do not work optimally, but they won't break my guests...
Guest memory is not COWed on fork (madvise(MADV_DONTFORK))

Which isn't present in upstream.

Signature · 1 line

error compiling committee.c: too many arguments to function

Jan Kiszka

author

2009-04-26 11:36 UTC #5 lore raw

In reply to Gleb Natapov

Quoted 7 lines · Gleb Natapov

Quoted 6 lines · Jan Kiszka
That raise a question for a kvm-mmu newbie like me:
If a page of the qemu process gets pushed around (here likely due to fork()->exec(smbd)->COW), how will kvm's shadow table catch up? Via MMU_NOTIFIER?

Quoted 5 lines

Quoted 4 lines
I'm on a 2.6.25 kernel, and that means without CONFIG_MMU_NOTIFIER. So far I assumed that kernels without this feature do not work optimally, but they won't break my guests...
Guest memory is not COWed on fork (madvise(MADV_DONTFORK))

Yeah... but that's missing upstream! Will cross-check and then post a fix for qemu.

Out of curiosity: What's the mechanism to update the shadow table after swap-out/swap-in?

Jan

Gleb Natapov

2009-04-26 11:39 UTC #6 lore raw

Quoted 12 lines · Jan Kiszka

Quoted 11 lines · Gleb Natapov
Quoted 10 lines · Jan Kiszka
That raise a question for a kvm-mmu newbie like me:
If a page of the qemu process gets pushed around (here likely due to fork()->exec(smbd)->COW), how will kvm's shadow table catch up? Via MMU_NOTIFIER?
I'm on a 2.6.25 kernel, and that means without CONFIG_MMU_NOTIFIER. So far I assumed that kernels without this feature do not work optimally, but they won't break my guests...

Quoted 8 lines

Quoted 1 line
Guest memory is not COWed on fork (madvise(MADV_DONTFORK))
Yeah... but that's missing upstream! Will cross-check and then post a fix for qemu.
Out of curiosity: What's the mechanism to update the shadow table after swap-out/swap-in?

I don't think guest memory is swappable without mmu notifiers.

--
Gleb.

Jan Kiszka

author

2009-04-26 11:42 UTC #7 lore raw

Quoted 13 lines · Gleb Natapov

Quoted 12 lines · Jan Kiszka
Quoted 11 lines · Gleb Natapov
Quoted 10 lines · Jan Kiszka
That raise a question for a kvm-mmu newbie like me:
If a page of the qemu process gets pushed around (here likely due to fork()->exec(smbd)->COW), how will kvm's shadow table catch up? Via MMU_NOTIFIER?
I'm on a 2.6.25 kernel, and that means without CONFIG_MMU_NOTIFIER. So far I assumed that kernels without this feature do not work optimally, but they won't break my guests...

Quoted 8 lines

Quoted 7 lines
Quoted 1 line
Guest memory is not COWed on fork (madvise(MADV_DONTFORK))
Yeah... but that's missing upstream! Will cross-check and then post a fix for qemu.
Out of curiosity: What's the mechanism to update the shadow table after swap-out/swap-in?
I don't think guest memory is swappable without mmu notifiers.

Given the experience with COW: How is this ensured, or where is this done upstream?

Jan

Avi Kivity

2009-04-26 11:42 UTC #8 lore raw

In reply to Jan Kiszka

Quoted 3 lines · Jan Kiszka

Out of curiosity: What's the mechanism to update the shadow table after swap-out/swap-in?

With mmu notifiers, the kernel informs kvm that a host pte has been invalidated, kvm looks in its reverse mappings and drops any sptes that correspond to the same hva.

Without mmu notifiers, the page reference count is kept elevated, so the kernel can't swap anything which has an spte pointing to it.

In both cases, the spte is reestablished on first guest access (which may or may not be immediately after swapin).

Signature · 1 line

error compiling committee.c: too many arguments to function