Out of sync shadow core breaks Hurd

DISCUSSION Aurelien Jarno 2 participants 5 messages Nov 12, 2008 – Nov 25, 2008

Subsystems (3)

x86 KVM KVM x86

lore

Share & tools

lore

Aurelien Jarno

author

2008-11-12 19:00 UTC #0 lore raw

Hi,

Starting with kvm-76 (and including kvm-79), Hurd does not boot anymore under KVM. The ext2fs translator issues a strange error message:

| Hurd server bootstrap: ext2fs.static[device:hd0s3] execext2fs.static: /build/bui
| ldd/hurd-20080607/build-tree/hurd/ext2fs/dir.c:494: dirscanblock: Assertion `dp-
| >dn->dirents[idx] == -1 || dp->dn->dirents[idx] == nentries' failed.           -
| >dn->dirents[idx] == -1 || dp->dn->dirents[idx] == nentries' failed.

Bisecting the problem, I have found that it comes from this patch:

| 641fb03992b20aa640781a245f6b7136f0b845e4 is first bad commit
| commit 641fb03992b20aa640781a245f6b7136f0b845e4
| Author: Marcelo Tosatti <[email protected]>
| Date:   Tue Sep 23 13:18:39 2008 -0300
| 
|     KVM: MMU: out of sync shadow core v2
| 
|     Allow guest pagetables to go out of sync.
| 
|     Signed-off-by: Marcelo Tosatti <[email protected]>
|     Signed-off-by: Avi Kivity <[email protected]>

The problem can be workarounded loading the kvm module with oos_shadow=0.

The easiest way to reproduce the problem is to download a ready to use Hurd image [1]. The error message from the ext2fs translator is not exactly the same, but it still fails.

Aurelien

[1] http://ftp.debian-ports.org/debian-cd/hurd-i386/current/debian-hurd-k16-qemu.img.tar.gz

Signature · 4 lines

  .''`.  Aurelien Jarno	            | GPG: 1024D/F1BCDB73
 : :' :  Debian developer           | Electrical Engineer
 `. `'   [email protected]         | [email protected]
   `-    people.debian.org/~aurel32 | www.aurel32.net

Marcelo Tosatti

2008-11-15 12:17 UTC #1 lore raw

Quoted 24 lines · Aurelien Jarno

Hi,

Starting with kvm-76 (and including kvm-79), Hurd does not boot anymore under KVM. The ext2fs translator issues a strange error message:

| Hurd server bootstrap: ext2fs.static[device:hd0s3] execext2fs.static: /build/bui
| ldd/hurd-20080607/build-tree/hurd/ext2fs/dir.c:494: dirscanblock: Assertion `dp-
| >dn->dirents[idx] == -1 || dp->dn->dirents[idx] == nentries' failed.           -
| >dn->dirents[idx] == -1 || dp->dn->dirents[idx] == nentries' failed.

Bisecting the problem, I have found that it comes from this patch:

| 641fb03992b20aa640781a245f6b7136f0b845e4 is first bad commit
| commit 641fb03992b20aa640781a245f6b7136f0b845e4
| Author: Marcelo Tosatti <[email protected]>
| Date:   Tue Sep 23 13:18:39 2008 -0300
| 
|     KVM: MMU: out of sync shadow core v2
| 
|     Allow guest pagetables to go out of sync.
| 
|     Signed-off-by: Marcelo Tosatti <[email protected]>
|     Signed-off-by: Avi Kivity <[email protected]>

Quoted 6 lines

The problem can be workarounded loading the kvm module with oos_shadow=0.
The easiest way to reproduce the problem is to download a ready to use Hurd image [1]. The error message from the ext2fs translator is not exactly the same, but it still fails.

Thanks Aurelien, I'll be looking at this next week.

Marcelo Tosatti

2008-11-20 12:50 UTC #2 lore raw

In reply to Aurelien Jarno

Hi Aurelien,

Quoted 24 lines · Aurelien Jarno

Hi,

Starting with kvm-76 (and including kvm-79), Hurd does not boot anymore under KVM. The ext2fs translator issues a strange error message:

| Hurd server bootstrap: ext2fs.static[device:hd0s3] execext2fs.static: /build/bui
| ldd/hurd-20080607/build-tree/hurd/ext2fs/dir.c:494: dirscanblock: Assertion `dp-
| >dn->dirents[idx] == -1 || dp->dn->dirents[idx] == nentries' failed.           -
| >dn->dirents[idx] == -1 || dp->dn->dirents[idx] == nentries' failed.

Bisecting the problem, I have found that it comes from this patch:

| 641fb03992b20aa640781a245f6b7136f0b845e4 is first bad commit
| commit 641fb03992b20aa640781a245f6b7136f0b845e4
| Author: Marcelo Tosatti <[email protected]>
| Date:   Tue Sep 23 13:18:39 2008 -0300
| 
|     KVM: MMU: out of sync shadow core v2
| 
|     Allow guest pagetables to go out of sync.
| 
|     Signed-off-by: Marcelo Tosatti <[email protected]>
|     Signed-off-by: Avi Kivity <[email protected]>

Quoted 6 lines

The problem can be workarounded loading the kvm module with oos_shadow=0.
The easiest way to reproduce the problem is to download a ready to use Hurd image [1]. The error message from the ext2fs translator is not exactly the same, but it still fails.

It seems Hurd does not always explicitly flush the TLB via cr0/cr3/cr4 writes or invlpg after updating pagetables. Debugging shows that OOS is properly syncing the sptes wrt the guest pagetables, and that all pages are synced before guest re-entry on TLB flush exits.

The Intel TLB doc says (5.1 "Invalidation Instructions"):

(Other instructions and operations may invalidate entries in the TLBs and the paging structure caches, but the instructions identified above are recommended.)

As a test, syncing on every exit makes it happy:

arch/x86/kvm/x86.c +2 -0

index 7a2aeba..47e2550 100644
@@ -3052,6 +3052,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
 
 	kvm_lapic_sync_from_vapic(vcpu);
 
+	kvm_mmu_sync_roots(vcpu);
+
 	r = kvm_x86_ops->handle_exit(kvm_run, vcpu);
 out:
 	return r;

It would be necessary to confirm this by hacking Hurd to flush on every pagetable update. Perhaps something like

RCS file: /sources/hurd/gnumach/i386/intel/pmap.c,v retrieving revision 1.4.2.22

pmap.c 20 Nov 2008 12:47:01 -0000 renamed from pmap.c 11 Nov 2008 02:24:18 -0000 1.4.2.22 +1 -1

diff -u -r1.4.2.22 pmap.c
@@ -82,7 +82,7 @@
 #include <i386/proc_reg.h>
 #include <i386/locore.h>
 
-#define    WRITE_PTE(pte_p, pte_entry)     *(pte_p) = (pte_entry);
+#define    WRITE_PTE(pte_p, pte_entry)     *(pte_p) = (pte_entry);

flush_tlb();

/*
* Private data structures.

Aurelien Jarno

author

2008-11-25 09:57 UTC #3 lore raw

Quoted 1 line · Marcelo Tosatti

Hi Aurelien,

Hi,

Quoted 28 lines · Aurelien Jarno

Quoted 27 lines

Hi,

Starting with kvm-76 (and including kvm-79), Hurd does not boot anymore under KVM. The ext2fs translator issues a strange error message:

| Hurd server bootstrap: ext2fs.static[device:hd0s3] execext2fs.static: /build/bui
| ldd/hurd-20080607/build-tree/hurd/ext2fs/dir.c:494: dirscanblock: Assertion `dp-
| >dn->dirents[idx] == -1 || dp->dn->dirents[idx] == nentries' failed.           -
| >dn->dirents[idx] == -1 || dp->dn->dirents[idx] == nentries' failed.

Bisecting the problem, I have found that it comes from this patch:

| 641fb03992b20aa640781a245f6b7136f0b845e4 is first bad commit
| commit 641fb03992b20aa640781a245f6b7136f0b845e4
| Author: Marcelo Tosatti <[email protected]>
| Date:   Tue Sep 23 13:18:39 2008 -0300
| 
|     KVM: MMU: out of sync shadow core v2
| 
|     Allow guest pagetables to go out of sync.
| 
|     Signed-off-by: Marcelo Tosatti <[email protected]>
|     Signed-off-by: Avi Kivity <[email protected]>

The problem can be workarounded loading the kvm module with oos_shadow=0.

Quoted 8 lines

Quoted 3 lines
The easiest way to reproduce the problem is to download a ready to use Hurd image [1]. The error message from the ext2fs translator is not exactly the same, but it still fails.
It seems Hurd does not always explicitly flush the TLB via cr0/cr3/cr4 writes or invlpg after updating pagetables. Debugging shows that OOS is properly syncing the sptes wrt the guest pagetables, and that all pages are synced before guest re-entry on TLB flush exits.

Thanks for your investigation.

Quoted 39 lines

The Intel TLB doc says (5.1 "Invalidation Instructions"):
(Other instructions and operations may invalidate entries in the TLBs and the paging structure caches, but the instructions identified above are recommended.)
As a test, syncing on every exit makes it happy:
arch/x86/kvm/x86.c +2 -0
index 7a2aeba..47e2550 100644
@@ -3052,6 +3052,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
 
 	kvm_lapic_sync_from_vapic(vcpu);
 
+	kvm_mmu_sync_roots(vcpu);
+
 	r = kvm_x86_ops->handle_exit(kvm_run, vcpu);
 out:
 	return r;
It would be necessary to confirm this by hacking Hurd to flush on every pagetable update. Perhaps something like
RCS file: /sources/hurd/gnumach/i386/intel/pmap.c,v retrieving revision 1.4.2.22
diff -u -r1.4.2.22 pmap.c
pmap.c 20 Nov 2008 12:47:01 -0000 renamed from pmap.c 11 Nov 2008 02:24:18 -0000 1.4.2.22 +1 -1
@@ -82,7 +82,7 @@
 #include <i386/proc_reg.h>
 #include <i386/locore.h>
 
-#define    WRITE_PTE(pte_p, pte_entry)     *(pte_p) = (pte_entry);
+#define    WRITE_PTE(pte_p, pte_entry)     *(pte_p) = (pte_entry);
 flush_tlb();
 
 /*

Quoted 3 lines

* Private data structures.

I have tried this patch, but it doesn't change anything. I'll try to see if there are more place when the PTE is written.

Signature · 4 lines

  .''`.  Aurelien Jarno	            | GPG: 1024D/F1BCDB73
 : :' :  Debian developer           | Electrical Engineer
 `. `'   [email protected]         | [email protected]
   `-    people.debian.org/~aurel32 | www.aurel32.net

Aurelien Jarno

author

2008-11-25 16:52 UTC #4 lore raw

Quoted 41 lines · Aurelien Jarno

Quoted 1 line · Marcelo Tosatti
Hi Aurelien,
Hi,
Quoted 36 lines · Aurelien Jarno
Quoted 24 lines
Hi,
Starting with kvm-76 (and including kvm-79), Hurd does not boot anymore under KVM. The ext2fs translator issues a strange error message:
| Hurd server bootstrap: ext2fs.static[device:hd0s3] execext2fs.static: /build/bui
| ldd/hurd-20080607/build-tree/hurd/ext2fs/dir.c:494: dirscanblock: Assertion `dp-
| >dn->dirents[idx] == -1 || dp->dn->dirents[idx] == nentries' failed.           -
| >dn->dirents[idx] == -1 || dp->dn->dirents[idx] == nentries' failed.
Bisecting the problem, I have found that it comes from this patch:
| 641fb03992b20aa640781a245f6b7136f0b845e4 is first bad commit
| commit 641fb03992b20aa640781a245f6b7136f0b845e4
| Author: Marcelo Tosatti <[email protected]>
| Date:   Tue Sep 23 13:18:39 2008 -0300
| 
|     KVM: MMU: out of sync shadow core v2
| 
|     Allow guest pagetables to go out of sync.
| 
|     Signed-off-by: Marcelo Tosatti <[email protected]>
|     Signed-off-by: Avi Kivity <[email protected]>
Quoted 6 lines
The problem can be workarounded loading the kvm module with oos_shadow=0.
The easiest way to reproduce the problem is to download a ready to use Hurd image [1]. The error message from the ext2fs translator is not exactly the same, but it still fails.
It seems Hurd does not always explicitly flush the TLB via cr0/cr3/cr4 writes or invlpg after updating pagetables. Debugging shows that OOS is properly syncing the sptes wrt the guest pagetables, and that all pages are synced before guest re-entry on TLB flush exits.

Looking more precisely at the code, Hurd (actually GNU Mach) flushes the TLB via cr3, but just *before* updating the pagetables. I have no idea why it is done that way, but it seems to be correct given the way the Intel MMU works. However, it fails to comply with the recommendations from Intel ("5.2 Recommended Invalidation"), which if I understand correctly, have been taken as a basis for implementing out of sync shadow.

I have confirmed that by patching the GNU Mach code so that TLB are flushed before and after modifying pagetables.

Signature · 4 lines

  .''`.  Aurelien Jarno	            | GPG: 1024D/F1BCDB73
 : :' :  Debian developer           | Electrical Engineer
 `. `'   [email protected]         | [email protected]
   `-    people.debian.org/~aurel32 | www.aurel32.net