Skip to contentSkip to conversation

Patching guest kernel code for better performance from HOST

PATCH Dushyant Bansal 2 participants 10 messages Apr 5, 2011 – May 19, 2011

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi all,

I understand that in order to improve performance by reducing vm
exits, kvm uses one shared page between kvm and guest. Now, guest tells the host to map the magic page to '-4096'.

How does kvm make sure that guest will not use this address ( -4096) for other purpose?

Then, guest itself patches its kernel.
I am trying to patch guest from host. So far, I have tried to patch MFMSR instruction. Patching is done when kvm tries to emulate MFMSR. Mechanism used to generate new instruction is same as used by guest in "arch/powerpc/kernel/kvm.c"

arch/powerpc/kvm/book3s_emulate.c +17 -0
@@ -88,6 +88,23 @@ int kvmppc_core_emulate_op(struct kvm_run *run, struct kvm_vcpu *vcpu,
                case OP_31_XOP_MFMSR:
                        kvmppc_set_gpr(vcpu, get_rt(inst),
                                       vcpu->arch.shared->msr);
+
+                       pc = kvmppc_get_pc(vcpu);
+
+                       //if pte.may_write==false then, kvmppc_st will not work
+                       kvmppc_xlate(vcpu,pc,false, &pte);
+                       pte.may_write=true;
+
+                       //Generate new instruction
+                       addr= (-4096L) + offsetof(struct kvm_vcpu_arch_shared, msr);
+                       u32 rt1 = inst & 0x03e00000;
+                       u32 new_inst=0x80000000 | rt1 | ((addr + 4) & 0x0000fffc);
+
+                       //patch only if magic page is mapped; try patching only for 0x7c0000a6 inst
+                       if (vcpu->arch.magic_page_pa==0xfffff000 && inst==0x7c0000a6) {
+                                       kvmppc_st(vcpu,&pc,sizeof(u32),&new_inst,true);
+                                       }
+
                        break;
                case OP_31_XOP_MTMSRD:
                {

But, the problem is guest hangs during boot. I have attached the screenshot of the guest. I am not able to figure out the problem. Is there something wrong with the approach?

I have checked that new_inst generated corresponding to 0x7c0000a6 instruction matches with the instruction that is generated when guest patches itself.

Thanks,
Dushyant

#1 lore raw
Quoted 12 lines · Dushyant Bansal

Hi all,

I understand that in order to improve performance by reducing vm exits, kvm uses one shared page between kvm and guest. Now, guest tells the host to map the magic page to '-4096'.

How does kvm make sure that guest will not use this address ( -4096) for other purpose?

Then, guest itself patches its kernel.
I am trying to patch guest from host. So far, I have tried to patch MFMSR instruction. Patching is done when kvm tries to emulate MFMSR. Mechanism used to generate new instruction is same as used by guest in "arch/powerpc/kernel/kvm.c"

arch/powerpc/kvm/book3s_emulate.c int kvmppc_core_emulate_op(struct kvm_run *run, struct kvm_vcpu *vcpu, · line 96

Quoted 9 lines
arch/powerpc/kvm/book3s_emulate.c +6 -0
@@ -88,6 +88,23 @@ int kvmppc_core_emulate_op(struct kvm_run *run, struct kvm_vcpu *vcpu,
               case OP_31_XOP_MFMSR:
                       kvmppc_set_gpr(vcpu, get_rt(inst),
                                      vcpu->arch.shared->msr);
+
+                       pc = kvmppc_get_pc(vcpu);
+
+                       //if pte.may_write=false then, kvmppc_st will not work
+                       kvmppc_xlate(vcpu,pc,false, &pte);
+                       pte.may_write=true;

The pte struct here is write-only from xlate's point of view. Any modification to it doesn't get reflected to the real translation layer. But your guess is good, I'd also assume that you're hitting a read-only page.

Alex

#2 lore raw
In reply to Dushyant Bansal
Quoted 12 lines · Alexander Graf
Quoted 9 lines · Dushyant Bansal

Hi all,

I understand that in order to improve performance by reducing vm exits, kvm uses one shared page between kvm and guest. Now, guest tells the host to map the magic page to '-4096'.

How does kvm make sure that guest will not use this address ( -4096) for other purpose?

Then, guest itself patches its kernel.
I am trying to patch guest from host. So far, I have tried to patch MFMSR instruction. Patching is done when kvm tries to emulate MFMSR. Mechanism used to generate new instruction is same as used by guest in "arch/powerpc/kernel/kvm.c"

Quoted 15 lines

arch/powerpc/kvm/book3s_emulate.c int kvmppc_core_emulate_op(struct kvm_run *run, struct kvm_vcpu *vcpu, · line 97

Quoted 13 lines
arch/powerpc/kvm/book3s_emulate.c +6 -0
@@ -88,6 +88,23 @@ int kvmppc_core_emulate_op(struct kvm_run *run, struct kvm_vcpu *vcpu,
                case OP_31_XOP_MFMSR:
                        kvmppc_set_gpr(vcpu, get_rt(inst),
                                       vcpu->arch.shared->msr);
+
+                       pc = kvmppc_get_pc(vcpu);
+
+                       //if pte.may_write=false then, kvmppc_st will not work
+                       kvmppc_xlate(vcpu,pc,false,&pte);
+                       pte.may_write=true;
 

The pte struct here is write-only from xlate's point of view. Any modification to it doesn't get reflected to the real translation layer. But your guess is good, I'd also assume that you're hitting a read-only page.

Yes, you are right. Thanks for catching this.
So, if it is a read-only page, 'kvmppc_st(vcpu,&pc,sizeof(u32),&new_inst,true)' will not write anything. Then, there is no reason for guest to hang. Right?

--
Dushyant

#3 lore raw
In reply to Dushyant Bansal
Quoted 36 lines · Dushyant Bansal
Quoted 27 lines · Alexander Graf
Quoted 25 lines · Dushyant Bansal

Hi all,

I understand that in order to improve performance by reducing vm
exits, kvm uses one shared page between kvm and guest. Now, guest tells the host to map the magic page to '-4096'.

How does kvm make sure that guest will not use this address ( -4096) for other purpose?

Then, guest itself patches its kernel.
I am trying to patch guest from host. So far, I have tried to patch MFMSR instruction. Patching is done when kvm tries to emulate MFMSR. Mechanism used to generate new instruction is same as used by guest in "arch/powerpc/kernel/kvm.c"

arch/powerpc/kvm/book3s_emulate.c +3 -0
@@ -88,6 +88,23 @@ int kvmppc_core_emulate_op(struct kvm_run *run, 
 struct kvm_vcpu *vcpu,
                case OP_31_XOP_MFMSR:
                        kvmppc_set_gpr(vcpu, get_rt(inst),
                                       vcpu->arch.shared->msr);
+
+                       pc = kvmppc_get_pc(vcpu);
+
Quoted 8 lines

arch/powerpc/kvm/book3s_emulate.c

Quoted 4 lines
arch/powerpc/kvm/book3s_emulate.c +1 -0
+                       //if pte.may_write=false then, kvmppc_st 

will not work

arch/powerpc/kvm/book3s_emulate.c +2 -0
+                       kvmppc_xlate(vcpu,pc,false,&pte);
+                       pte.may_write=true;

The pte struct here is write-only from xlate's point of view. Any modification to it doesn't get reflected to the real translation layer. But your guess is good, I'd also assume that you're hitting a read-only page.

Quoted 4 lines

Yes, you are right. Thanks for catching this.
So, if it is a read-only page, 'kvmppc_st(vcpu,&pc,sizeof(u32),&new_inst,true)' will not write anything. Then, there is no reason for guest to hang. Right?

Oh? It should inject a page fault. Maybe it doesn't, but then it behaves badly :).

Either way - just enable the debug tracepoints and check the exits you get right after you hit an mfmsr emulation. That should give you hints on what's going wrong.

Alex

#4 lore raw
In reply to Dushyant Bansal
Quoted 24 lines
Quoted 19 lines
Quoted 11 lines
Quoted 11 lines
arch/powerpc/kvm/book3s_emulate.c +3 -0
@@ -88,6 +88,23 @@ int kvmppc_core_emulate_op(struct kvm_run *run, 
 struct kvm_vcpu *vcpu,
                case OP_31_XOP_MFMSR:
                        kvmppc_set_gpr(vcpu, get_rt(inst),
                                       vcpu->arch.shared->msr);
+
+                       pc = kvmppc_get_pc(vcpu);
+
Quoted 8 lines

arch/powerpc/kvm/book3s_emulate.c

Quoted 4 lines
arch/powerpc/kvm/book3s_emulate.c +1 -0
+                       //if pte.may_write=false then, kvmppc_st 

will not work

arch/powerpc/kvm/book3s_emulate.c +2 -0
+                       kvmppc_xlate(vcpu,pc,false,&pte);
+                       pte.may_write=true;

The pte struct here is write-only from xlate's point of view. Any modification to it doesn't get reflected to the real translation layer. But your guess is good, I'd also assume that you're hitting a read-only page.

Quoted 4 lines

Yes, you are right. Thanks for catching this.
So, if it is a read-only page, 'kvmppc_st(vcpu,&pc,sizeof(u32),&new_inst,true)' will not write anything. Then, there is no reason for guest to hang. Right?

Quoted 6 lines

Oh? It should inject a page fault. Maybe it doesn't, but then it behaves badly :).

Either way - just enable the debug tracepoints and check the exits you get right after you hit an mfmsr emulation. That should give you hints on what's going wrong.

I tried a couple of things. But I am still having trouble getting this to work.

It seems to me that problem lies in my method of shared page mapping.

Before patching any instruction, shared page needs to be mapped to guest effective address. I think, this is all that we need for mapping.

+        struct kvmppc_pte pte;
+        vcpu->arch.mmu.xlate(vcpu,vcpu->arch.magic_page_ea,&pte,true);
+        kvmppc_mmu_map_page(vcpu,&pte);

And, then we can start with patching.

arch/powerpc/kvm/book3s_emulate.c +0 -0
@@ -88,6 +88,23 @@ int kvmppc_core_emulate_op(struct kvm_run *run, 

struct kvm_vcpu *vcpu,

                 case OP_31_XOP_MFMSR:
                         kvmppc_set_gpr(vcpu, get_rt(inst),
                                        vcpu->arch.shared->msr);
+
+                       pc = kvmppc_get_pc(vcpu);
+                       //Generate new instruction
+                       addr= (-4096L) + offsetof(struct 

kvm_vcpu_arch_shared, msr);
+ u32 rt1 = inst & 0x03e00000;
+ u32 new_inst=0x80000000 | rt1 | ((addr + 4) &
0x0000fffc);
+
+ //patch only if magic page is mapped; try
patching only for 0x7c0000a6 inst
+ if (vcpu->arch.magic_page_pa=0xfffff000 &&
inst=0x7c0000a6) {
+
kvmppc_st(vcpu,&pc,sizeof(u32),&new_inst,true);
+ }
+

Thanks,
Dushyant

#5 lore raw
In reply to Dushyant Bansal

Hi,

On patching 'mfmsr' instruction with 'lwz', guest exits when it tries to execute that 'lwz' instruction. I am looking for possible causes for this exit.

Here are the details:
Initially,
pc: 0xc0019420, instruction: 0x7c0000a6 [mfmsr r0] As this is a privileged instruction, this causes an exit.

qemu-system-ppc-4443 [000] 19733.740013: kvm_book3s_exit: exit=0x700 | pc=0xc0019420 | inst=0x7c0000a6 | msr=0x1032 | dar=0xe1736a00 | srr1=0x100000000004d032
qemu-system-ppc-4443 [000] 19733.740029: kvm_book3s_patch: return=0 |
pc=0xc0019420 | inst=0x7c0000a6 | msr=0x1032 | new_inst=0x8000f05c
qemu-system-ppc-4443 [000] 19733.740030: kvm_ppc_instr: inst
2080374950 pc 0xc0019420 emulate 0
qemu-system-ppc-4443 [000] 19733.740037: kvm_book3s_reenter: reentry
r=1 | pc=0xc0019420

I patched this instruction with:
0x8000f05c: lwz r0, -4096(offset of msr)
This instruction reads the 'msr' field of the magic page into register r0.

Then, I do not increment the pc value, so the guest starts at the same pc which now points to the new patched instruction.

This 'lwz' instruction is causing a exit due to 'BOOK3S_INTERRUPT_PROGRAM' (exit_nr: 0x700).
What could be the reason for this exit? As, 'lwz' is not a privileged instruction, I am unable to think of any reason.

qemu-system-ppc-4443 [000] 19733.740040: kvm_book3s_exit: exit=0x700

| pc=0xc0019420 | inst=0x8000f05c | msr=0x1032 | dar=0xf00000000105d720 
| srr1=0x100000000004d032
  qemu-system-ppc-4443  [000] 19733.740042: kvm_ppc_instr: inst 

2147545180 pc 0xc0019420 emulate 1
qemu-system-ppc-4443 [000] 19733.740043: kvm_book3s_reenter: reentry
r=3 | pc=0xc0019424
qemu-system-ppc-4443 [000] 19733.740046: kvm_userspace_exit: reason
KVM_EXIT_MMIO (6)

Any help/clue would be greatly appreciated.

Thanks,
Dushyant

#6 lore raw
In reply to Dushyant Bansal
Quoted 14 lines · Dushyant Bansal

Hi,

On patching 'mfmsr' instruction with 'lwz', guest exits when it tries to execute that 'lwz' instruction. I am looking for possible causes for this exit.

Here are the details:
Initially,
pc: 0xc0019420, instruction: 0x7c0000a6 [mfmsr r0] As this is a privileged instruction, this causes an exit.

qemu-system-ppc-4443  [000] 19733.740013: kvm_book3s_exit: exit=0x700 | pc=0xc0019420 | inst=0x7c0000a6 | msr=0x1032 | dar=0xe1736a00 | srr1=0x100000000004d032
qemu-system-ppc-4443  [000] 19733.740029: kvm_book3s_patch: return=0 | pc=0xc0019420 | inst=0x7c0000a6 | msr=0x1032 | new_inst=0x8000f05c
qemu-system-ppc-4443  [000] 19733.740030: kvm_ppc_instr: inst 2080374950 pc 0xc0019420 emulate 0
qemu-system-ppc-4443  [000] 19733.740037: kvm_book3s_reenter: reentry r=1 | pc=0xc0019420
Quoted 8 lines

I patched this instruction with:
0x8000f05c: lwz r0, -4096(offset of msr)
This instruction reads the 'msr' field of the magic page into register r0.

Then, I do not increment the pc value, so the guest starts at the same pc which now points to the new patched instruction.

This 'lwz' instruction is causing a exit due to 'BOOK3S_INTERRUPT_PROGRAM' (exit_nr: 0x700).
What could be the reason for this exit? As, 'lwz' is not a privileged instruction, I am unable to think of any reason.

Did you flush the icache after you patched the instruction? See the function flush_icache_range. Without, your CPU still has the old instruction in its cache, making it trap again :).

Alex

#7 lore raw
In reply to Dushyant Bansal
Quoted 28 lines · Dushyant Bansal
Quoted 20 lines · Alexander Graf
Quoted 18 lines · Dushyant Bansal

Hi,

On patching 'mfmsr' instruction with 'lwz', guest exits when it tries to execute that 'lwz' instruction. I am looking for possible causes for this exit.

Here are the details:
Initially,
pc: 0xc0019420, instruction: 0x7c0000a6 [mfmsr r0] As this is a privileged instruction, this causes an exit.

qemu-system-ppc-4443  [000] 19733.740013: kvm_book3s_exit: exit=0x700 | pc=0xc0019420 | inst=0x7c0000a6 | msr=0x1032 | dar=0xe1736a00 | srr1=0x100000000004d032
qemu-system-ppc-4443  [000] 19733.740029: kvm_book3s_patch: return=0 | pc=0xc0019420 | inst=0x7c0000a6 | msr=0x1032 | new_inst=0x8000f05c
qemu-system-ppc-4443  [000] 19733.740030: kvm_ppc_instr: inst 2080374950 pc 0xc0019420 emulate 0
qemu-system-ppc-4443  [000] 19733.740037: kvm_book3s_reenter: reentry r=1 | pc=0xc0019420

I patched this instruction with:
0x8000f05c: lwz r0, -4096(offset of msr)
This instruction reads the 'msr' field of the magic page into register r0.

Quoted 5 lines
Quoted 4 lines

Then, I do not increment the pc value, so the guest starts at the same pc which now points to the new patched instruction.

This 'lwz' instruction is causing a exit due to 'BOOK3S_INTERRUPT_PROGRAM' (exit_nr: 0x700).
What could be the reason for this exit? As, 'lwz' is not a privileged instruction, I am unable to think of any reason.

Did you flush the icache after you patched the instruction? See the function flush_icache_range. Without, your CPU still has the old instruction in its cache, making it trap again :).

Thanks.

Quoted 8 lines

I tried flush_icache_range((ulong)pc, (ulong)pc + 4);
The system becomes unresponsive and I have to use force shut down.

Here, pc will have the address of guest instruction and flush_icache_range is called from host. Maybe, I am not using flush_icache_range in the correct way. Also, my host os is ppc64 and the guest is ppc32.

I also tried: flush_cache_all()
But the instruction is still present in the instruction cache.

Just patch the _st function to flush the icache on the host virtual address every time it gets invoked :).

Alex

#8 lore raw
In reply to Dushyant Bansal
Quoted 20 lines · Alexander Graf
Quoted 18 lines · Dushyant Bansal

Hi,

On patching 'mfmsr' instruction with 'lwz', guest exits when it tries to execute that 'lwz' instruction. I am looking for possible causes for this exit.

Here are the details:
Initially,
pc: 0xc0019420, instruction: 0x7c0000a6 [mfmsr r0] As this is a privileged instruction, this causes an exit.

qemu-system-ppc-4443  [000] 19733.740013: kvm_book3s_exit: exit=0x700 | pc=0xc0019420 | inst=0x7c0000a6 | msr=0x1032 | dar=0xe1736a00 | srr1=0x100000000004d032
qemu-system-ppc-4443  [000] 19733.740029: kvm_book3s_patch: return=0 | pc=0xc0019420 | inst=0x7c0000a6 | msr=0x1032 | new_inst=0x8000f05c
qemu-system-ppc-4443  [000] 19733.740030: kvm_ppc_instr: inst 2080374950 pc 0xc0019420 emulate 0
qemu-system-ppc-4443  [000] 19733.740037: kvm_book3s_reenter: reentry r=1 | pc=0xc0019420

I patched this instruction with:
0x8000f05c: lwz r0, -4096(offset of msr)
This instruction reads the 'msr' field of the magic page into register r0.

Quoted 5 lines
Quoted 4 lines

Then, I do not increment the pc value, so the guest starts at the same pc which now points to the new patched instruction.

This 'lwz' instruction is causing a exit due to 'BOOK3S_INTERRUPT_PROGRAM' (exit_nr: 0x700).
What could be the reason for this exit? As, 'lwz' is not a privileged instruction, I am unable to think of any reason.

Did you flush the icache after you patched the instruction? See the function flush_icache_range. Without, your CPU still has the old instruction in its cache, making it trap again :).

Thanks.

I tried flush_icache_range((ulong)pc, (ulong)pc + 4);
The system becomes unresponsive and I have to use force shut down.

Here, pc will have the address of guest instruction and flush_icache_range is called from host. Maybe, I am not using flush_icache_range in the correct way.
Also, my host os is ppc64 and the guest is ppc32.

I also tried: flush_cache_all()
But the instruction is still present in the instruction cache.

Dushyant

#9 lore raw
In reply to Dushyant Bansal
Quoted 33 lines · Alexander Graf
Quoted 31 lines · Dushyant Bansal
Quoted 20 lines · Alexander Graf
Quoted 18 lines · Dushyant Bansal

Hi,

On patching 'mfmsr' instruction with 'lwz', guest exits when it tries to execute that 'lwz' instruction. I am looking for possible causes for this exit.

Here are the details:
Initially,
pc: 0xc0019420, instruction: 0x7c0000a6 [mfmsr r0] As this is a privileged instruction, this causes an exit.

qemu-system-ppc-4443  [000] 19733.740013: kvm_book3s_exit: exit=0x700 | pc=0xc0019420 | inst=0x7c0000a6 | msr=0x1032 | dar=0xe1736a00 | srr1=0x100000000004d032
qemu-system-ppc-4443  [000] 19733.740029: kvm_book3s_patch: return=0 | pc=0xc0019420 | inst=0x7c0000a6 | msr=0x1032 | new_inst=0x8000f05c
qemu-system-ppc-4443  [000] 19733.740030: kvm_ppc_instr: inst 2080374950 pc 0xc0019420 emulate 0
qemu-system-ppc-4443  [000] 19733.740037: kvm_book3s_reenter: reentry r=1 | pc=0xc0019420

I patched this instruction with:
0x8000f05c: lwz r0, -4096(offset of msr)
This instruction reads the 'msr' field of the magic page into register r0.

Quoted 5 lines
Quoted 4 lines

Then, I do not increment the pc value, so the guest starts at the same pc which now points to the new patched instruction.

This 'lwz' instruction is causing a exit due to 'BOOK3S_INTERRUPT_PROGRAM' (exit_nr: 0x700).
What could be the reason for this exit? As, 'lwz' is not a privileged instruction, I am unable to think of any reason.

Did you flush the icache after you patched the instruction? See the function flush_icache_range. Without, your CPU still has the old instruction in its cache, making it trap again :).

Thanks.

I tried flush_icache_range((ulong)pc, (ulong)pc + 4);
The system becomes unresponsive and I have to use force shut down.

Quoted 6 lines
Quoted 5 lines

Here, pc will have the address of guest instruction and flush_icache_range is called from host. Maybe, I am not using flush_icache_range in the correct way. Also, my host os is ppc64 and the guest is ppc32.

I also tried: flush_cache_all()
But the instruction is still present in the instruction cache.

Just patch the _st function to flush the icache on the host virtual address every time it gets invoked :).

Sorry for replying so late.
It worked. Thanks a lot :)

Dushyant