Skip to contentSkip to conversation

Add AMD SEV page encryption bitmap support.

PATCH In review Ashish Kalra 7 participants 38 messages Dec 1, 2020 – Jan 8, 2021 1 message not in the archive

Subsystems (7)
Documentation x86 KVM KVM x86 +3 more subsystems
lore
Share & tools
lore
#0 lore raw

v2 cover Add AMD SEV page encryption bitmap support.

From: Ashish Kalra <[email protected]>

The series add support for AMD SEV page encryption bitmap.

SEV guest VMs have the concept of private and shared memory. Private memory is encrypted with the guest-specific key, while shared memory may be encrypted with hypervisor key. The patch series introduces a new hypercall. The guest OS can use this hypercall to notify the page encryption status.

The patch adds new ioctls KVM_{SET,GET}_PAGE_ENC_BITMAP. The ioctl can be used by qemu to get the page encryption bitmap. Qemu can consult this bitmap during guest live migration / page migration and/or guest debugging to know whether the page is encrypted.

The page encryption bitmap support is required for SEV guest live migration, guest page migration and guest debugging.

The patch-set also adds support for bypassing unencrypted guest memory regions for DBG_DECRYPT API calls, guest memory region encryption status in sev_dbg_decrypt() is now referenced using the page encryption bitmap.

A branch containing these patches is available here:
https://github.com/AMDESE/linux/tree/sev-page-encryption-bitmap-v2

Changes since v1:
- Fix in sev_dbg_crypt() to release RCU read lock if hva_to_gfn() fails when bypassing DBG_DECRYPT API calls for unencrypted guest memory.
- Comment fix for Patch 7/9.

Ashish Kalra (4):
KVM: SVM: Add support for static allocation of unified Page Encryption
Bitmap.
KVM: x86: Mark _bss_decrypted section variables as decrypted in page
encryption bitmap.
KVM: x86: Add kexec support for SEV page encryption bitmap.
KVM: SVM: Bypass DBG_DECRYPT API calls for unecrypted guest memory.

Brijesh Singh (5):
KVM: x86: Add AMD SEV specific Hypercall3
KVM: X86: Introduce KVM_HC_PAGE_ENC_STATUS hypercall
KVM: x86: Introduce KVM_GET_PAGE_ENC_BITMAP ioctl
mm: x86: Invoke hypercall when page encryption status is changed.
KVM: x86: Introduce KVM_SET_PAGE_ENC_BITMAP ioctl

 Documentation/virt/kvm/api.rst        |  71 ++++++
 Documentation/virt/kvm/hypercalls.rst |  15 ++
 arch/x86/include/asm/kvm_host.h       |   7 +
 arch/x86/include/asm/kvm_para.h       |  12 +
 arch/x86/include/asm/mem_encrypt.h    |   4 +
 arch/x86/include/asm/paravirt.h       |  10 +
 arch/x86/include/asm/paravirt_types.h |   2 +
 arch/x86/kernel/kvm.c                 |  28 +++
 arch/x86/kernel/kvmclock.c            |  12 +
 arch/x86/kernel/paravirt.c            |   1 +
 arch/x86/kvm/svm/sev.c                | 321 ++++++++++++++++++++++++++
 arch/x86/kvm/svm/svm.c                |   5 +
 arch/x86/kvm/svm/svm.h                |   7 +
 arch/x86/kvm/vmx/vmx.c                |   1 +
 arch/x86/kvm/x86.c                    |  35 +++
 arch/x86/mm/mem_encrypt.c             |  63 ++++-
 arch/x86/mm/pat/set_memory.c          |   7 +
 include/uapi/linux/kvm.h              |  13 ++
 include/uapi/linux/kvm_para.h         |   1 +
 19 files changed, 614 insertions(+), 1 deletion(-)
Signature · 1 line
2.17.1
#1 lore raw

PATCH v2 1/9 KVM: x86: Add AMD SEV specific Hypercall3

From: Brijesh Singh <[email protected]>

KVM hypercall framework relies on alternative framework to patch the VMCALL -> VMMCALL on AMD platform. If a hypercall is made before apply_alternative() is called then it defaults to VMCALL. The approach works fine on non SEV guest. A VMCALL would causes #UD, and hypervisor will be able to decode the instruction and do the right things. But when SEV is active, guest memory is encrypted with guest key and hypervisor will not be able to decode the instruction bytes.

Add SEV specific hypercall3, it unconditionally uses VMMCALL. The hypercall will be used by the SEV guest to notify encrypted pages to the hypervisor.

---
 arch/x86/include/asm/kvm_para.h | 12 ++++++++++++
 1 file changed, 12 insertions(+)
arch/x86/include/asm/kvm_para.h +12 -0
index 338119852512..bc1b11d057fc 100644
@@ -85,6 +85,18 @@ static inline long kvm_hypercall4(unsigned int nr, unsigned long p1,
 	return ret;
 }
 
+static inline long kvm_sev_hypercall3(unsigned int nr, unsigned long p1,
+				      unsigned long p2, unsigned long p3)
+{
+	long ret;
+
+	asm volatile("vmmcall"
+		     : "=a"(ret)
+		     : "a"(nr), "b"(p1), "c"(p2), "d"(p3)
+		     : "memory");
+	return ret;
+}
+
 #ifdef CONFIG_KVM_GUEST
 bool kvm_para_available(void);
 unsigned int kvm_arch_para_features(void);
Signature · 1 line
2.17.1
#26 lore raw

PATCH v2 2/9 KVM: X86: Introduce KVM_HC_PAGE_ENC_STATUS hypercall

In reply to Ashish Kalra

From: Brijesh Singh <[email protected]>

This hypercall is used by the SEV guest to notify a change in the page encryption status to the hypervisor. The hypercall should be invoked only when the encryption attribute is changed from encrypted -> decrypted and vice versa. By default all guest pages are considered encrypted.

Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Paolo Bonzini <[email protected]>
Cc "Radim Krčmář" <[email protected]>
Cc Joerg Roedel <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Tom Lendacky <[email protected]>
---
 Documentation/virt/kvm/hypercalls.rst | 15 +++++
 arch/x86/include/asm/kvm_host.h       |  2 +
 arch/x86/kvm/svm/sev.c                | 90 +++++++++++++++++++++++++++
 arch/x86/kvm/svm/svm.c                |  2 +
 arch/x86/kvm/svm/svm.h                |  4 ++
 arch/x86/kvm/vmx/vmx.c                |  1 +
 arch/x86/kvm/x86.c                    |  6 ++
 include/uapi/linux/kvm_para.h         |  1 +
 8 files changed, 121 insertions(+)
Documentation/virt/kvm/hypercalls.rst +15 -0
index ed4fddd364ea..7aff0cebab7c 100644
@@ -169,3 +169,18 @@ a0: destination APIC ID
 
 :Usage example: When sending a call-function IPI-many to vCPUs, yield if
 	        any of the IPI target vCPUs was preempted.
+
+
+8. KVM_HC_PAGE_ENC_STATUS
+-------------------------
+:Architecture: x86
+:Status: active
+:Purpose: Notify the encryption status changes in guest page table (SEV guest)
+
+a0: the guest physical address of the start page
+a1: the number of pages
+a2: encryption attribute
+
+   Where:
+	* 1: Encryption attribute is set
+	* 0: Encryption attribute is cleared
arch/x86/include/asm/kvm_host.h +2 -0
index f002cdb13a0b..d035dc983a7a 100644
@@ -1282,6 +1282,8 @@ struct kvm_x86_ops {
 
 	void (*migrate_timers)(struct kvm_vcpu *vcpu);
 	void (*msr_filter_changed)(struct kvm_vcpu *vcpu);
+	int (*page_enc_status_hc)(struct kvm *kvm, unsigned long gpa,
+				  unsigned long sz, unsigned long mode);
 };
 
 struct kvm_x86_nested_ops {
arch/x86/kvm/svm/sev.c +90 -0

102 lines not on this page · raw message

arch/x86/kvm/svm/svm.c +2 -0
index 6dc337b9c231..7122ea5f7c47 100644
@@ -4312,6 +4312,8 @@ static struct kvm_x86_ops svm_x86_ops __initdata = {
 	.apic_init_signal_blocked = svm_apic_init_signal_blocked,
 
 	.msr_filter_changed = svm_msr_filter_changed,
+
+	.page_enc_status_hc = svm_page_enc_status_hc,
 };
 
 static struct kvm_x86_init_ops svm_init_ops __initdata = {
arch/x86/kvm/svm/svm.h +4 -0
index fdff76eb6ceb..0103a23ca174 100644
@@ -66,6 +66,8 @@ struct kvm_sev_info {
 	int fd;			/* SEV device fd */
 	unsigned long pages_locked; /* Number of pages locked */
 	struct list_head regions_list;  /* List of registered regions */
+	unsigned long *page_enc_bmap;
+	unsigned long page_enc_bmap_size;
 };
 
 struct kvm_svm {
@@ -409,6 +411,8 @@ int nested_svm_check_exception(struct vcpu_svm *svm, unsigned nr,
 			       bool has_error_code, u32 error_code);
 int nested_svm_exit_special(struct vcpu_svm *svm);
 void sync_nested_vmcb_control(struct vcpu_svm *svm);
+int svm_page_enc_status_hc(struct kvm *kvm, unsigned long gpa,
+                           unsigned long npages, unsigned long enc);
 
 extern struct kvm_x86_nested_ops svm_nested_ops;
 
arch/x86/kvm/vmx/vmx.c +1 -0
index c3441e7e5a87..5bc37a38e6be 100644
@@ -7722,6 +7722,7 @@ static struct kvm_x86_ops vmx_x86_ops __initdata = {
 
 	.msr_filter_changed = vmx_msr_filter_changed,
 	.cpu_dirty_log_size = vmx_cpu_dirty_log_size,
+	.page_enc_status_hc = NULL,
 };
 
 static __init int hardware_setup(void)
arch/x86/kvm/x86.c +6 -0
index a3fdc16cfd6f..3afc78f18f69 100644
@@ -8125,6 +8125,12 @@ int kvm_emulate_hypercall(struct kvm_vcpu *vcpu)
 		kvm_sched_yield(vcpu->kvm, a0);
 		ret = 0;
 		break;
+	case KVM_HC_PAGE_ENC_STATUS:
+		ret = -KVM_ENOSYS;
+		if (kvm_x86_ops.page_enc_status_hc)
+			ret = kvm_x86_ops.page_enc_status_hc(vcpu->kvm,
+					a0, a1, a2);
+		break;
 	default:
 		ret = -KVM_ENOSYS;
 		break;
include/uapi/linux/kvm_para.h +1 -0
index 8b86609849b9..847b83b75dc8 100644
@@ -29,6 +29,7 @@
 #define KVM_HC_CLOCK_PAIRING		9
 #define KVM_HC_SEND_IPI		10
 #define KVM_HC_SCHED_YIELD		11
+#define KVM_HC_PAGE_ENC_STATUS		12
 
 /*
  * hypercalls use architecture specific
Signature · 1 line
2.17.1
#30 lore raw

PATCH v2 3/9 KVM: x86: Introduce KVM_GET_PAGE_ENC_BITMAP ioctl

In reply to Ashish Kalra

From: Brijesh Singh <[email protected]>

The ioctl can be used to retrieve page encryption bitmap for a given gfn range.

Return the correct bitmap as per the number of pages being requested by the user. Ensure that we only copy bmap->num_pages bytes in the userspace buffer, if bmap->num_pages is not byte aligned we read the trailing bits from the userspace and copy those bits as is.

Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Paolo Bonzini <[email protected]>
Cc "Radim Krčmář" <[email protected]>
Cc Joerg Roedel <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Tom Lendacky <[email protected]>
---
 Documentation/virt/kvm/api.rst  | 27 +++++++++++++
 arch/x86/include/asm/kvm_host.h |  2 +
 arch/x86/kvm/svm/sev.c          | 70 +++++++++++++++++++++++++++++++++
 arch/x86/kvm/svm/svm.c          |  1 +
 arch/x86/kvm/svm/svm.h          |  1 +
 arch/x86/kvm/x86.c              | 12 ++++++
 include/uapi/linux/kvm.h        | 12 ++++++
 7 files changed, 125 insertions(+)
Documentation/virt/kvm/api.rst +27 -0
index 70254eaa5229..ae410f4332ab 100644
@@ -4671,6 +4671,33 @@ This ioctl resets VCPU registers and control structures according to
 the clear cpu reset definition in the POP. However, the cpu is not put
 into ESA mode. This reset is a superset of the initial reset.
 
+4.125 KVM_GET_PAGE_ENC_BITMAP (vm ioctl)
+---------------------------------------
+
+:Capability: basic
+:Architectures: x86
+:Type: vm ioctl
+:Parameters: struct kvm_page_enc_bitmap (in/out)
+:Returns: 0 on success, -1 on error
+
+/* for KVM_GET_PAGE_ENC_BITMAP */
+struct kvm_page_enc_bitmap {
+	__u64 start_gfn;
+	__u64 num_pages;
+	union {
+		void __user *enc_bitmap; /* one bit per page */
+		__u64 padding2;
+	};
+};
+
+The encrypted VMs have the concept of private and shared pages. The private
+pages are encrypted with the guest-specific key, while the shared pages may
+be encrypted with the hypervisor key. The KVM_GET_PAGE_ENC_BITMAP can
+be used to get the bitmap indicating whether the guest page is private
+or shared. The bitmap can be used during the guest migration. If the page
+is private then the userspace need to use SEV migration commands to transmit
+the page.
+
 
 4.125 KVM_S390_PV_COMMAND
 -------------------------
arch/x86/include/asm/kvm_host.h +2 -0
index d035dc983a7a..8c2e40199ecb 100644
@@ -1284,6 +1284,8 @@ struct kvm_x86_ops {
 	void (*msr_filter_changed)(struct kvm_vcpu *vcpu);
 	int (*page_enc_status_hc)(struct kvm *kvm, unsigned long gpa,
 				  unsigned long sz, unsigned long mode);
+	int (*get_page_enc_bitmap)(struct kvm *kvm,
+				struct kvm_page_enc_bitmap *bmap);
 };
 
 struct kvm_x86_nested_ops {
arch/x86/kvm/svm/sev.c +70 -0

76 lines not on this page · raw message

arch/x86/kvm/svm/svm.c +1 -0
index 7122ea5f7c47..bff89cab3ed0 100644
@@ -4314,6 +4314,7 @@ static struct kvm_x86_ops svm_x86_ops __initdata = {
 	.msr_filter_changed = svm_msr_filter_changed,
 
 	.page_enc_status_hc = svm_page_enc_status_hc,
+	.get_page_enc_bitmap = svm_get_page_enc_bitmap,
 };
 
 static struct kvm_x86_init_ops svm_init_ops __initdata = {
arch/x86/kvm/svm/svm.h +1 -0
index 0103a23ca174..4ce73f1034b9 100644
@@ -413,6 +413,7 @@ int nested_svm_exit_special(struct vcpu_svm *svm);
 void sync_nested_vmcb_control(struct vcpu_svm *svm);
 int svm_page_enc_status_hc(struct kvm *kvm, unsigned long gpa,
                            unsigned long npages, unsigned long enc);
+int svm_get_page_enc_bitmap(struct kvm *kvm, struct kvm_page_enc_bitmap *bmap);
 
 extern struct kvm_x86_nested_ops svm_nested_ops;
 
arch/x86/kvm/x86.c +12 -0
index 3afc78f18f69..d3cb95a4dd55 100644
@@ -5695,6 +5695,18 @@ long kvm_arch_vm_ioctl(struct file *filp,
 	case KVM_X86_SET_MSR_FILTER:
 		r = kvm_vm_ioctl_set_msr_filter(kvm, argp);
 		break;
+	case KVM_GET_PAGE_ENC_BITMAP: {
+		struct kvm_page_enc_bitmap bitmap;
+
+		r = -EFAULT;
+		if (copy_from_user(&bitmap, argp, sizeof(bitmap)))
+			goto out;
+
+		r = -ENOTTY;
+		if (kvm_x86_ops.get_page_enc_bitmap)
+			r = kvm_x86_ops.get_page_enc_bitmap(kvm, &bitmap);
+		break;
+	}
 	default:
 		r = -ENOTTY;
 	}
include/uapi/linux/kvm.h +12 -0
index 886802b8ffba..d0b9171bdb03 100644
@@ -532,6 +532,16 @@ struct kvm_dirty_log {
 	};
 };
 
+/* for KVM_GET_PAGE_ENC_BITMAP */
+struct kvm_page_enc_bitmap {
+	__u64 start_gfn;
+	__u64 num_pages;
+	union {
+		void __user *enc_bitmap; /* one bit per page */
+		__u64 padding2;
+	};
+};
+
 /* for KVM_CLEAR_DIRTY_LOG */
 struct kvm_clear_dirty_log {
 	__u32 slot;
@@ -1563,6 +1573,8 @@ struct kvm_pv_cmd {
 /* Available with KVM_CAP_DIRTY_LOG_RING */
 #define KVM_RESET_DIRTY_RINGS		_IO(KVMIO, 0xc7)
 
+#define KVM_GET_PAGE_ENC_BITMAP	_IOW(KVMIO, 0xc6, struct kvm_page_enc_bitmap)
+
 /* Secure Encrypted Virtualization command */
 enum sev_cmd_id {
 	/* Guest initialization commands */
Signature · 1 line
2.17.1
#34 lore raw

PATCH v2 5/9 KVM: x86: Introduce KVM_SET_PAGE_ENC_BITMAP ioctl

In reply to Ashish Kalra

From: Brijesh Singh <[email protected]>

The ioctl can be used to set page encryption bitmap for an incoming guest.

Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Paolo Bonzini <[email protected]>
Cc "Radim Krčmář" <[email protected]>
Cc Joerg Roedel <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Tom Lendacky <[email protected]>
---
 Documentation/virt/kvm/api.rst  | 44 +++++++++++++++++++++++++++++
 arch/x86/include/asm/kvm_host.h |  2 ++
 arch/x86/kvm/svm/sev.c          | 50 +++++++++++++++++++++++++++++++++
 arch/x86/kvm/svm/svm.c          |  1 +
 arch/x86/kvm/svm/svm.h          |  1 +
 arch/x86/kvm/x86.c              | 12 ++++++++
 include/uapi/linux/kvm.h        |  1 +
 7 files changed, 111 insertions(+)
Documentation/virt/kvm/api.rst +44 -0
index ae410f4332ab..1a3336cbbfe8 100644
@@ -4698,6 +4698,28 @@ or shared. The bitmap can be used during the guest migration. If the page
 is private then the userspace need to use SEV migration commands to transmit
 the page.
 
+4.126 KVM_SET_PAGE_ENC_BITMAP (vm ioctl)
+---------------------------------------
+
+:Capability: basic
+:Architectures: x86
+:Type: vm ioctl
+:Parameters: struct kvm_page_enc_bitmap (in/out)
+:Returns: 0 on success, -1 on error
+
+/* for KVM_SET_PAGE_ENC_BITMAP */
+struct kvm_page_enc_bitmap {
+	__u64 start_gfn;
+	__u64 num_pages;
+	union {
+		void __user *enc_bitmap; /* one bit per page */
+		__u64 padding2;
+	};
+};
+
+During the guest live migration the outgoing guest exports its page encryption
+bitmap, the KVM_SET_PAGE_ENC_BITMAP can be used to build the page encryption
+bitmap for an incoming guest.
 
 4.125 KVM_S390_PV_COMMAND
 -------------------------
@@ -4852,6 +4874,28 @@ into user space.
 If a vCPU is in running state while this ioctl is invoked, the vCPU may
 experience inconsistent filtering behavior on MSR accesses.
 
+4.126 KVM_SET_PAGE_ENC_BITMAP (vm ioctl)
+---------------------------------------
+
+:Capability: basic
+:Architectures: x86
+:Type: vm ioctl
+:Parameters: struct kvm_page_enc_bitmap (in/out)
+:Returns: 0 on success, -1 on error
+
+/* for KVM_SET_PAGE_ENC_BITMAP */
+struct kvm_page_enc_bitmap {
+	__u64 start_gfn;
+	__u64 num_pages;
+	union {
+		void __user *enc_bitmap; /* one bit per page */
+		__u64 padding2;
+	};
+};
+
+During the guest live migration the outgoing guest exports its page encryption
+bitmap, the KVM_SET_PAGE_ENC_BITMAP can be used to build the page encryption
+bitmap for an incoming guest.
 
 5. The kvm_run structure
 ========================
arch/x86/include/asm/kvm_host.h +2 -0
index 8c2e40199ecb..352ebc576036 100644
@@ -1286,6 +1286,8 @@ struct kvm_x86_ops {
 				  unsigned long sz, unsigned long mode);
 	int (*get_page_enc_bitmap)(struct kvm *kvm,
 				struct kvm_page_enc_bitmap *bmap);
+	int (*set_page_enc_bitmap)(struct kvm *kvm,
+				struct kvm_page_enc_bitmap *bmap);
 };
 
 struct kvm_x86_nested_ops {
arch/x86/kvm/svm/sev.c +50 -0

56 lines not on this page · raw message

arch/x86/kvm/svm/svm.c +1 -0
index bff89cab3ed0..6ebdf20773ea 100644
@@ -4315,6 +4315,7 @@ static struct kvm_x86_ops svm_x86_ops __initdata = {
 
 	.page_enc_status_hc = svm_page_enc_status_hc,
 	.get_page_enc_bitmap = svm_get_page_enc_bitmap,
+	.set_page_enc_bitmap = svm_set_page_enc_bitmap,
 };
 
 static struct kvm_x86_init_ops svm_init_ops __initdata = {
arch/x86/kvm/svm/svm.h +1 -0
index 4ce73f1034b9..2268c0ab650b 100644
@@ -414,6 +414,7 @@ void sync_nested_vmcb_control(struct vcpu_svm *svm);
 int svm_page_enc_status_hc(struct kvm *kvm, unsigned long gpa,
                            unsigned long npages, unsigned long enc);
 int svm_get_page_enc_bitmap(struct kvm *kvm, struct kvm_page_enc_bitmap *bmap);
+int svm_set_page_enc_bitmap(struct kvm *kvm, struct kvm_page_enc_bitmap *bmap);
 
 extern struct kvm_x86_nested_ops svm_nested_ops;
 
arch/x86/kvm/x86.c +12 -0
index d3cb95a4dd55..3cf64a94004f 100644
@@ -5707,6 +5707,18 @@ long kvm_arch_vm_ioctl(struct file *filp,
 			r = kvm_x86_ops.get_page_enc_bitmap(kvm, &bitmap);
 		break;
 	}
+	case KVM_SET_PAGE_ENC_BITMAP: {
+		struct kvm_page_enc_bitmap bitmap;
+
+		r = -EFAULT;
+		if (copy_from_user(&bitmap, argp, sizeof(bitmap)))
+			goto out;
+
+		r = -ENOTTY;
+		if (kvm_x86_ops.set_page_enc_bitmap)
+			r = kvm_x86_ops.set_page_enc_bitmap(kvm, &bitmap);
+		break;
+	}
 	default:
 		r = -ENOTTY;
 	}
include/uapi/linux/kvm.h +1 -0
index d0b9171bdb03..8e1adcd598a8 100644
@@ -1574,6 +1574,7 @@ struct kvm_pv_cmd {
 #define KVM_RESET_DIRTY_RINGS		_IO(KVMIO, 0xc7)
 
 #define KVM_GET_PAGE_ENC_BITMAP	_IOW(KVMIO, 0xc6, struct kvm_page_enc_bitmap)
+#define KVM_SET_PAGE_ENC_BITMAP	_IOW(KVMIO, 0xc7, struct kvm_page_enc_bitmap)
 
 /* Secure Encrypted Virtualization command */
 enum sev_cmd_id {
Signature · 1 line
2.17.1
#33 lore raw

PATCH v2 4/9 mm: x86: Invoke hypercall when page encryption status is changed.

In reply to Ashish Kalra

From: Brijesh Singh <[email protected]>

Invoke a hypercall when a memory region is changed from encrypted -> decrypted and vice versa. Hypervisor needs to know the page encryption status during the guest migration.

Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Paolo Bonzini <[email protected]>
Cc "Radim Krčmář" <[email protected]>
Cc Joerg Roedel <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Tom Lendacky <[email protected]>
---
 arch/x86/include/asm/paravirt.h       | 10 +++++
 arch/x86/include/asm/paravirt_types.h |  2 +
 arch/x86/kernel/paravirt.c            |  1 +
 arch/x86/mm/mem_encrypt.c             | 57 ++++++++++++++++++++++++++-
 arch/x86/mm/pat/set_memory.c          |  7 ++++
 5 files changed, 76 insertions(+), 1 deletion(-)
arch/x86/include/asm/paravirt.h +10 -0
index d25cc6830e89..7aeb7c508c53 100644
@@ -84,6 +84,12 @@ static inline void paravirt_arch_exit_mmap(struct mm_struct *mm)
 	PVOP_VCALL1(mmu.exit_mmap, mm);
 }
 
+static inline void page_encryption_changed(unsigned long vaddr, int npages,
+						bool enc)
+{
+	PVOP_VCALL3(mmu.page_encryption_changed, vaddr, npages, enc);
+}
+
 #ifdef CONFIG_PARAVIRT_XXL
 static inline void load_sp0(unsigned long sp0)
 {
@@ -840,6 +846,10 @@ static inline void paravirt_arch_dup_mmap(struct mm_struct *oldmm,
 static inline void paravirt_arch_exit_mmap(struct mm_struct *mm)
 {
 }
+
+static inline void page_encryption_changed(unsigned long vaddr, int npages, bool enc)
+{
+}
 #endif
 #endif /* __ASSEMBLY__ */
 #endif /* _ASM_X86_PARAVIRT_H */
arch/x86/include/asm/paravirt_types.h +2 -0
index 0fad9f61c76a..d7787ec4d19f 100644
@@ -209,6 +209,8 @@ struct pv_mmu_ops {
 
 	/* Hook for intercepting the destruction of an mm_struct. */
 	void (*exit_mmap)(struct mm_struct *mm);
+	void (*page_encryption_changed)(unsigned long vaddr, int npages,
+					bool enc);
 
 #ifdef CONFIG_PARAVIRT_XXL
 	struct paravirt_callee_save read_cr2;
arch/x86/kernel/paravirt.c +1 -0
index 6c3407ba6ee9..52913356b6fa 100644
@@ -340,6 +340,7 @@ struct paravirt_patch_template pv_ops = {
 			(void (*)(struct mmu_gather *, void *))tlb_remove_page,
 
 	.mmu.exit_mmap		= paravirt_nop,
+	.mmu.page_encryption_changed	= paravirt_nop,
 
 #ifdef CONFIG_PARAVIRT_XXL
 	.mmu.read_cr2		= __PV_IS_CALLEE_SAVE(native_read_cr2),
arch/x86/mm/mem_encrypt.c +56 -1

98 lines not on this page · raw message

arch/x86/mm/pat/set_memory.c +7 -0
index 40baa90e74f4..dcd4557bb7fa 100644
@@ -27,6 +27,7 @@
 #include <asm/proto.h>
 #include <asm/memtype.h>
 #include <asm/set_memory.h>
+#include <asm/paravirt.h>
 
 #include "../mm_internal.h"
 
@@ -2012,6 +2013,12 @@ static int __set_memory_enc_dec(unsigned long addr, int numpages, bool enc)
 	 */
 	cpa_flush(&cpa, 0);
 
+	/* Notify hypervisor that a given memory range is mapped encrypted
+	 * or decrypted. The hypervisor will use this information during the
+	 * VM migration.
+	 */
+	page_encryption_changed(addr, numpages, enc);
+
 	return ret;
 }
 
Signature · 1 line
2.17.1
#36 lore raw

PATCH v2 7/9 KVM: x86: Mark _bss_decrypted section variables as decrypted in page encryption bitmap.

In reply to Ashish Kalra

From: Ashish Kalra <[email protected]>

Ensure that _bss_decrypted section variables such as hv_clock_boot and wall_clock are marked as decrypted in the page encryption bitmap if sev guest is active.

---
 arch/x86/include/asm/mem_encrypt.h |  4 ++++
 arch/x86/kernel/kvmclock.c         | 12 ++++++++++++
 arch/x86/mm/mem_encrypt.c          |  6 ++++++
 3 files changed, 22 insertions(+)
arch/x86/include/asm/mem_encrypt.h +4 -0
index 2f62bbdd9d12..a4fd6a4229eb 100644
@@ -43,6 +43,8 @@ void __init sme_enable(struct boot_params *bp);
 
 int __init early_set_memory_decrypted(unsigned long vaddr, unsigned long size);
 int __init early_set_memory_encrypted(unsigned long vaddr, unsigned long size);
+void __init early_set_mem_enc_dec_hypercall(unsigned long vaddr, int npages,
+					    bool enc);
 
 void __init mem_encrypt_free_decrypted_mem(void);
 
@@ -82,6 +84,8 @@ static inline int __init
 early_set_memory_decrypted(unsigned long vaddr, unsigned long size) { return 0; }
 static inline int __init
 early_set_memory_encrypted(unsigned long vaddr, unsigned long size) { return 0; }
+static inline void __init
+early_set_mem_enc_dec_hypercall(unsigned long vaddr, int npages, bool enc) {}
 
 static inline void mem_encrypt_free_decrypted_mem(void) { }
 
arch/x86/kernel/kvmclock.c +12 -0
index aa593743acf6..94a4fbf80e44 100644
@@ -333,6 +333,18 @@ void __init kvmclock_init(void)
 	pr_info("kvm-clock: Using msrs %x and %x",
 		msr_kvm_system_time, msr_kvm_wall_clock);
 
+	if (sev_active()) {
+		unsigned long nr_pages;
+		/*
+		 * sizeof(hv_clock_boot) is already PAGE_SIZE aligned
+		 */
+		early_set_mem_enc_dec_hypercall((unsigned long)hv_clock_boot,
+						1, 0);
+		nr_pages = DIV_ROUND_UP(sizeof(wall_clock), PAGE_SIZE);
+		early_set_mem_enc_dec_hypercall((unsigned long)&wall_clock,
+						nr_pages, 0);
+	}
+
 	this_cpu_write(hv_clock_per_cpu, &hv_clock_boot[0]);
 	kvm_register_clock("primary cpu clock");
 	pvclock_set_pvti_cpu0_va(hv_clock_boot);
arch/x86/mm/mem_encrypt.c +6 -0
index 9d1ac65050d0..1bcfbcd2bfd7 100644
@@ -376,6 +376,12 @@ int __init early_set_memory_encrypted(unsigned long vaddr, unsigned long size)
 	return early_set_memory_enc_dec(vaddr, size, true);
 }
 
+void __init early_set_mem_enc_dec_hypercall(unsigned long vaddr, int npages,
+					    bool enc)
+{
+	set_memory_enc_dec_hypercall(vaddr, npages, enc);
+}
+
 /*
  * SME and SEV are very similar but they are not the same, so there are
  * times that the kernel will need to distinguish between SME and SEV. The
Signature · 1 line
2.17.1
#35 lore raw

PATCH v2 6/9 KVM: SVM: Add support for static allocation of unified Page Encryption Bitmap.

In reply to Ashish Kalra

From: Ashish Kalra <[email protected]>

Add support for static allocation of the unified Page encryption bitmap by extending kvm_arch_commit_memory_region() callack to add svm specific x86_ops which can read the userspace provided memory region/memslots and calculate the amount of guest RAM managed by the KVM and grow the bitmap based on that information, i.e. the highest guest PA that is mapped by a memslot.

---
 arch/x86/include/asm/kvm_host.h |  1 +
 arch/x86/kvm/svm/sev.c          | 35 +++++++++++++++++++++++++++++++++
 arch/x86/kvm/svm/svm.c          |  1 +
 arch/x86/kvm/svm/svm.h          |  1 +
 arch/x86/kvm/x86.c              |  5 +++++
 5 files changed, 43 insertions(+)
arch/x86/include/asm/kvm_host.h +1 -0
index 352ebc576036..91fc22d793e8 100644
@@ -1282,6 +1282,7 @@ struct kvm_x86_ops {
 
 	void (*migrate_timers)(struct kvm_vcpu *vcpu);
 	void (*msr_filter_changed)(struct kvm_vcpu *vcpu);
+	void (*commit_memory_region)(struct kvm *kvm, enum kvm_mr_change change);
 	int (*page_enc_status_hc)(struct kvm *kvm, unsigned long gpa,
 				  unsigned long sz, unsigned long mode);
 	int (*get_page_enc_bitmap)(struct kvm *kvm,
arch/x86/kvm/svm/sev.c +35 -0
index e99ea9c711de..8b089cef1eba 100644
@@ -957,6 +957,41 @@ static int sev_resize_page_enc_bitmap(struct kvm *kvm, unsigned long new_size)
 	return 0;
 }
 
+void svm_commit_memory_region(struct kvm *kvm, enum kvm_mr_change change)
+{
+	struct kvm_memslots *slots;
+	struct kvm_memory_slot *memslot;
+	gfn_t start, end = 0;
+
+	spin_lock(&kvm->mmu_lock);
+	if (change == KVM_MR_CREATE) {
+		slots = kvm_memslots(kvm);
+		kvm_for_each_memslot(memslot, slots) {
+			start = memslot->base_gfn;
+			end = memslot->base_gfn + memslot->npages;
+			/*
+			 * KVM memslots is a sorted list, starting with
+			 * the highest mapped guest PA, so pick the topmost
+			 * valid guest PA.
+			 */
+			if (memslot->npages)
+				break;
+		}
+	}
+	spin_unlock(&kvm->mmu_lock);
+
+	if (end) {
+		/*
+		 * NORE: This callback is invoked in vm ioctl
+		 * set_user_memory_region, hence we can use a
+		 * mutex here.
+		 */
+		mutex_lock(&kvm->lock);
+		sev_resize_page_enc_bitmap(kvm, end);
+		mutex_unlock(&kvm->lock);
+	}
+}
+
 int svm_page_enc_status_hc(struct kvm *kvm, unsigned long gpa,
 				  unsigned long npages, unsigned long enc)
 {
arch/x86/kvm/svm/svm.c +1 -0
index 6ebdf20773ea..7aa7858c8209 100644
@@ -4313,6 +4313,7 @@ static struct kvm_x86_ops svm_x86_ops __initdata = {
 
 	.msr_filter_changed = svm_msr_filter_changed,
 
+	.commit_memory_region = svm_commit_memory_region,
 	.page_enc_status_hc = svm_page_enc_status_hc,
 	.get_page_enc_bitmap = svm_get_page_enc_bitmap,
 	.set_page_enc_bitmap = svm_set_page_enc_bitmap,
arch/x86/kvm/svm/svm.h +1 -0
index 2268c0ab650b..5a4656bad681 100644
@@ -415,6 +415,7 @@ int svm_page_enc_status_hc(struct kvm *kvm, unsigned long gpa,
                            unsigned long npages, unsigned long enc);
 int svm_get_page_enc_bitmap(struct kvm *kvm, struct kvm_page_enc_bitmap *bmap);
 int svm_set_page_enc_bitmap(struct kvm *kvm, struct kvm_page_enc_bitmap *bmap);
+void svm_commit_memory_region(struct kvm *kvm, enum kvm_mr_change change);
 
 extern struct kvm_x86_nested_ops svm_nested_ops;
 
arch/x86/kvm/x86.c +5 -0
index 3cf64a94004f..c1acbd397b50 100644
@@ -10717,6 +10717,11 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,
 	/* Free the arrays associated with the old memslot. */
 	if (change == KVM_MR_MOVE)
 		kvm_arch_free_memslot(kvm, old);
+
+	if (change == KVM_MR_CREATE || change == KVM_MR_DELETE) {
+		if (kvm_x86_ops.commit_memory_region)
+			kvm_x86_ops.commit_memory_region(kvm, change);
+	}
 }
 
 void kvm_arch_flush_shadow_all(struct kvm *kvm)
Signature · 1 line
2.17.1
#37 lore raw

PATCH v2 8/9 KVM: x86: Add kexec support for SEV page encryption bitmap.

In reply to Ashish Kalra

From: Ashish Kalra <[email protected]>

Reset the host's page encryption bitmap related to kernel specific page encryption status settings before we load a new kernel by kexec. We cannot reset the complete page encryption bitmap here as we need to retain the UEFI/OVMF firmware specific settings.

The host's page encryption bitmap is maintained for the guest to keep the encrypted/decrypted state of the guest pages, therefore we need to explicitly mark all shared pages as encrypted again before rebooting into the new guest kernel.

---
 arch/x86/kernel/kvm.c | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)
arch/x86/kernel/kvm.c +28 -0
index 7f57ede3cb8e..55d845e025b2 100644
@@ -38,6 +38,7 @@
 #include <asm/cpuidle_haltpoll.h>
 #include <asm/ptrace.h>
 #include <asm/svm.h>
+#include <asm/e820/api.h>
 
 DEFINE_STATIC_KEY_FALSE(kvm_async_pf_enabled);
 
@@ -383,6 +384,33 @@ static void kvm_pv_guest_cpu_reboot(void *unused)
 	 */
 	if (kvm_para_has_feature(KVM_FEATURE_PV_EOI))
 		wrmsrl(MSR_KVM_PV_EOI_EN, 0);
+	/*
+	 * Reset the host's page encryption bitmap related to kernel
+	 * specific page encryption status settings before we load a
+	 * new kernel by kexec. NOTE: We cannot reset the complete
+	 * page encryption bitmap here as we need to retain the
+	 * UEFI/OVMF firmware specific settings.
+	 */
+	if (sev_active() & (smp_processor_id() == 0)) {
+		int i;
+		unsigned long nr_pages;
+
+		for (i = 0; i < e820_table->nr_entries; i++) {
+			struct e820_entry *entry = &e820_table->entries[i];
+			unsigned long start_pfn;
+			unsigned long end_pfn;
+
+			if (entry->type != E820_TYPE_RAM)
+				continue;
+
+			start_pfn = entry->addr >> PAGE_SHIFT;
+			end_pfn = (entry->addr + entry->size) >> PAGE_SHIFT;
+			nr_pages = DIV_ROUND_UP(entry->size, PAGE_SIZE);
+
+			kvm_sev_hypercall3(KVM_HC_PAGE_ENC_STATUS,
+					   entry->addr, nr_pages, 1);
+		}
+	}
 	kvm_pv_disable_apf();
 	kvm_disable_steal_time();
 }
Signature · 1 line
2.17.1
#38 lore raw

PATCH v2 9/9 KVM: SVM: Bypass DBG_DECRYPT API calls for unecrypted guest memory.

In reply to Ashish Kalra

From: Ashish Kalra <[email protected]>

For all explicitly unecrypted guest memory regions such as S/W IOTLB bounce buffers, dma_decrypted() allocated regions and for guest regions marked as "__bss_decrypted", ensure that DBG_DECRYPT API calls are bypassed for such regions. The guest memory regions encryption status is referenced using the page encryption bitmap.

Uses the two added infrastructure functions hva_to_memslot() and hva_to_gfn().

---
 arch/x86/kvm/svm/sev.c | 76 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 76 insertions(+)
arch/x86/kvm/svm/sev.c +76 -0
index 8b089cef1eba..2524a47531ee 100644
@@ -763,6 +763,37 @@ static int __sev_dbg_encrypt_user(struct kvm *kvm, unsigned long paddr,
 	return ret;
 }
 
+static struct kvm_memory_slot *hva_to_memslot(struct kvm *kvm,
+					      unsigned long hva)
+{
+	struct kvm_memslots *slots = kvm_memslots(kvm);
+	struct kvm_memory_slot *memslot;
+
+	kvm_for_each_memslot(memslot, slots) {
+		if (hva >= memslot->userspace_addr &&
+		    hva < memslot->userspace_addr +
+			      (memslot->npages << PAGE_SHIFT))
+			return memslot;
+	}
+
+	return NULL;
+}
+
+static bool hva_to_gfn(struct kvm *kvm, unsigned long hva, gfn_t *gfn)
+{
+	struct kvm_memory_slot *memslot;
+	gpa_t gpa_offset;
+
+	memslot = hva_to_memslot(kvm, hva);
+	if (!memslot)
+		return false;
+
+	gpa_offset = hva - memslot->userspace_addr;
+	*gfn = ((memslot->base_gfn << PAGE_SHIFT) + gpa_offset) >> PAGE_SHIFT;
+
+	return true;
+}
+
 static int sev_dbg_crypt(struct kvm *kvm, struct kvm_sev_cmd *argp, bool dec)
 {
 	unsigned long vaddr, vaddr_end, next_vaddr;
@@ -792,6 +823,50 @@ static int sev_dbg_crypt(struct kvm *kvm, struct kvm_sev_cmd *argp, bool dec)
 	for (; vaddr < vaddr_end; vaddr = next_vaddr) {
 		int len, s_off, d_off;
 
+		if (dec) {
+			struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info;
+			struct page *src_tpage = NULL;
+			gfn_t gfn_start;
+			int srcu_idx;
+
+			/* ensure hva_to_gfn translations remain valid */
+			srcu_idx = srcu_read_lock(&kvm->srcu);
+			if (!hva_to_gfn(kvm, vaddr, &gfn_start)) {
+				srcu_read_unlock(&kvm->srcu, srcu_idx);
+				return -EINVAL;
+			}
+			if (sev->page_enc_bmap) {
+				if (!test_bit(gfn_start, sev->page_enc_bmap)) {
+					src_tpage = alloc_page(GFP_KERNEL);
+					if (!src_tpage) {
+						srcu_read_unlock(&kvm->srcu, srcu_idx);
+						return -ENOMEM;
+					}
+					/*
+					 * Since user buffer may not be page aligned, calculate the
+					 * offset within the page.
+					*/
+					s_off = vaddr & ~PAGE_MASK;
+					d_off = dst_vaddr & ~PAGE_MASK;
+					len = min_t(size_t, (PAGE_SIZE - s_off), size);
+
+					if (copy_from_user(page_address(src_tpage),
+							   (void __user *)(uintptr_t)vaddr, len)) {
+						__free_page(src_tpage);
+						srcu_read_unlock(&kvm->srcu, srcu_idx);
+						return -EFAULT;
+					}
+					if (copy_to_user((void __user *)(uintptr_t)dst_vaddr,
+							 page_address(src_tpage), len)) {
+						ret = -EFAULT;
+					}
+					__free_page(src_tpage);
+					srcu_read_unlock(&kvm->srcu, srcu_idx);
+					goto already_decrypted;
+				}
+			}
+		}
+
 		/* lock userspace source and destination page */
 		src_p = sev_pin_memory(kvm, vaddr & PAGE_MASK, PAGE_SIZE, &n, 0);
 		if (IS_ERR(src_p))
@@ -836,6 +911,7 @@ static int sev_dbg_crypt(struct kvm *kvm, struct kvm_sev_cmd *argp, bool dec)
 		sev_unpin_memory(kvm, src_p, n);
 		sev_unpin_memory(kvm, dst_p, n);
 
+already_decrypted:
 		if (ret)
 			goto err;
 
Signature · 1 line
2.17.1
#27 lore raw

Re: [PATCH v2 2/9] KVM: X86: Introduce KVM_HC_PAGE_ENC_STATUS hypercall

In reply to Ashish Kalra
Quoted 6 lines · * Ashish Kalra ([email protected]) wrote:

From: Brijesh Singh <[email protected]>

This hypercall is used by the SEV guest to notify a change in the page encryption status to the hypervisor. The hypercall should be invoked only when the encryption attribute is changed from encrypted -> decrypted and vice versa. By default all guest pages are considered encrypted.

Is it defined whether these are supposed to be called before or after the the page type has been changed; is it change the type and then notify or the other way around?

Dave

Quoted 248 lines
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Paolo Bonzini <[email protected]>
Cc "Radim Krčmář" <[email protected]>
Cc Joerg Roedel <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Tom Lendacky <[email protected]>
---
 Documentation/virt/kvm/hypercalls.rst | 15 +++++
 arch/x86/include/asm/kvm_host.h       |  2 +
 arch/x86/kvm/svm/sev.c                | 90 +++++++++++++++++++++++++++
 arch/x86/kvm/svm/svm.c                |  2 +
 arch/x86/kvm/svm/svm.h                |  4 ++
 arch/x86/kvm/vmx/vmx.c                |  1 +
 arch/x86/kvm/x86.c                    |  6 ++
 include/uapi/linux/kvm_para.h         |  1 +
 8 files changed, 121 insertions(+)
Documentation/virt/kvm/hypercalls.rst +15 -0
index ed4fddd364ea..7aff0cebab7c 100644
@@ -169,3 +169,18 @@ a0: destination APIC ID
 
 :Usage example: When sending a call-function IPI-many to vCPUs, yield if
 	        any of the IPI target vCPUs was preempted.
+
+
+8. KVM_HC_PAGE_ENC_STATUS
+-------------------------
+:Architecture: x86
+:Status: active
+:Purpose: Notify the encryption status changes in guest page table (SEV guest)
+
+a0: the guest physical address of the start page
+a1: the number of pages
+a2: encryption attribute
+
+   Where:
+	* 1: Encryption attribute is set
+	* 0: Encryption attribute is cleared
arch/x86/include/asm/kvm_host.h +2 -0
index f002cdb13a0b..d035dc983a7a 100644
@@ -1282,6 +1282,8 @@ struct kvm_x86_ops {
 
 	void (*migrate_timers)(struct kvm_vcpu *vcpu);
 	void (*msr_filter_changed)(struct kvm_vcpu *vcpu);
+	int (*page_enc_status_hc)(struct kvm *kvm, unsigned long gpa,
+				  unsigned long sz, unsigned long mode);
 };
 
 struct kvm_x86_nested_ops {
arch/x86/kvm/svm/sev.c +90 -0
index c0b14106258a..6b8bc1297f9c 100644
@@ -927,6 +927,93 @@ static int sev_launch_secret(struct kvm *kvm, struct kvm_sev_cmd *argp)
 	return ret;
 }
 
+static int sev_resize_page_enc_bitmap(struct kvm *kvm, unsigned long new_size)
+{
+	struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info;
+	unsigned long *map;
+	unsigned long sz;
+
+	if (sev->page_enc_bmap_size >= new_size)
+		return 0;
+
+	sz = ALIGN(new_size, BITS_PER_LONG) / 8;
+
+	map = vmalloc(sz);
+	if (!map) {
+		pr_err_once("Failed to allocate encrypted bitmap size %lx\n",
+				sz);
+		return -ENOMEM;
+	}
+
+	/* mark the page encrypted (by default) */
+	memset(map, 0xff, sz);
+
+	bitmap_copy(map, sev->page_enc_bmap, sev->page_enc_bmap_size);
+	kvfree(sev->page_enc_bmap);
+
+	sev->page_enc_bmap = map;
+	sev->page_enc_bmap_size = new_size;
+
+	return 0;
+}
+
+int svm_page_enc_status_hc(struct kvm *kvm, unsigned long gpa,
+				  unsigned long npages, unsigned long enc)
+{
+	struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info;
+	kvm_pfn_t pfn_start, pfn_end;
+	gfn_t gfn_start, gfn_end;
+
+	if (!sev_guest(kvm))
+		return -EINVAL;
+
+	if (!npages)
+		return 0;
+
+	gfn_start = gpa_to_gfn(gpa);
+	gfn_end = gfn_start + npages;
+
+	/* out of bound access error check */
+	if (gfn_end <= gfn_start)
+		return -EINVAL;
+
+	/* lets make sure that gpa exist in our memslot */
+	pfn_start = gfn_to_pfn(kvm, gfn_start);
+	pfn_end = gfn_to_pfn(kvm, gfn_end);
+
+	if (is_error_noslot_pfn(pfn_start) && !is_noslot_pfn(pfn_start)) {
+		/*
+		 * Allow guest MMIO range(s) to be added
+		 * to the page encryption bitmap.
+		 */
+		return -EINVAL;
+	}
+
+	if (is_error_noslot_pfn(pfn_end) && !is_noslot_pfn(pfn_end)) {
+		/*
+		 * Allow guest MMIO range(s) to be added
+		 * to the page encryption bitmap.
+		 */
+		return -EINVAL;
+	}
+
+	mutex_lock(&kvm->lock);
+
+	if (sev->page_enc_bmap_size < gfn_end)
+		goto unlock;
+
+	if (enc)
+		__bitmap_set(sev->page_enc_bmap, gfn_start,
+				gfn_end - gfn_start);
+	else
+		__bitmap_clear(sev->page_enc_bmap, gfn_start,
+				gfn_end - gfn_start);
+
+unlock:
+	mutex_unlock(&kvm->lock);
+	return 0;
+}
+
 int svm_mem_enc_op(struct kvm *kvm, void __user *argp)
 {
 	struct kvm_sev_cmd sev_cmd;
@@ -1123,6 +1210,9 @@ void sev_vm_destroy(struct kvm *kvm)
 
 	sev_unbind_asid(kvm, sev->handle);
 	sev_asid_free(sev->asid);
+
+	kvfree(sev->page_enc_bmap);
+	sev->page_enc_bmap = NULL;
 }
 
 int __init sev_hardware_setup(void)
arch/x86/kvm/svm/svm.c +2 -0
index 6dc337b9c231..7122ea5f7c47 100644
@@ -4312,6 +4312,8 @@ static struct kvm_x86_ops svm_x86_ops __initdata = {
 	.apic_init_signal_blocked = svm_apic_init_signal_blocked,
 
 	.msr_filter_changed = svm_msr_filter_changed,
+
+	.page_enc_status_hc = svm_page_enc_status_hc,
 };
 
 static struct kvm_x86_init_ops svm_init_ops __initdata = {
arch/x86/kvm/svm/svm.h +4 -0
index fdff76eb6ceb..0103a23ca174 100644
@@ -66,6 +66,8 @@ struct kvm_sev_info {
 	int fd;			/* SEV device fd */
 	unsigned long pages_locked; /* Number of pages locked */
 	struct list_head regions_list;  /* List of registered regions */
+	unsigned long *page_enc_bmap;
+	unsigned long page_enc_bmap_size;
 };
 
 struct kvm_svm {
@@ -409,6 +411,8 @@ int nested_svm_check_exception(struct vcpu_svm *svm, unsigned nr,
 			       bool has_error_code, u32 error_code);
 int nested_svm_exit_special(struct vcpu_svm *svm);
 void sync_nested_vmcb_control(struct vcpu_svm *svm);
+int svm_page_enc_status_hc(struct kvm *kvm, unsigned long gpa,
+                           unsigned long npages, unsigned long enc);
 
 extern struct kvm_x86_nested_ops svm_nested_ops;
 
arch/x86/kvm/vmx/vmx.c +1 -0
index c3441e7e5a87..5bc37a38e6be 100644
@@ -7722,6 +7722,7 @@ static struct kvm_x86_ops vmx_x86_ops __initdata = {
 
 	.msr_filter_changed = vmx_msr_filter_changed,
 	.cpu_dirty_log_size = vmx_cpu_dirty_log_size,
+	.page_enc_status_hc = NULL,
 };
 
 static __init int hardware_setup(void)
arch/x86/kvm/x86.c +6 -0
index a3fdc16cfd6f..3afc78f18f69 100644
@@ -8125,6 +8125,12 @@ int kvm_emulate_hypercall(struct kvm_vcpu *vcpu)
 		kvm_sched_yield(vcpu->kvm, a0);
 		ret = 0;
 		break;
+	case KVM_HC_PAGE_ENC_STATUS:
+		ret = -KVM_ENOSYS;
+		if (kvm_x86_ops.page_enc_status_hc)
+			ret = kvm_x86_ops.page_enc_status_hc(vcpu->kvm,
+					a0, a1, a2);
+		break;
 	default:
 		ret = -KVM_ENOSYS;
 		break;
include/uapi/linux/kvm_para.h +1 -0
index 8b86609849b9..847b83b75dc8 100644
@@ -29,6 +29,7 @@
 #define KVM_HC_CLOCK_PAIRING		9
 #define KVM_HC_SEND_IPI		10
 #define KVM_HC_SCHED_YIELD		11
+#define KVM_HC_PAGE_ENC_STATUS		12
 
 /*
  * hypercalls use architecture specific
Signature · 1 line
2.17.1
Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
#28 lore raw

Re: [PATCH v2 2/9] KVM: X86: Introduce KVM_HC_PAGE_ENC_STATUS hypercall

Hello Dave,

Quoted 12 lines · Dr. David Alan Gilbert
Quoted 6 lines · * Ashish Kalra ([email protected]) wrote:

From: Brijesh Singh <[email protected]>

This hypercall is used by the SEV guest to notify a change in the page encryption status to the hypervisor. The hypercall should be invoked only when the encryption attribute is changed from encrypted -> decrypted and vice versa. By default all guest pages are considered encrypted.

Is it defined whether these are supposed to be called before or after the the page type has been changed; is it change the type and then notify or the other way around?

There is nothing really specified as such, the guest makes the hypercall immediately after modifying the page tables. There is surely going to be some latency before the VMM knows about this and the guest page table update.

Thanks,
Ashish

Quoted 252 lines
Quoted 249 lines
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Paolo Bonzini <[email protected]>
Cc "Radim Krčmář" <[email protected]>
Cc Joerg Roedel <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Tom Lendacky <[email protected]>
---
 Documentation/virt/kvm/hypercalls.rst | 15 +++++
 arch/x86/include/asm/kvm_host.h       |  2 +
 arch/x86/kvm/svm/sev.c                | 90 +++++++++++++++++++++++++++
 arch/x86/kvm/svm/svm.c                |  2 +
 arch/x86/kvm/svm/svm.h                |  4 ++
 arch/x86/kvm/vmx/vmx.c                |  1 +
 arch/x86/kvm/x86.c                    |  6 ++
 include/uapi/linux/kvm_para.h         |  1 +
 8 files changed, 121 insertions(+)
Documentation/virt/kvm/hypercalls.rst +15 -0
index ed4fddd364ea..7aff0cebab7c 100644
@@ -169,3 +169,18 @@ a0: destination APIC ID
 
 :Usage example: When sending a call-function IPI-many to vCPUs, yield if
 	        any of the IPI target vCPUs was preempted.
+
+
+8. KVM_HC_PAGE_ENC_STATUS
+-------------------------
+:Architecture: x86
+:Status: active
+:Purpose: Notify the encryption status changes in guest page table (SEV guest)
+
+a0: the guest physical address of the start page
+a1: the number of pages
+a2: encryption attribute
+
+   Where:
+	* 1: Encryption attribute is set
+	* 0: Encryption attribute is cleared
arch/x86/include/asm/kvm_host.h +2 -0
index f002cdb13a0b..d035dc983a7a 100644
@@ -1282,6 +1282,8 @@ struct kvm_x86_ops {
 
 	void (*migrate_timers)(struct kvm_vcpu *vcpu);
 	void (*msr_filter_changed)(struct kvm_vcpu *vcpu);
+	int (*page_enc_status_hc)(struct kvm *kvm, unsigned long gpa,
+				  unsigned long sz, unsigned long mode);
 };
 
 struct kvm_x86_nested_ops {
arch/x86/kvm/svm/sev.c +90 -0
index c0b14106258a..6b8bc1297f9c 100644
@@ -927,6 +927,93 @@ static int sev_launch_secret(struct kvm *kvm, struct kvm_sev_cmd *argp)
 	return ret;
 }
 
+static int sev_resize_page_enc_bitmap(struct kvm *kvm, unsigned long new_size)
+{
+	struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info;
+	unsigned long *map;
+	unsigned long sz;
+
+	if (sev->page_enc_bmap_size >= new_size)
+		return 0;
+
+	sz = ALIGN(new_size, BITS_PER_LONG) / 8;
+
+	map = vmalloc(sz);
+	if (!map) {
+		pr_err_once("Failed to allocate encrypted bitmap size %lx\n",
+				sz);
+		return -ENOMEM;
+	}
+
+	/* mark the page encrypted (by default) */
+	memset(map, 0xff, sz);
+
+	bitmap_copy(map, sev->page_enc_bmap, sev->page_enc_bmap_size);
+	kvfree(sev->page_enc_bmap);
+
+	sev->page_enc_bmap = map;
+	sev->page_enc_bmap_size = new_size;
+
+	return 0;
+}
+
+int svm_page_enc_status_hc(struct kvm *kvm, unsigned long gpa,
+				  unsigned long npages, unsigned long enc)
+{
+	struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info;
+	kvm_pfn_t pfn_start, pfn_end;
+	gfn_t gfn_start, gfn_end;
+
+	if (!sev_guest(kvm))
+		return -EINVAL;
+
+	if (!npages)
+		return 0;
+
+	gfn_start = gpa_to_gfn(gpa);
+	gfn_end = gfn_start + npages;
+
+	/* out of bound access error check */
+	if (gfn_end <= gfn_start)
+		return -EINVAL;
+
+	/* lets make sure that gpa exist in our memslot */
+	pfn_start = gfn_to_pfn(kvm, gfn_start);
+	pfn_end = gfn_to_pfn(kvm, gfn_end);
+
+	if (is_error_noslot_pfn(pfn_start) && !is_noslot_pfn(pfn_start)) {
+		/*
+		 * Allow guest MMIO range(s) to be added
+		 * to the page encryption bitmap.
+		 */
+		return -EINVAL;
+	}
+
+	if (is_error_noslot_pfn(pfn_end) && !is_noslot_pfn(pfn_end)) {
+		/*
+		 * Allow guest MMIO range(s) to be added
+		 * to the page encryption bitmap.
+		 */
+		return -EINVAL;
+	}
+
+	mutex_lock(&kvm->lock);
+
+	if (sev->page_enc_bmap_size < gfn_end)
+		goto unlock;
+
+	if (enc)
+		__bitmap_set(sev->page_enc_bmap, gfn_start,
+				gfn_end - gfn_start);
+	else
+		__bitmap_clear(sev->page_enc_bmap, gfn_start,
+				gfn_end - gfn_start);
+
+unlock:
+	mutex_unlock(&kvm->lock);
+	return 0;
+}
+
 int svm_mem_enc_op(struct kvm *kvm, void __user *argp)
 {
 	struct kvm_sev_cmd sev_cmd;
@@ -1123,6 +1210,9 @@ void sev_vm_destroy(struct kvm *kvm)
 
 	sev_unbind_asid(kvm, sev->handle);
 	sev_asid_free(sev->asid);
+
+	kvfree(sev->page_enc_bmap);
+	sev->page_enc_bmap = NULL;
 }
 
 int __init sev_hardware_setup(void)
arch/x86/kvm/svm/svm.c +2 -0
index 6dc337b9c231..7122ea5f7c47 100644
@@ -4312,6 +4312,8 @@ static struct kvm_x86_ops svm_x86_ops __initdata = {
 	.apic_init_signal_blocked = svm_apic_init_signal_blocked,
 
 	.msr_filter_changed = svm_msr_filter_changed,
+
+	.page_enc_status_hc = svm_page_enc_status_hc,
 };
 
 static struct kvm_x86_init_ops svm_init_ops __initdata = {
arch/x86/kvm/svm/svm.h +4 -0
index fdff76eb6ceb..0103a23ca174 100644
@@ -66,6 +66,8 @@ struct kvm_sev_info {
 	int fd;			/* SEV device fd */
 	unsigned long pages_locked; /* Number of pages locked */
 	struct list_head regions_list;  /* List of registered regions */
+	unsigned long *page_enc_bmap;
+	unsigned long page_enc_bmap_size;
 };
 
 struct kvm_svm {
@@ -409,6 +411,8 @@ int nested_svm_check_exception(struct vcpu_svm *svm, unsigned nr,
 			       bool has_error_code, u32 error_code);
 int nested_svm_exit_special(struct vcpu_svm *svm);
 void sync_nested_vmcb_control(struct vcpu_svm *svm);
+int svm_page_enc_status_hc(struct kvm *kvm, unsigned long gpa,
+                           unsigned long npages, unsigned long enc);
 
 extern struct kvm_x86_nested_ops svm_nested_ops;
 
arch/x86/kvm/vmx/vmx.c +1 -0
index c3441e7e5a87..5bc37a38e6be 100644
@@ -7722,6 +7722,7 @@ static struct kvm_x86_ops vmx_x86_ops __initdata = {
 
 	.msr_filter_changed = vmx_msr_filter_changed,
 	.cpu_dirty_log_size = vmx_cpu_dirty_log_size,
+	.page_enc_status_hc = NULL,
 };
 
 static __init int hardware_setup(void)
arch/x86/kvm/x86.c +6 -0
index a3fdc16cfd6f..3afc78f18f69 100644
@@ -8125,6 +8125,12 @@ int kvm_emulate_hypercall(struct kvm_vcpu *vcpu)
 		kvm_sched_yield(vcpu->kvm, a0);
 		ret = 0;
 		break;
+	case KVM_HC_PAGE_ENC_STATUS:
+		ret = -KVM_ENOSYS;
+		if (kvm_x86_ops.page_enc_status_hc)
+			ret = kvm_x86_ops.page_enc_status_hc(vcpu->kvm,
+					a0, a1, a2);
+		break;
 	default:
 		ret = -KVM_ENOSYS;
 		break;
include/uapi/linux/kvm_para.h +1 -0
index 8b86609849b9..847b83b75dc8 100644
@@ -29,6 +29,7 @@
 #define KVM_HC_CLOCK_PAIRING		9
 #define KVM_HC_SEND_IPI		10
 #define KVM_HC_SCHED_YIELD		11
+#define KVM_HC_PAGE_ENC_STATUS		12
 
 /*
  * hypercalls use architecture specific
Signature · 1 line
2.17.1
Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
#2 lore raw

Re: [PATCH v2 1/9] KVM: x86: Add AMD SEV specific Hypercall3

In reply to Ashish Kalra
Quoted 10 lines · Ashish Kalra

From: Brijesh Singh <[email protected]>

KVM hypercall framework relies on alternative framework to patch the VMCALL -> VMMCALL on AMD platform. If a hypercall is made before apply_alternative() is called then it defaults to VMCALL. The approach works fine on non SEV guest. A VMCALL would causes #UD, and hypervisor will be able to decode the instruction and do the right things. But when SEV is active, guest memory is encrypted with guest key and hypervisor will not be able to decode the instruction bytes.

Quoted 2 lines

Add SEV specific hypercall3, it unconditionally uses VMMCALL. The hypercall will be used by the SEV guest to notify encrypted pages to the hypervisor.

What if we invert KVM_HYPERCALL and X86_FEATURE_VMMCALL to default to VMMCALL and opt into VMCALL? It's a synthetic feature flag either way, and I don't think there are any existing KVM hypercalls that happen before alternatives are patched, i.e. it'll be a nop for sane kernel builds.

I'm also skeptical that a KVM specific hypercall is the right approach for the encryption behavior, but I'll take that up in the patches later in the series.

Quoted 45 lines
---
 arch/x86/include/asm/kvm_para.h | 12 ++++++++++++
 1 file changed, 12 insertions(+)
arch/x86/include/asm/kvm_para.h +12 -0
index 338119852512..bc1b11d057fc 100644
@@ -85,6 +85,18 @@ static inline long kvm_hypercall4(unsigned int nr, unsigned long p1,
 	return ret;
 }
 
+static inline long kvm_sev_hypercall3(unsigned int nr, unsigned long p1,
+				      unsigned long p2, unsigned long p3)
+{
+	long ret;
+
+	asm volatile("vmmcall"
+		     : "=a"(ret)
+		     : "a"(nr), "b"(p1), "c"(p2), "d"(p3)
+		     : "memory");
+	return ret;
+}
+
 #ifdef CONFIG_KVM_GUEST
 bool kvm_para_available(void);
 unsigned int kvm_arch_para_features(void);
Signature · 1 line
2.17.1
#3 lore raw

Re: [PATCH v2 1/9] KVM: x86: Add AMD SEV specific Hypercall3

Quoted 11 lines · Sean Christopherson
Quoted 10 lines · Ashish Kalra

From: Brijesh Singh <[email protected]>

KVM hypercall framework relies on alternative framework to patch the VMCALL -> VMMCALL on AMD platform. If a hypercall is made before apply_alternative() is called then it defaults to VMCALL. The approach works fine on non SEV guest. A VMCALL would causes #UD, and hypervisor will be able to decode the instruction and do the right things. But when SEV is active, guest memory is encrypted with guest key and hypervisor will not be able to decode the instruction bytes.

Quoted 6 lines
Quoted 2 lines

Add SEV specific hypercall3, it unconditionally uses VMMCALL. The hypercall will be used by the SEV guest to notify encrypted pages to the hypervisor.

What if we invert KVM_HYPERCALL and X86_FEATURE_VMMCALL to default to VMMCALL and opt into VMCALL? It's a synthetic feature flag either way, and I don't think there are any existing KVM hypercalls that happen before alternatives are patched, i.e. it'll be a nop for sane kernel builds.

If we invert the X86_FEATURE_VMMCALL to default to VMMCALL then it should work fine without this patch. So far there was no hypercall made before the alternative patching took place. Since the page state change can occur much before the alternative patching so we need to default to VMMCALL when SEV is active.

Quoted 2 lines

I'm also skeptical that a KVM specific hypercall is the right approach for the encryption behavior, but I'll take that up in the patches later in the series.

Great, I am open to explore other alternative approaches.

Quoted 46 lines
Quoted 46 lines
---
 arch/x86/include/asm/kvm_para.h | 12 ++++++++++++
 1 file changed, 12 insertions(+)
arch/x86/include/asm/kvm_para.h +12 -0
index 338119852512..bc1b11d057fc 100644
@@ -85,6 +85,18 @@ static inline long kvm_hypercall4(unsigned int nr, unsigned long p1,
 	return ret;
 }
 
+static inline long kvm_sev_hypercall3(unsigned int nr, unsigned long p1,
+				      unsigned long p2, unsigned long p3)
+{
+	long ret;
+
+	asm volatile("vmmcall"
+		     : "=a"(ret)
+		     : "a"(nr), "b"(p1), "c"(p2), "d"(p3)
+		     : "memory");
+	return ret;
+}
+
 #ifdef CONFIG_KVM_GUEST
 bool kvm_para_available(void);
 unsigned int kvm_arch_para_features(void);
Signature · 1 line
2.17.1
maintainer
#29 lore raw

Re: [PATCH v2 2/9] KVM: X86: Introduce KVM_HC_PAGE_ENC_STATUS hypercall

In reply to Ashish Kalra
Quoted 14 lines · Ashish Kalra

Hello Dave,

Quoted 11 lines · Dr. David Alan Gilbert
Quoted 6 lines · * Ashish Kalra ([email protected]) wrote:

From: Brijesh Singh <[email protected]>

This hypercall is used by the SEV guest to notify a change in the page encryption status to the hypervisor. The hypercall should be invoked only when the encryption attribute is changed from encrypted -> decrypted and vice versa. By default all guest pages are considered encrypted.

Is it defined whether these are supposed to be called before or after the the page type has been changed; is it change the type and then notify or the other way around?

It doesn't matter. However, you have do it before writing to the page, and the content of the page is unspecified between the hypercall and the write to the page.

So you cannot for example encrypt a page in place (using the same PFN but different settings of the C bit), you need to do:

	hypercall(); /* mark enc_data as encrypted */
	/*
	 * The contents of enc_data is now undefined as it can change
	 * across migration.
	 */
	memset(enc_data, unenc_data, PAGE_SIZE);

Paolo

maintainer
#4 lore raw

Re: [PATCH v2 1/9] KVM: x86: Add AMD SEV specific Hypercall3

In reply to Sean Christopherson
Quoted 14 lines · Sean Christopherson
Quoted 10 lines · Ashish Kalra

From: Brijesh Singh <[email protected]>

KVM hypercall framework relies on alternative framework to patch the VMCALL -> VMMCALL on AMD platform. If a hypercall is made before apply_alternative() is called then it defaults to VMCALL. The approach works fine on non SEV guest. A VMCALL would causes #UD, and hypervisor will be able to decode the instruction and do the right things. But when SEV is active, guest memory is encrypted with guest key and hypervisor will not be able to decode the instruction bytes.

Quoted 2 lines

Add SEV specific hypercall3, it unconditionally uses VMMCALL. The hypercall will be used by the SEV guest to notify encrypted pages to the hypervisor.

Quoted 7 lines

What if we invert KVM_HYPERCALL and X86_FEATURE_VMMCALL to default to VMMCALL and opt into VMCALL? It's a synthetic feature flag either way, and I don't think there are any existing KVM hypercalls that happen before alternatives are patched, i.e. it'll be a nop for sane kernel builds.

I'm also skeptical that a KVM specific hypercall is the right approach for the encryption behavior, but I'll take that up in the patches later in the series.

Do you think that it's the guest that should "donate" memory for the bitmap instead?

Paolo

Quoted 47 lines
Quoted 46 lines
---
  arch/x86/include/asm/kvm_para.h | 12 ++++++++++++
  1 file changed, 12 insertions(+)
arch/x86/include/asm/kvm_para.h +12 -0
index 338119852512..bc1b11d057fc 100644
@@ -85,6 +85,18 @@ static inline long kvm_hypercall4(unsigned int nr, unsigned long p1,
  	return ret;
  }
 
+static inline long kvm_sev_hypercall3(unsigned int nr, unsigned long p1,
+				      unsigned long p2, unsigned long p3)
+{
+	long ret;
+
+	asm volatile("vmmcall"
+		     : "=a"(ret)
+		     : "a"(nr), "b"(p1), "c"(p2), "d"(p3)
+		     : "memory");
+	return ret;
+}
+
  #ifdef CONFIG_KVM_GUEST
  bool kvm_para_available(void);
  unsigned int kvm_arch_para_features(void);
Signature · 1 line
2.17.1
#31 lore raw

Re: [PATCH v2 3/9] KVM: x86: Introduce KVM_GET_PAGE_ENC_BITMAP ioctl

In reply to Ashish Kalra
Quoted 9 lines · Ashish Kalra

From: Brijesh Singh <[email protected]>

The ioctl can be used to retrieve page encryption bitmap for a given gfn range.

Return the correct bitmap as per the number of pages being requested by the user. Ensure that we only copy bmap->num_pages bytes in the userspace buffer, if bmap->num_pages is not byte aligned we read the trailing bits from the userspace and copy those bits as is.

I think you meant to say "Ensure that we only copy bmap->num_pages *bits* in the userspace buffer". But maybe I'm missed something.

Quoted 91 lines
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Paolo Bonzini <[email protected]>
Cc "Radim Krčmář" <[email protected]>
Cc Joerg Roedel <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Tom Lendacky <[email protected]>
---
  Documentation/virt/kvm/api.rst  | 27 +++++++++++++
  arch/x86/include/asm/kvm_host.h |  2 +
  arch/x86/kvm/svm/sev.c          | 70 +++++++++++++++++++++++++++++++++
  arch/x86/kvm/svm/svm.c          |  1 +
  arch/x86/kvm/svm/svm.h          |  1 +
  arch/x86/kvm/x86.c              | 12 ++++++
  include/uapi/linux/kvm.h        | 12 ++++++
  7 files changed, 125 insertions(+)
Documentation/virt/kvm/api.rst +27 -0
index 70254eaa5229..ae410f4332ab 100644
@@ -4671,6 +4671,33 @@ This ioctl resets VCPU registers and control structures according to
  the clear cpu reset definition in the POP. However, the cpu is not put
  into ESA mode. This reset is a superset of the initial reset.
 
+4.125 KVM_GET_PAGE_ENC_BITMAP (vm ioctl)
+---------------------------------------
+
+:Capability: basic
+:Architectures: x86
+:Type: vm ioctl
+:Parameters: struct kvm_page_enc_bitmap (in/out)
+:Returns: 0 on success, -1 on error
+
+/* for KVM_GET_PAGE_ENC_BITMAP */
+struct kvm_page_enc_bitmap {
+	__u64 start_gfn;
+	__u64 num_pages;
+	union {
+		void __user *enc_bitmap; /* one bit per page */
+		__u64 padding2;
+	};
+};
+
+The encrypted VMs have the concept of private and shared pages. The private
+pages are encrypted with the guest-specific key, while the shared pages may
+be encrypted with the hypervisor key. The KVM_GET_PAGE_ENC_BITMAP can
+be used to get the bitmap indicating whether the guest page is private
+or shared. The bitmap can be used during the guest migration. If the page
+is private then the userspace need to use SEV migration commands to transmit
+the page.
+
 
  4.125 KVM_S390_PV_COMMAND
  -------------------------
arch/x86/include/asm/kvm_host.h +2 -0
index d035dc983a7a..8c2e40199ecb 100644
@@ -1284,6 +1284,8 @@ struct kvm_x86_ops {
  	void (*msr_filter_changed)(struct kvm_vcpu *vcpu);
  	int (*page_enc_status_hc)(struct kvm *kvm, unsigned long gpa,
  				  unsigned long sz, unsigned long mode);
+	int (*get_page_enc_bitmap)(struct kvm *kvm,
+				struct kvm_page_enc_bitmap *bmap);
  };
 
  struct kvm_x86_nested_ops {
arch/x86/kvm/svm/sev.c +7 -0
index 6b8bc1297f9c..a6586dd29767 100644
@@ -1014,6 +1014,76 @@ int svm_page_enc_status_hc(struct kvm *kvm, unsigned long gpa,
  	return 0;
  }
 
+int svm_get_page_enc_bitmap(struct kvm *kvm,
+				   struct kvm_page_enc_bitmap *bmap)
+{
+	struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info;
+	unsigned long gfn_start, gfn_end;
+	unsigned long sz, i, sz_bytes;
+	unsigned long *bitmap;

arch/x86/kvm/svm/sev.c int svm_page_enc_status_hc(struct kvm *kvm, unsigned long gpa, · line 1033

Quoted 10 lines
arch/x86/kvm/svm/sev.c +10 -0
+	int ret, n;
+
+	if (!sev_guest(kvm))
+		return -ENOTTY;
+
+	gfn_start = bmap->start_gfn;
+	gfn_end = gfn_start + bmap->num_pages;
+
+	sz = ALIGN(bmap->num_pages, BITS_PER_LONG) / BITS_PER_BYTE;
+	bitmap = kmalloc(sz, GFP_KERNEL);

Maybe use bitmap_alloc which accepts size in bits (and corresponding bitmap_free)?

arch/x86/kvm/svm/sev.c int svm_page_enc_status_hc(struct kvm *kvm, unsigned long gpa, · line 1038

Quoted 5 lines
arch/x86/kvm/svm/sev.c +5 -0
+	if (!bitmap)
+		return -ENOMEM;
+
+	/* by default all pages are marked encrypted */
+	memset(bitmap, 0xff, sz);

Maybe use bitmap_fill to clarify the intent?

Quoted 4 lines
arch/x86/kvm/svm/sev.c +4 -0
+
+	mutex_lock(&kvm->lock);
+	if (sev->page_enc_bmap) {
+		i = gfn_start;

arch/x86/kvm/svm/sev.c int svm_page_enc_status_hc(struct kvm *kvm, unsigned long gpa, · line 1052

Quoted 10 lines
arch/x86/kvm/svm/sev.c +10 -0
+		for_each_clear_bit_from(i, sev->page_enc_bmap,
+				      min(sev->page_enc_bmap_size, gfn_end))
+			clear_bit(i - gfn_start, bitmap);
+	}
+	mutex_unlock(&kvm->lock);
+
+	ret = -EFAULT;
+
+	n = bmap->num_pages % BITS_PER_BYTE;
+	sz_bytes = ALIGN(bmap->num_pages, BITS_PER_BYTE) / BITS_PER_BYTE;

Maybe clearer:

	sz_bytes = BITS_TO_BYTES(bmap->num_pages);

arch/x86/kvm/svm/sev.c int svm_page_enc_status_hc(struct kvm *kvm, unsigned long gpa, · line 1060

Quoted 8 lines
arch/x86/kvm/svm/sev.c +8 -0
+
+	/*
+	 * Return the correct bitmap as per the number of pages being
+	 * requested by the user. Ensure that we only copy bmap->num_pages
+	 * bytes in the userspace buffer, if bmap->num_pages is not byte
+	 * aligned we read the trailing bits from the userspace and copy
+	 * those bits as is.
+	 */

(see my comment on the commit message above.)

Quoted 98 lines
arch/x86/kvm/svm/sev.c +26 -0
+
+	if (n) {
+		unsigned char *bitmap_kernel = (unsigned char *)bitmap;
+		unsigned char bitmap_user;
+		unsigned long offset, mask;
+
+		offset = bmap->num_pages / BITS_PER_BYTE;
+		if (copy_from_user(&bitmap_user, bmap->enc_bitmap + offset,
+				sizeof(unsigned char)))
+			goto out;
+
+		mask = GENMASK(n - 1, 0);
+		bitmap_user &= ~mask;
+		bitmap_kernel[offset] &= mask;
+		bitmap_kernel[offset] |= bitmap_user;
+	}
+
+	if (copy_to_user(bmap->enc_bitmap, bitmap, sz_bytes))
+		goto out;
+
+	ret = 0;
+out:
+	kfree(bitmap);
+	return ret;
+}
+
  int svm_mem_enc_op(struct kvm *kvm, void __user *argp)
  {
  	struct kvm_sev_cmd sev_cmd;
arch/x86/kvm/svm/svm.c +1 -0
index 7122ea5f7c47..bff89cab3ed0 100644
@@ -4314,6 +4314,7 @@ static struct kvm_x86_ops svm_x86_ops __initdata = {
  	.msr_filter_changed = svm_msr_filter_changed,
 
  	.page_enc_status_hc = svm_page_enc_status_hc,
+	.get_page_enc_bitmap = svm_get_page_enc_bitmap,
  };
 
  static struct kvm_x86_init_ops svm_init_ops __initdata = {
arch/x86/kvm/svm/svm.h +1 -0
index 0103a23ca174..4ce73f1034b9 100644
@@ -413,6 +413,7 @@ int nested_svm_exit_special(struct vcpu_svm *svm);
  void sync_nested_vmcb_control(struct vcpu_svm *svm);
  int svm_page_enc_status_hc(struct kvm *kvm, unsigned long gpa,
                             unsigned long npages, unsigned long enc);
+int svm_get_page_enc_bitmap(struct kvm *kvm, struct kvm_page_enc_bitmap *bmap);
 
  extern struct kvm_x86_nested_ops svm_nested_ops;
 
arch/x86/kvm/x86.c +12 -0
index 3afc78f18f69..d3cb95a4dd55 100644
@@ -5695,6 +5695,18 @@ long kvm_arch_vm_ioctl(struct file *filp,
  	case KVM_X86_SET_MSR_FILTER:
  		r = kvm_vm_ioctl_set_msr_filter(kvm, argp);
  		break;
+	case KVM_GET_PAGE_ENC_BITMAP: {
+		struct kvm_page_enc_bitmap bitmap;
+
+		r = -EFAULT;
+		if (copy_from_user(&bitmap, argp, sizeof(bitmap)))
+			goto out;
+
+		r = -ENOTTY;
+		if (kvm_x86_ops.get_page_enc_bitmap)
+			r = kvm_x86_ops.get_page_enc_bitmap(kvm, &bitmap);
+		break;
+	}
  	default:
  		r = -ENOTTY;
  	}
include/uapi/linux/kvm.h +10 -0
index 886802b8ffba..d0b9171bdb03 100644
@@ -532,6 +532,16 @@ struct kvm_dirty_log {
  	};
  };
 
+/* for KVM_GET_PAGE_ENC_BITMAP */
+struct kvm_page_enc_bitmap {
+	__u64 start_gfn;
+	__u64 num_pages;
+	union {
+		void __user *enc_bitmap; /* one bit per page */
+		__u64 padding2;
+	};
+};
+
  /* for KVM_CLEAR_DIRTY_LOG */
  struct kvm_clear_dirty_log {
  	__u32 slot;

include/uapi/linux/kvm.h struct kvm_pv_cmd { · line 1576

Quoted 4 lines
include/uapi/linux/kvm.h +1 -0
@@ -1563,6 +1573,8 @@ struct kvm_pv_cmd {
  /* Available with KVM_CAP_DIRTY_LOG_RING */
  #define KVM_RESET_DIRTY_RINGS		_IO(KVMIO, 0xc7)
 
+#define KVM_GET_PAGE_ENC_BITMAP	_IOW(KVMIO, 0xc6, struct kvm_page_enc_bitmap)

I see that kvm/next already defines ioctls numbered 0xc6 and 0xc7. Wouldn't these new ioctls (KVM_GET_PAGE_ENC_BITMAP, KVM_SET_PAGE_ENC_BITMAP) collide?

include/uapi/linux/kvm.h struct kvm_pv_cmd { · line 1580

Quoted 5 lines
include/uapi/linux/kvm.h +1 -0
+
  /* Secure Encrypted Virtualization command */
  enum sev_cmd_id {
  	/* Guest initialization commands */

-Dov

#5 lore raw

Re: [PATCH v2 1/9] KVM: x86: Add AMD SEV specific Hypercall3

In reply to Paolo Bonzini
Quoted 20 lines · Paolo Bonzini
Quoted 19 lines · Sean Christopherson
Quoted 10 lines · Ashish Kalra

From: Brijesh Singh <[email protected]>

KVM hypercall framework relies on alternative framework to patch the VMCALL -> VMMCALL on AMD platform. If a hypercall is made before apply_alternative() is called then it defaults to VMCALL. The approach works fine on non SEV guest. A VMCALL would causes #UD, and hypervisor will be able to decode the instruction and do the right things. But when SEV is active, guest memory is encrypted with guest key and hypervisor will not be able to decode the instruction bytes.

Quoted 2 lines

Add SEV specific hypercall3, it unconditionally uses VMMCALL. The hypercall will be used by the SEV guest to notify encrypted pages to the hypervisor.

What if we invert KVM_HYPERCALL and X86_FEATURE_VMMCALL to default to VMMCALL and opt into VMCALL? It's a synthetic feature flag either way, and I don't think there are any existing KVM hypercalls that happen before alternatives are patched, i.e. it'll be a nop for sane kernel builds.

Quoted 5 lines
Quoted 2 lines

I'm also skeptical that a KVM specific hypercall is the right approach for the encryption behavior, but I'll take that up in the patches later in the series.

Do you think that it's the guest that should "donate" memory for the bitmap instead?

No. Two things I'd like to explore:

1. Making the hypercall to announce/request private vs. shared common across hypervisors (KVM, Hyper-V, VMware, etc...) and technologies (SEV-* and TDX). I'm concerned that we'll end up with multiple hypercalls that do more or less the same thing, e.g. KVM+SEV, Hyper-V+SEV, TDX, etc... Maybe it's a pipe dream, but I'd like to at least explore options before shoving in KVM- only hypercalls.

2. Tracking shared memory via a list of ranges instead of a using bitmap to track all of guest memory. For most use cases, the vast majority of guest memory will be private, most ranges will be 2mb+, and conversions between private and shared will be uncommon events, i.e. the overhead to walk and split/merge list entries is hopefully not a big concern. I suspect a list would consume far less memory, hopefully without impacting performance.

#32 lore raw

Re: [PATCH v2 3/9] KVM: x86: Introduce KVM_GET_PAGE_ENC_BITMAP ioctl

In reply to Dov Murik

Hello Dov,

Quoted 8 lines · Dov Murik
Quoted 5 lines · Ashish Kalra

From: Brijesh Singh <[email protected]>

The ioctl can be used to retrieve page encryption bitmap for a given gfn range.

Quoted 8 lines
Quoted 4 lines

Return the correct bitmap as per the number of pages being requested by the user. Ensure that we only copy bmap->num_pages bytes in the userspace buffer, if bmap->num_pages is not byte aligned we read the trailing bits from the userspace and copy those bits as is.

I think you meant to say "Ensure that we only copy bmap->num_pages *bits* in the userspace buffer". But maybe I'm missed something.

Yes, that is correct. It should read bmap->num_pages *bits* instead of *bytes*, i will fix the comments.

Quoted 97 lines
Quoted 97 lines
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Paolo Bonzini <[email protected]>
Cc "Radim Krčmář" <[email protected]>
Cc Joerg Roedel <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Tom Lendacky <[email protected]>
---
  Documentation/virt/kvm/api.rst  | 27 +++++++++++++
  arch/x86/include/asm/kvm_host.h |  2 +
  arch/x86/kvm/svm/sev.c          | 70 +++++++++++++++++++++++++++++++++
  arch/x86/kvm/svm/svm.c          |  1 +
  arch/x86/kvm/svm/svm.h          |  1 +
  arch/x86/kvm/x86.c              | 12 ++++++
  include/uapi/linux/kvm.h        | 12 ++++++
  7 files changed, 125 insertions(+)
Documentation/virt/kvm/api.rst +27 -0
index 70254eaa5229..ae410f4332ab 100644
@@ -4671,6 +4671,33 @@ This ioctl resets VCPU registers and control structures according to
  the clear cpu reset definition in the POP. However, the cpu is not put
  into ESA mode. This reset is a superset of the initial reset.
 
+4.125 KVM_GET_PAGE_ENC_BITMAP (vm ioctl)
+---------------------------------------
+
+:Capability: basic
+:Architectures: x86
+:Type: vm ioctl
+:Parameters: struct kvm_page_enc_bitmap (in/out)
+:Returns: 0 on success, -1 on error
+
+/* for KVM_GET_PAGE_ENC_BITMAP */
+struct kvm_page_enc_bitmap {
+	__u64 start_gfn;
+	__u64 num_pages;
+	union {
+		void __user *enc_bitmap; /* one bit per page */
+		__u64 padding2;
+	};
+};
+
+The encrypted VMs have the concept of private and shared pages. The private
+pages are encrypted with the guest-specific key, while the shared pages may
+be encrypted with the hypervisor key. The KVM_GET_PAGE_ENC_BITMAP can
+be used to get the bitmap indicating whether the guest page is private
+or shared. The bitmap can be used during the guest migration. If the page
+is private then the userspace need to use SEV migration commands to transmit
+the page.
+
 
  4.125 KVM_S390_PV_COMMAND
  -------------------------
arch/x86/include/asm/kvm_host.h +2 -0
index d035dc983a7a..8c2e40199ecb 100644
@@ -1284,6 +1284,8 @@ struct kvm_x86_ops {
  	void (*msr_filter_changed)(struct kvm_vcpu *vcpu);
  	int (*page_enc_status_hc)(struct kvm *kvm, unsigned long gpa,
  				  unsigned long sz, unsigned long mode);
+	int (*get_page_enc_bitmap)(struct kvm *kvm,
+				struct kvm_page_enc_bitmap *bmap);
  };
 
  struct kvm_x86_nested_ops {
arch/x86/kvm/svm/sev.c +12 -0
index 6b8bc1297f9c..a6586dd29767 100644
@@ -1014,6 +1014,76 @@ int svm_page_enc_status_hc(struct kvm *kvm, unsigned long gpa,
  	return 0;
  }
 
+int svm_get_page_enc_bitmap(struct kvm *kvm,
+				   struct kvm_page_enc_bitmap *bmap)
+{
+	struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info;
+	unsigned long gfn_start, gfn_end;
+	unsigned long sz, i, sz_bytes;
+	unsigned long *bitmap;
+	int ret, n;
+
+	if (!sev_guest(kvm))
+		return -ENOTTY;
+
Quoted 9 lines

arch/x86/kvm/svm/sev.c int svm_page_enc_status_hc(struct kvm *kvm, unsigned long gpa, · line 1033

Quoted 5 lines
arch/x86/kvm/svm/sev.c +5 -0
+	gfn_start = bmap->start_gfn;
+	gfn_end = gfn_start + bmap->num_pages;
+
+	sz = ALIGN(bmap->num_pages, BITS_PER_LONG) / BITS_PER_BYTE;
+	bitmap = kmalloc(sz, GFP_KERNEL);

Maybe use bitmap_alloc which accepts size in bits (and corresponding bitmap_free)?

I will look at this.

Quoted 9 lines

arch/x86/kvm/svm/sev.c int svm_page_enc_status_hc(struct kvm *kvm, unsigned long gpa, · line 1038

Quoted 6 lines
arch/x86/kvm/svm/sev.c +5 -0
+	if (!bitmap)
+		return -ENOMEM;
+
+	/* by default all pages are marked encrypted */
+	memset(bitmap, 0xff, sz);

Maybe use bitmap_fill to clarify the intent?

Again, i will look at this.

Quoted 22 lines
Quoted 5 lines
arch/x86/kvm/svm/sev.c +4 -0
+
+	mutex_lock(&kvm->lock);
+	if (sev->page_enc_bmap) {
+		i = gfn_start;

arch/x86/kvm/svm/sev.c int svm_page_enc_status_hc(struct kvm *kvm, unsigned long gpa, · line 1052

Quoted 10 lines
arch/x86/kvm/svm/sev.c +10 -0
+		for_each_clear_bit_from(i, sev->page_enc_bmap,
+				      min(sev->page_enc_bmap_size, gfn_end))
+			clear_bit(i - gfn_start, bitmap);
+	}
+	mutex_unlock(&kvm->lock);
+
+	ret = -EFAULT;
+
+	n = bmap->num_pages % BITS_PER_BYTE;
+	sz_bytes = ALIGN(bmap->num_pages, BITS_PER_BYTE) / BITS_PER_BYTE;

Maybe clearer:

	sz_bytes = BITS_TO_BYTES(bmap->num_pages);
Quoted 11 lines

arch/x86/kvm/svm/sev.c int svm_page_enc_status_hc(struct kvm *kvm, unsigned long gpa, · line 1060

Quoted 8 lines
arch/x86/kvm/svm/sev.c +8 -0
+
+	/*
+	 * Return the correct bitmap as per the number of pages being
+	 * requested by the user. Ensure that we only copy bmap->num_pages
+	 * bytes in the userspace buffer, if bmap->num_pages is not byte
+	 * aligned we read the trailing bits from the userspace and copy
+	 * those bits as is.
+	 */

(see my comment on the commit message above.)

Yes, as i mentioned above, this need to be bmap->num pages *bits* and not *bytes*.

Quoted 102 lines
Quoted 102 lines
arch/x86/kvm/svm/sev.c +26 -0
+
+	if (n) {
+		unsigned char *bitmap_kernel = (unsigned char *)bitmap;
+		unsigned char bitmap_user;
+		unsigned long offset, mask;
+
+		offset = bmap->num_pages / BITS_PER_BYTE;
+		if (copy_from_user(&bitmap_user, bmap->enc_bitmap + offset,
+				sizeof(unsigned char)))
+			goto out;
+
+		mask = GENMASK(n - 1, 0);
+		bitmap_user &= ~mask;
+		bitmap_kernel[offset] &= mask;
+		bitmap_kernel[offset] |= bitmap_user;
+	}
+
+	if (copy_to_user(bmap->enc_bitmap, bitmap, sz_bytes))
+		goto out;
+
+	ret = 0;
+out:
+	kfree(bitmap);
+	return ret;
+}
+
  int svm_mem_enc_op(struct kvm *kvm, void __user *argp)
  {
  	struct kvm_sev_cmd sev_cmd;
arch/x86/kvm/svm/svm.c +1 -0
index 7122ea5f7c47..bff89cab3ed0 100644
@@ -4314,6 +4314,7 @@ static struct kvm_x86_ops svm_x86_ops __initdata = {
  	.msr_filter_changed = svm_msr_filter_changed,
 
  	.page_enc_status_hc = svm_page_enc_status_hc,
+	.get_page_enc_bitmap = svm_get_page_enc_bitmap,
  };
 
  static struct kvm_x86_init_ops svm_init_ops __initdata = {
arch/x86/kvm/svm/svm.h +1 -0
index 0103a23ca174..4ce73f1034b9 100644
@@ -413,6 +413,7 @@ int nested_svm_exit_special(struct vcpu_svm *svm);
  void sync_nested_vmcb_control(struct vcpu_svm *svm);
  int svm_page_enc_status_hc(struct kvm *kvm, unsigned long gpa,
                             unsigned long npages, unsigned long enc);
+int svm_get_page_enc_bitmap(struct kvm *kvm, struct kvm_page_enc_bitmap *bmap);
 
  extern struct kvm_x86_nested_ops svm_nested_ops;
 
arch/x86/kvm/x86.c +12 -0
index 3afc78f18f69..d3cb95a4dd55 100644
@@ -5695,6 +5695,18 @@ long kvm_arch_vm_ioctl(struct file *filp,
  	case KVM_X86_SET_MSR_FILTER:
  		r = kvm_vm_ioctl_set_msr_filter(kvm, argp);
  		break;
+	case KVM_GET_PAGE_ENC_BITMAP: {
+		struct kvm_page_enc_bitmap bitmap;
+
+		r = -EFAULT;
+		if (copy_from_user(&bitmap, argp, sizeof(bitmap)))
+			goto out;
+
+		r = -ENOTTY;
+		if (kvm_x86_ops.get_page_enc_bitmap)
+			r = kvm_x86_ops.get_page_enc_bitmap(kvm, &bitmap);
+		break;
+	}
  	default:
  		r = -ENOTTY;
  	}
include/uapi/linux/kvm.h +10 -0
index 886802b8ffba..d0b9171bdb03 100644
@@ -532,6 +532,16 @@ struct kvm_dirty_log {
  	};
  };
 
+/* for KVM_GET_PAGE_ENC_BITMAP */
+struct kvm_page_enc_bitmap {
+	__u64 start_gfn;
+	__u64 num_pages;
+	union {
+		void __user *enc_bitmap; /* one bit per page */
+		__u64 padding2;
+	};
+};
+
  /* for KVM_CLEAR_DIRTY_LOG */
  struct kvm_clear_dirty_log {
  	__u32 slot;
@@ -1563,6 +1573,8 @@ struct kvm_pv_cmd {
  /* Available with KVM_CAP_DIRTY_LOG_RING */
  #define KVM_RESET_DIRTY_RINGS		_IO(KVMIO, 0xc7)
 
Quoted 5 lines

include/uapi/linux/kvm.h struct kvm_pv_cmd { · line 1576

Quoted 1 line
include/uapi/linux/kvm.h +1 -0
+#define KVM_GET_PAGE_ENC_BITMAP	_IOW(KVMIO, 0xc6, struct kvm_page_enc_bitmap)

I see that kvm/next already defines ioctls numbered 0xc6 and 0xc7. Wouldn't these new ioctls (KVM_GET_PAGE_ENC_BITMAP, KVM_SET_PAGE_ENC_BITMAP) collide?

Yes, but they will be fixed for the next version of the patch-set i am going to post.

Thanks for your feedback.
Ashish

Quoted 8 lines

include/uapi/linux/kvm.h struct kvm_pv_cmd { · line 1580

Quoted 6 lines
include/uapi/linux/kvm.h +1 -0
+
  /* Secure Encrypted Virtualization command */
  enum sev_cmd_id {
  	/* Guest initialization commands */

-Dov

#6 lore raw

Re: [PATCH v2 1/9] KVM: x86: Add AMD SEV specific Hypercall3

In reply to Sean Christopherson
Quoted 38 lines · Sean Christopherson
Quoted 20 lines · Paolo Bonzini
Quoted 19 lines · Sean Christopherson
Quoted 10 lines · Ashish Kalra

From: Brijesh Singh <[email protected]>

KVM hypercall framework relies on alternative framework to patch the VMCALL -> VMMCALL on AMD platform. If a hypercall is made before apply_alternative() is called then it defaults to VMCALL. The approach works fine on non SEV guest. A VMCALL would causes #UD, and hypervisor will be able to decode the instruction and do the right things. But when SEV is active, guest memory is encrypted with guest key and hypervisor will not be able to decode the instruction bytes.

Quoted 2 lines

Add SEV specific hypercall3, it unconditionally uses VMMCALL. The hypercall will be used by the SEV guest to notify encrypted pages to the hypervisor.

What if we invert KVM_HYPERCALL and X86_FEATURE_VMMCALL to default to VMMCALL and opt into VMCALL? It's a synthetic feature flag either way, and I don't think there are any existing KVM hypercalls that happen before alternatives are patched, i.e. it'll be a nop for sane kernel builds.

Quoted 5 lines
Quoted 2 lines

I'm also skeptical that a KVM specific hypercall is the right approach for the encryption behavior, but I'll take that up in the patches later in the series.

Do you think that it's the guest that should "donate" memory for the bitmap instead?

No. Two things I'd like to explore:

1. Making the hypercall to announce/request private vs. shared common across hypervisors (KVM, Hyper-V, VMware, etc...) and technologies (SEV-* and TDX). I'm concerned that we'll end up with multiple hypercalls that do more or less the same thing, e.g. KVM+SEV, Hyper-V+SEV, TDX, etc... Maybe it's a pipe dream, but I'd like to at least explore options before shoving in KVM- only hypercalls.

Quoted 6 lines

2. Tracking shared memory via a list of ranges instead of a using bitmap to track all of guest memory. For most use cases, the vast majority of guest memory will be private, most ranges will be 2mb+, and conversions between private and shared will be uncommon events, i.e. the overhead to walk and split/merge list entries is hopefully not a big concern. I suspect a list would consume far less memory, hopefully without impacting performance.

For a fancier data structure, I'd suggest an interval tree. Linux already has an rbtree-based interval tree implementation, which would likely work, and would probably assuage any performance concerns.

Something like this would not be worth doing unless most of the shared pages were physically contiguous. A sample Ubuntu 20.04 VM on GCP had 60ish discontiguous shared regions. This is by no means a thorough search, but it's suggestive. If this is typical, then the bitmap would be far less efficient than most any interval-based data structure.

You'd have to allow userspace to upper bound the number of intervals (similar to the maximum bitmap size), to prevent host OOMs due to malicious guests. There's something nice about the guest donating memory for this, since that would eliminate the OOM risk.

#7 lore raw

Re: [PATCH v2 1/9] KVM: x86: Add AMD SEV specific Hypercall3

I don’t think that the bitmap by itself is really a performance bottleneck here.

Thanks,
Ashish

Quoted 62 lines · Steve Rutherford
Quoted 38 lines · Sean Christopherson
Quoted 22 lines · Paolo Bonzini
Quoted 19 lines · Sean Christopherson
Quoted 10 lines · Ashish Kalra

From: Brijesh Singh <[email protected]>

KVM hypercall framework relies on alternative framework to patch the VMCALL -> VMMCALL on AMD platform. If a hypercall is made before apply_alternative() is called then it defaults to VMCALL. The approach works fine on non SEV guest. A VMCALL would causes #UD, and hypervisor will be able to decode the instruction and do the right things. But when SEV is active, guest memory is encrypted with guest key and hypervisor will not be able to decode the instruction bytes.

Quoted 2 lines

Add SEV specific hypercall3, it unconditionally uses VMMCALL. The hypercall will be used by the SEV guest to notify encrypted pages to the hypervisor.

What if we invert KVM_HYPERCALL and X86_FEATURE_VMMCALL to default to VMMCALL and opt into VMCALL? It's a synthetic feature flag either way, and I don't think there are any existing KVM hypercalls that happen before alternatives are patched, i.e. it'll be a nop for sane kernel builds.

Quoted 5 lines
Quoted 2 lines

I'm also skeptical that a KVM specific hypercall is the right approach for the encryption behavior, but I'll take that up in the patches later in the series.

Do you think that it's the guest that should "donate" memory for the bitmap instead?

No. Two things I'd like to explore:

1. Making the hypercall to announce/request private vs. shared common across hypervisors (KVM, Hyper-V, VMware, etc...) and technologies (SEV-* and TDX). I'm concerned that we'll end up with multiple hypercalls that do more or less the same thing, e.g. KVM+SEV, Hyper-V+SEV, TDX, etc... Maybe it's a pipe dream, but I'd like to at least explore options before shoving in KVM- only hypercalls.

Quoted 6 lines

2. Tracking shared memory via a list of ranges instead of a using bitmap to track all of guest memory. For most use cases, the vast majority of guest memory will be private, most ranges will be 2mb+, and conversions between private and shared will be uncommon events, i.e. the overhead to walk and split/merge list entries is hopefully not a big concern. I suspect a list would consume far less memory, hopefully without impacting performance.

For a fancier data structure, I'd suggest an interval tree. Linux already has an rbtree-based interval tree implementation, which would likely work, and would probably assuage any performance concerns.

Something like this would not be worth doing unless most of the shared pages were physically contiguous. A sample Ubuntu 20.04 VM on GCP had 60ish discontiguous shared regions. This is by no means a thorough search, but it's suggestive. If this is typical, then the bitmap would be far less efficient than most any interval-based data structure.

You'd have to allow userspace to upper bound the number of intervals (similar to the maximum bitmap size), to prevent host OOMs due to malicious guests. There's something nice about the guest donating memory for this, since that would eliminate the OOM risk.

#8 lore raw

Re: [PATCH v2 1/9] KVM: x86: Add AMD SEV specific Hypercall3

In reply to Steve Rutherford
Quoted 51 lines · Steve Rutherford
Quoted 34 lines · Sean Christopherson
Quoted 19 lines · Paolo Bonzini
Quoted 18 lines · Sean Christopherson
Quoted 10 lines · Ashish Kalra

From: Brijesh Singh <[email protected]>

KVM hypercall framework relies on alternative framework to patch the VMCALL -> VMMCALL on AMD platform. If a hypercall is made before apply_alternative() is called then it defaults to VMCALL. The approach works fine on non SEV guest. A VMCALL would causes #UD, and hypervisor will be able to decode the instruction and do the right things. But when SEV is active, guest memory is encrypted with guest key and hypervisor will not be able to decode the instruction bytes.

Quoted 2 lines

Add SEV specific hypercall3, it unconditionally uses VMMCALL. The hypercall will be used by the SEV guest to notify encrypted pages to the hypervisor.

What if we invert KVM_HYPERCALL and X86_FEATURE_VMMCALL to default to VMMCALL and opt into VMCALL? It's a synthetic feature flag either way, and I don't think there are any existing KVM hypercalls that happen before alternatives are patched, i.e. it'll be a nop for sane kernel builds.

Quoted 4 lines
Quoted 2 lines

I'm also skeptical that a KVM specific hypercall is the right approach for the encryption behavior, but I'll take that up in the patches later in the series.

Do you think that it's the guest that should "donate" memory for the bitmap instead?

No. Two things I'd like to explore:

1. Making the hypercall to announce/request private vs. shared common across hypervisors (KVM, Hyper-V, VMware, etc...) and technologies (SEV-* and TDX). I'm concerned that we'll end up with multiple hypercalls that do more or less the same thing, e.g. KVM+SEV, Hyper-V+SEV, TDX, etc... Maybe it's a pipe dream, but I'd like to at least explore options before shoving in KVM- only hypercalls.

Quoted 6 lines

2. Tracking shared memory via a list of ranges instead of a using bitmap to track all of guest memory. For most use cases, the vast majority of guest memory will be private, most ranges will be 2mb+, and conversions between private and shared will be uncommon events, i.e. the overhead to walk and split/merge list entries is hopefully not a big concern. I suspect a list would consume far less memory, hopefully without impacting performance.

For a fancier data structure, I'd suggest an interval tree. Linux already has an rbtree-based interval tree implementation, which would likely work, and would probably assuage any performance concerns.

Something like this would not be worth doing unless most of the shared pages were physically contiguous. A sample Ubuntu 20.04 VM on GCP had 60ish discontiguous shared regions. This is by no means a thorough search, but it's suggestive. If this is typical, then the bitmap would be far less efficient than most any interval-based data structure.

Quoted 4 lines

You'd have to allow userspace to upper bound the number of intervals (similar to the maximum bitmap size), to prevent host OOMs due to malicious guests. There's something nice about the guest donating memory for this, since that would eliminate the OOM risk.

Tracking the list of ranges may not be bad idea, especially if we use the some kind of rbtree-based data structure to update the ranges. It will certainly be better than bitmap which grows based on the guest memory size and as you guys see in the practice most of the pages will be guest private. I am not sure if guest donating a memory will cover all the cases, e.g what if we do a memory hotplug (increase the guest ram from 2GB to 64GB), will donated memory range will be enough to store the metadata.

#9 lore raw

Re: [PATCH v2 1/9] KVM: x86: Add AMD SEV specific Hypercall3

Hello All,

Quoted 68 lines · Brijesh Singh
Quoted 51 lines · Steve Rutherford
Quoted 34 lines · Sean Christopherson
Quoted 19 lines · Paolo Bonzini
Quoted 18 lines · Sean Christopherson
Quoted 10 lines · Ashish Kalra

From: Brijesh Singh <[email protected]>

KVM hypercall framework relies on alternative framework to patch the VMCALL -> VMMCALL on AMD platform. If a hypercall is made before apply_alternative() is called then it defaults to VMCALL. The approach works fine on non SEV guest. A VMCALL would causes #UD, and hypervisor will be able to decode the instruction and do the right things. But when SEV is active, guest memory is encrypted with guest key and hypervisor will not be able to decode the instruction bytes.

Quoted 2 lines

Add SEV specific hypercall3, it unconditionally uses VMMCALL. The hypercall will be used by the SEV guest to notify encrypted pages to the hypervisor.

What if we invert KVM_HYPERCALL and X86_FEATURE_VMMCALL to default to VMMCALL and opt into VMCALL? It's a synthetic feature flag either way, and I don't think there are any existing KVM hypercalls that happen before alternatives are patched, i.e. it'll be a nop for sane kernel builds.

Quoted 4 lines
Quoted 2 lines

I'm also skeptical that a KVM specific hypercall is the right approach for the encryption behavior, but I'll take that up in the patches later in the series.

Do you think that it's the guest that should "donate" memory for the bitmap instead?

No. Two things I'd like to explore:

1. Making the hypercall to announce/request private vs. shared common across hypervisors (KVM, Hyper-V, VMware, etc...) and technologies (SEV-* and TDX). I'm concerned that we'll end up with multiple hypercalls that do more or less the same thing, e.g. KVM+SEV, Hyper-V+SEV, TDX, etc... Maybe it's a pipe dream, but I'd like to at least explore options before shoving in KVM- only hypercalls.

Quoted 6 lines

2. Tracking shared memory via a list of ranges instead of a using bitmap to track all of guest memory. For most use cases, the vast majority of guest memory will be private, most ranges will be 2mb+, and conversions between private and shared will be uncommon events, i.e. the overhead to walk and split/merge list entries is hopefully not a big concern. I suspect a list would consume far less memory, hopefully without impacting performance.

For a fancier data structure, I'd suggest an interval tree. Linux already has an rbtree-based interval tree implementation, which would likely work, and would probably assuage any performance concerns.

Something like this would not be worth doing unless most of the shared pages were physically contiguous. A sample Ubuntu 20.04 VM on GCP had 60ish discontiguous shared regions. This is by no means a thorough search, but it's suggestive. If this is typical, then the bitmap would be far less efficient than most any interval-based data structure.

Quoted 4 lines

You'd have to allow userspace to upper bound the number of intervals (similar to the maximum bitmap size), to prevent host OOMs due to malicious guests. There's something nice about the guest donating memory for this, since that would eliminate the OOM risk.

Tracking the list of ranges may not be bad idea, especially if we use the some kind of rbtree-based data structure to update the ranges. It will certainly be better than bitmap which grows based on the guest memory size and as you guys see in the practice most of the pages will be guest private. I am not sure if guest donating a memory will cover all the cases, e.g what if we do a memory hotplug (increase the guest ram from 2GB to 64GB), will donated memory range will be enough to store the metadata.

Quoted 1 line

.

With reference to internal discussions regarding the above, i am going to look into specific items as listed below :

1). "hypercall" related :
a). Explore the SEV-SNP page change request structure (included in GHCB), see if there is something common there than can be re-used for SEV/SEV-ES page encryption status hypercalls.
b). Explore if there is any common hypercall framework i can use in Linux/KVM.

2). related to the "backing" data structure - explore using a range-based list or something like rbtree-based interval tree data structure (as mentioned by Steve above) to replace the current bitmap based implementation.

Thanks,
Ashish

#10 lore raw

Re: [PATCH v2 1/9] KVM: x86: Add AMD SEV specific Hypercall3

Quoted 83 lines · Ashish Kalra

Hello All,

Quoted 68 lines · Brijesh Singh
Quoted 51 lines · Steve Rutherford
Quoted 34 lines · Sean Christopherson
Quoted 19 lines · Paolo Bonzini
Quoted 18 lines · Sean Christopherson
Quoted 10 lines · Ashish Kalra

From: Brijesh Singh <[email protected]>

KVM hypercall framework relies on alternative framework to patch the VMCALL -> VMMCALL on AMD platform. If a hypercall is made before apply_alternative() is called then it defaults to VMCALL. The approach works fine on non SEV guest. A VMCALL would causes #UD, and hypervisor will be able to decode the instruction and do the right things. But when SEV is active, guest memory is encrypted with guest key and hypervisor will not be able to decode the instruction bytes.

Quoted 2 lines

Add SEV specific hypercall3, it unconditionally uses VMMCALL. The hypercall will be used by the SEV guest to notify encrypted pages to the hypervisor.

What if we invert KVM_HYPERCALL and X86_FEATURE_VMMCALL to default to VMMCALL and opt into VMCALL? It's a synthetic feature flag either way, and I don't think there are any existing KVM hypercalls that happen before alternatives are patched, i.e. it'll be a nop for sane kernel builds.

Quoted 4 lines
Quoted 2 lines

I'm also skeptical that a KVM specific hypercall is the right approach for the encryption behavior, but I'll take that up in the patches later in the series.

Do you think that it's the guest that should "donate" memory for the bitmap instead?

No. Two things I'd like to explore:

1. Making the hypercall to announce/request private vs. shared common across hypervisors (KVM, Hyper-V, VMware, etc...) and technologies (SEV-* and TDX). I'm concerned that we'll end up with multiple hypercalls that do more or less the same thing, e.g. KVM+SEV, Hyper-V+SEV, TDX, etc... Maybe it's a pipe dream, but I'd like to at least explore options before shoving in KVM- only hypercalls.

Quoted 6 lines

2. Tracking shared memory via a list of ranges instead of a using bitmap to track all of guest memory. For most use cases, the vast majority of guest memory will be private, most ranges will be 2mb+, and conversions between private and shared will be uncommon events, i.e. the overhead to walk and split/merge list entries is hopefully not a big concern. I suspect a list would consume far less memory, hopefully without impacting performance.

For a fancier data structure, I'd suggest an interval tree. Linux already has an rbtree-based interval tree implementation, which would likely work, and would probably assuage any performance concerns.

Something like this would not be worth doing unless most of the shared pages were physically contiguous. A sample Ubuntu 20.04 VM on GCP had 60ish discontiguous shared regions. This is by no means a thorough search, but it's suggestive. If this is typical, then the bitmap would be far less efficient than most any interval-based data structure.

Quoted 4 lines

You'd have to allow userspace to upper bound the number of intervals (similar to the maximum bitmap size), to prevent host OOMs due to malicious guests. There's something nice about the guest donating memory for this, since that would eliminate the OOM risk.

Tracking the list of ranges may not be bad idea, especially if we use the some kind of rbtree-based data structure to update the ranges. It will certainly be better than bitmap which grows based on the guest memory size and as you guys see in the practice most of the pages will be guest private. I am not sure if guest donating a memory will cover all the cases, e.g what if we do a memory hotplug (increase the guest ram from 2GB to 64GB), will donated memory range will be enough to store the metadata.

Quoted 1 line

.

With reference to internal discussions regarding the above, i am going to look into specific items as listed below :

1). "hypercall" related :
a). Explore the SEV-SNP page change request structure (included in GHCB), see if there is something common there than can be re-used for SEV/SEV-ES page encryption status hypercalls.
b). Explore if there is any common hypercall framework i can use in Linux/KVM.

Quoted 6 lines

2). related to the "backing" data structure - explore using a range-based list or something like rbtree-based interval tree data structure (as mentioned by Steve above) to replace the current bitmap based implementation.

I do agree that a range-based list or an interval tree data structure is a really good "logical" fit for the guest page encryption status tracking.

We can only keep track of the guest unencrypted shared pages in the range(s) list (which will keep the data structure quite compact) and all the guest private/encrypted memory does not really need any tracking in the list, anything not in the list will be encrypted/private.

Also looking at a more "practical" use case, here is the current log of page encryption status hypercalls when booting a linux guest :

...

[   55.961331] page_enc_status_hc invoked, gpa = a0000, npages  = 60, enc = 0
[   55.961351] page_enc_status_hc invoked, gpa = 20000000, npages  = 90000, enc = 0
[   55.961374] page_enc_status_hc invoked, gpa = c0000000, npages  = 3c000, enc = 0
[   55.961384] page_enc_status_hc invoked, gpa = fc000000, npages  = 2c00, enc = 0
[   55.961391] page_enc_status_hc invoked, gpa = fec00000, npages  = 1, enc = 0
[   55.961397] page_enc_status_hc invoked, gpa = fec01000, npages  = ff, enc = 0
[   55.961402] page_enc_status_hc invoked, gpa = fed00000, npages  = 1, enc = 0
[   55.961406] page_enc_status_hc invoked, gpa = fed00400, npages  = 1c, enc = 0
[   55.961411] page_enc_status_hc invoked, gpa = fed1c000, npages  = 4, enc = 0
[   55.961417] page_enc_status_hc invoked, gpa = fed20000, npages  = e0, enc = 0
[   55.961426] page_enc_status_hc invoked, gpa = fee00000, npages  = 100, enc = 0
[   55.961803] page_enc_status_hc invoked, gpa = fef00000, npages  = f01100, enc = 0
[   56.146021] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.146050] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.146072] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.146086] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.146101] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.146111] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.146125] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.146137] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.146151] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.146160] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.146175] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.146187] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.146201] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.146211] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.146225] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.146237] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.146251] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.146260] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.146275] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.146286] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.146301] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.146310] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.146324] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.146336] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.146351] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.146360] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.146374] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.146386] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.146400] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.146409] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.146423] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.146435] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.146450] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.146459] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.146473] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.146485] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.146500] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.146509] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.146523] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.146535] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.146549] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.146558] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.146573] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.146584] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.146598] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.146607] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.146622] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.146633] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.146648] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.146657] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.146671] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.146683] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.146698] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.146707] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.146721] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.146733] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.146747] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.146775] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.146794] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.146809] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.146823] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.146833] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.146848] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.146860] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.146875] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.146884] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.146899] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.146912] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.146926] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.146936] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.146951] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.146963] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.146977] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.146988] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.147003] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.147015] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.147029] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.147039] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.147080] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.147092] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.147107] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.147117] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.147132] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.147144] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.147159] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.147168] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.147184] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.147196] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.147210] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.147220] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.147235] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.147247] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.147261] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.147271] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.147286] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.147298] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.147313] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.147324] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.147340] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.147352] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.147367] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.147376] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.147391] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.147404] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.147419] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.147429] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.147444] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.147455] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.147470] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.147479] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.147494] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.147507] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.147522] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.147532] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.147547] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.147559] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.147573] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.147584] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.147599] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.147611] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.147625] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.147635] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.147650] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.147662] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.147677] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.147686] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.147701] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.147713] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.147728] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.147737] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.147752] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.147773] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.147791] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.147804] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.147819] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.147831] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.147846] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.147855] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.147870] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.147882] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.147897] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.147909] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.147924] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.147936] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.147951] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.147961] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.147976] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.147988] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.148003] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.148012] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.148028] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.148040] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.148054] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.148064] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.148079] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.148091] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.148105] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.148115] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.148130] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.148142] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.148157] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.148166] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.148182] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.148194] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.148209] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.148218] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.148233] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.148245] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.148260] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.148269] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.148284] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.148296] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.148311] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.148321] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.148336] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.148349] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.148363] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.148373] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.148388] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.148400] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.148415] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.148424] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.148439] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.148451] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.148466] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.148476] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.148491] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.148503] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.148518] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.148527] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.148542] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.148554] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.148569] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.148579] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.148594] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.148606] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.148621] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.148630] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.148645] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.148657] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.148672] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.148682] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.148698] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.148710] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.148725] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.148735] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.148751] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.148771] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.148789] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.148801] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.148816] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.148828] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.148843] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.148853] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.148868] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.148880] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.148895] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.148904] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.148919] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.148931] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.148946] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.148955] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.148970] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.148982] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.148997] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.149006] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.149021] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.149034] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.149049] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.149058] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.149072] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.149084] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.149099] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.149109] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.149124] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.149137] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.149171] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.149181] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.149198] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.149210] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.149225] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.149235] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.149250] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.149262] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.149277] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.149287] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.149302] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.149314] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.149328] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.149338] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.149353] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.149364] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.149379] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.149388] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.149403] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.149416] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.149431] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.149441] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0
[   56.149456] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 1
[   56.149468] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.180730] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.180741] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.180768] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.180782] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.180797] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.180807] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.180822] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.180834] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.180849] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.180859] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.180874] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.180886] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.180901] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.180910] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.180925] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.180937] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.180952] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.180961] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.180976] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.180988] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.181003] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.181014] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.181029] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.181041] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.181056] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.181065] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.181080] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.181093] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.181108] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.181118] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.181132] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.181144] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.181159] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.181169] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.181186] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.181199] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.181215] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.181225] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.181241] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.181252] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.181267] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.181276] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.181291] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.181303] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.181318] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.181328] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.181343] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.181354] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.181369] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.181379] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.181394] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.181407] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.181422] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.181431] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.181446] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.181458] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.181474] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.181483] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.181498] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.181510] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.181524] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.181534] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.181548] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.181560] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.181575] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.181585] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.181600] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.181612] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.181627] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.181637] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.181651] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.181663] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.181678] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.181688] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.181703] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.181714] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.181729] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.181738] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.181761] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.181779] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.181799] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.181813] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.181828] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.181840] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.181855] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.181865] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.181879] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.181891] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.181906] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.181916] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.181930] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.181943] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.181958] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.181967] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.181982] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.181994] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.182009] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.182019] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.182034] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.182046] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.182061] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.182071] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.182086] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.182098] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.182113] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.182122] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.182137] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.182150] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.182165] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.182174] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.182189] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.182201] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.182216] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.182226] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.182241] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.182253] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.182472] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.182482] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.182497] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.182509] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.182524] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.182533] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.182548] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.182560] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.182575] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.182585] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.182600] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.182613] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.182628] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.182638] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.182652] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.182664] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.182679] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.182689] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.182704] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.182716] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.182731] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.182742] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.182764] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.182780] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.182797] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.182806] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.182821] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.182833] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.182848] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.182858] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.182873] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.182885] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.182900] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.182910] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.182925] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.182937] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.182952] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.182961] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.182976] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.182989] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.183005] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.183014] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.183029] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.183040] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.183055] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.183065] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.183080] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.183092] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.183106] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.183116] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.183131] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.183144] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.183159] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.183168] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.183183] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.183195] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.183209] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.183218] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.183233] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.183245] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.183259] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.183269] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.183284] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.183295] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.183310] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.183319] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.183335] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.183347] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.183362] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.183371] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.183386] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.183398] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.183413] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.183423] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.183437] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.183449] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.183464] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.183473] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.183488] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.183500] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.183515] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.183524] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.183539] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.183552] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.183567] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.183576] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.183591] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.183603] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.183619] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.183629] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.183643] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.183655] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.183670] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.183679] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.183694] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.183706] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.183721] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.183730] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.183745] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.183763] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.183779] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.183791] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.183809] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.183821] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.183836] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.183845] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.183859] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.183871] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.183886] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.183896] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.183911] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.183923] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.183938] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.183949] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.183966] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.183978] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.183994] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.184004] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.184019] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.184031] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.184046] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.184055] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.184070] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.184082] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.184097] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.184106] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.184121] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.184134] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.184149] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.184158] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.184173] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.184185] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.184403] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.184413] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.184428] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.184440] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1
[   56.197110] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.197120] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.197136] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.197148] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.197163] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.197172] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.197186] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.197198] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.197212] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.197221] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.197236] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.197247] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.197261] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.197270] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.197285] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.197296] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.197310] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.197319] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.197334] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.197346] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.197360] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.197370] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.197384] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.197395] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.197410] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.197419] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.197433] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.197444] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.197459] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.197468] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.197482] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.197493] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.197508] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.197517] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.197531] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.197543] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.197557] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.197566] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.197581] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.197592] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.197606] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.197615] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.197630] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.197641] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.197655] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.197664] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.197679] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.197690] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.197704] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.197713] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.197728] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.197740] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.197766] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.197778] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.197794] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.197806] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.197820] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.197830] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.197845] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.197857] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.197873] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.197884] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.197899] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.197911] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.197926] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.197935] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.197951] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.197963] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.197978] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.197988] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.198002] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.198014] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.198029] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.198039] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.198053] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.198065] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.198080] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.198090] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.198105] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.198117] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.198131] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.198141] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.198156] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.198168] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.198183] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.198192] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.198207] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.198219] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.198234] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.198244] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.198260] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.198273] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.198288] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.198298] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.198312] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.198326] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.198340] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.198350] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.198365] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.198378] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.198393] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.198402] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.198417] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.198429] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.198444] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.198453] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.198468] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.198480] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.198494] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.198504] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.198518] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.198530] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.198545] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.198554] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.198569] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.198583] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.198598] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.198607] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.198622] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.198634] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.198649] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.198659] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.198674] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.198686] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.198701] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.198710] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.198725] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.198738] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.198762] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.198778] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.198795] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.198807] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.198822] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.198831] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.198846] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.198858] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.198873] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.198886] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.198902] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.198919] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.198934] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.198943] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.198958] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.198970] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.198985] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.198995] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.199010] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.199022] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.199037] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.199047] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.199061] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.199073] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.199088] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.199098] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.199113] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.199125] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.199139] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.199153] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.199177] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.199189] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.199205] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.199216] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.199232] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.199244] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.199259] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.199268] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.199282] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.199294] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.199309] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.199319] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.199333] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.199345] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.199360] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.199369] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.199385] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.199396] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.199411] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.199421] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.199436] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.199448] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.199463] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.199472] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.199487] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.199499] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.199514] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.199523] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.199539] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.199551] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.199566] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.199576] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.199591] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.199603] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.199617] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.199627] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.199643] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.199655] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.199879] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.199889] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.199904] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.199916] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1
[   56.222679] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.222691] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.222707] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.222720] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.222737] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.222746] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.222770] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.222783] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.222797] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.222807] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.222823] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.222834] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.222849] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.222858] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.222872] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.222884] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.222899] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.222908] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.222923] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.222934] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.222949] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.222958] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.222972] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.222984] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.222998] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.223008] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.223022] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.223034] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.223049] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.223060] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.223074] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.223087] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.223101] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.223111] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.223126] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.223138] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.223152] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.223162] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.223177] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.223189] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.223203] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.223213] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.223227] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.223239] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.223253] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.223263] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.223278] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.223289] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.223304] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.223313] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.223329] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.223341] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.223355] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.223365] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.223379] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.223391] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.223406] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.223415] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.223430] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.223444] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.223458] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.223468] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.223483] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.223494] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.223509] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.223518] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.223533] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.223546] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.223562] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.223571] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.223586] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.223598] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.223613] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.223623] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.223639] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.223651] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.223666] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.223675] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.223692] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.223704] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.223719] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.223728] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.223743] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.223769] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.223791] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.223805] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.223821] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.223833] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.223848] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.223857] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.223872] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.223884] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.223899] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.223909] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.223924] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.223936] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.223951] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.223960] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.223976] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.223988] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.224002] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.224013] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.224028] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.224040] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.224055] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.224064] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.224079] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.224091] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.224106] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.224115] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.224130] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.224142] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.224157] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.224166] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.224181] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.224193] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.224208] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.224218] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.224233] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.224244] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.224259] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.224269] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.224284] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.224295] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.224310] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.224320] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.224335] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.224347] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.224361] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.224370] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.224385] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.224398] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.224412] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.224422] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.224437] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.224449] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.224463] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.224473] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.224488] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.224500] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.224514] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.224524] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.224539] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.224550] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.224565] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.224574] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.224590] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.224602] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.224617] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.224626] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.224641] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.224652] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.224667] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.224677] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.224693] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.224704] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.224719] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.224729] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.224743] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.224763] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.224780] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.224793] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.224813] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.224826] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.224840] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.224850] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.224865] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.224877] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.224891] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.224901] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.224916] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.224927] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.224942] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.224952] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.224967] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.224979] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.224993] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.225003] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.225019] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.225032] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.225047] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.225056] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.225071] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.225083] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.225097] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.225107] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.225122] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.225134] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.225148] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.225158] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.225172] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.225184] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.225198] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.225208] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.225223] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.225235] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.225250] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.225260] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.225274] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.225286] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.225301] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.225310] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.225325] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.225337] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.225352] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.225361] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.225376] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.225388] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.225402] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.225412] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.225429] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.225440] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.225456] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.225466] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.225481] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.225492] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.225507] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.225517] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.225532] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.225544] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.225558] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.225568] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.225583] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.225594] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.225610] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.225619] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.225634] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.225646] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.225661] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.225670] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.225685] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.225697] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.225711] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.225721] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.225736] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.225748] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.225775] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.225786] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.225804] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.225816] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.225831] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.225840] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.225856] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.225868] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.225884] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.225899] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.225914] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.225928] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.225946] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.225955] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.225970] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.225982] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.225997] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.226006] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.226021] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.226033] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.226048] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.226057] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.226073] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.226084] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1
[   56.313747] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.313771] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.313789] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.313803] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
[   56.313821] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.313831] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.313846] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.313858] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
[   56.313873] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.313884] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.313898] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.313910] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
[   56.313925] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.313937] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.313952] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.313964] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
[   56.313979] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.313989] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.314003] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.314015] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
[   56.314031] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.314041] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.314056] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.314068] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
[   56.314083] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.314093] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.314108] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.314120] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
[   56.314135] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.314145] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.314160] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.314172] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
[   56.314187] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.314197] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.314213] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.314225] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
[   56.314241] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.314252] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.314267] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.314279] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
[   56.314294] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.314304] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.314319] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.314331] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
[   56.314347] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.314357] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.314371] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.314383] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
[   56.314399] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.314409] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.314424] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.314436] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
[   56.314452] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.314462] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.314477] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.314489] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
[   56.314504] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.314514] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.314529] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.314541] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
[   56.314556] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.314566] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.314581] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.314593] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
[   56.314608] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.314618] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.314633] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.314645] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
[   56.314661] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.314671] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.314686] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.314698] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
[   56.314713] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.314723] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.314738] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.314750] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
[   56.314780] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.314793] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.314814] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.314826] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
[   56.314841] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.314851] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.314867] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.314879] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
[   56.314894] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.314904] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.314919] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.314932] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
[   56.314947] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.314957] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.314972] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.314984] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
[   56.314999] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.315009] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.315024] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.315036] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
[   56.315051] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.315061] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.315076] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.315089] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
[   56.315104] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.315116] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.315131] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.315143] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
[   56.315158] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.315168] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.315182] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.315194] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
[   56.315209] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.315220] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.315234] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.315246] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
[   56.315262] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.315272] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.315287] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.315299] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
[   56.315315] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.315324] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.315339] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.315351] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
[   56.315366] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.315376] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.315391] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.315403] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
[   56.315419] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.315429] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.315444] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.315456] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
[   56.315471] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.315481] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.315496] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.315509] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
[   56.315723] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.315735] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.315751] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.315770] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
[   56.337896] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.337908] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.337923] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.337936] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.337952] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.337962] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.337977] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.337988] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.338004] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.338013] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.338028] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.338039] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.338054] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.338064] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.338079] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.338090] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.338106] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.338115] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.338130] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.338141] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.338156] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.338166] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.338181] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.338192] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.338207] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.338217] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.338232] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.338243] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.338258] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.338268] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.338283] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.338294] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.338309] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.338319] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.338334] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.338346] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.338361] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.338371] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.338385] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.338397] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.338412] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.338421] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.338436] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.338448] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.338463] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.338473] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.338490] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.338504] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.338519] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.338528] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.338543] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.338555] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.338570] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.338580] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.338595] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.338606] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.338621] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.338631] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.338646] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.338657] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.338672] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.338682] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.338696] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.338708] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.338723] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.338733] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.338747] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.338771] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.338793] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.338806] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.338825] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.338837] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.338852] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.338862] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.338877] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.338889] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.338905] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.338917] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.338932] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.338944] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.338960] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.338970] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.338986] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.338998] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.339014] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.339024] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.339039] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.339051] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.339067] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.339078] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.339093] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.339105] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.339121] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.339131] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.339149] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.339180] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.339196] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.339208] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.339224] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.339236] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.339252] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.339262] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.339277] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.339289] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.339305] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.339315] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.339330] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.339342] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.339358] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.339368] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.339383] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.339395] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.339410] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.339421] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.339436] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.339449] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.339464] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.339475] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.339491] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.339503] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.339518] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.339528] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.339543] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.339555] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.339571] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.339581] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.339596] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.339608] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.339626] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.339637] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.339653] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.339665] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.339681] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.339691] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.339706] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.339718] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.339734] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.339744] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.339771] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.339788] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.339804] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.339814] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.339829] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.339841] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.339858] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.339867] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.339883] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.339895] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.339911] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.339923] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.339941] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.339953] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.339969] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.339979] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.339994] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.340006] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.340021] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.340031] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.340046] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.340059] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.340074] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.340086] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.340101] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.340113] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.340129] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.340140] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.340155] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.340167] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.340182] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.340192] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.340208] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.340220] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.340235] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.340245] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.340260] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.340272] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.340288] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.340298] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.340313] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.340326] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.340341] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.340351] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.340366] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.340378] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.340394] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.340404] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.340418] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.340430] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.340445] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.340455] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.340472] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.340486] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.340503] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.340513] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.340528] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.340541] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.340770] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 0
[   56.340786] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 0
[   56.340801] page_enc_status_hc invoked, gpa = 1e5e4000, npages  = 1, enc = 1
[   56.340814] page_enc_status_hc invoked, gpa = 1e5e2000, npages  = 1, enc = 1
[   56.457487] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 0
[   56.457504] page_enc_status_hc invoked, gpa = 1e4e7000, npages  = 1, enc = 0
[   56.457524] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 1
[   56.457539] page_enc_status_hc invoked, gpa = 1e4e7000, npages  = 1, enc = 1
[   56.457747] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 0
[   56.457774] page_enc_status_hc invoked, gpa = 1e4e7000, npages  = 1, enc = 0
[   56.457792] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 1
[   56.457806] page_enc_status_hc invoked, gpa = 1e4e7000, npages  = 1, enc = 1
[   56.458514] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 0
[   56.458528] page_enc_status_hc invoked, gpa = 1e4d4000, npages  = 4, enc = 0
[   56.458546] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 1
[   56.458574] page_enc_status_hc invoked, gpa = 1e4d4000, npages  = 4, enc = 1
[   56.458807] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 0
[   56.459276] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 0
[   56.459428] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 1
[   56.460037] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 1
[   56.460216] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 0
[   56.460299] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 0
[   56.460448] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 1
[   56.461050] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 1
[   56.461183] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 0
[   56.461265] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 0
[   56.461419] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 1
[   56.462029] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 1
[   56.462165] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 0
[   56.462247] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 0
[   56.462406] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 1
[   56.463006] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 1
[   56.463143] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 0
[   56.463225] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 0
[   56.463388] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 1
[   56.463984] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 1
[   56.464125] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 0
[   56.464207] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 0
[   56.464368] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 1
[   56.464964] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 1
[   56.465102] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 0
[   56.465185] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 0
[   56.465349] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 1
[   56.465949] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 1
[   56.466088] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 0
[   56.466170] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 0
[   56.466336] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 1
[   56.466935] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 1
[   56.467073] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 0
[   56.467155] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 0
[   56.467315] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 1
[   56.467917] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 1
[   56.468056] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 0
[   56.468349] page_enc_status_hc invoked, gpa = 1e433000, npages  = 95, enc = 0
[   56.468462] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 1
[   56.468823] page_enc_status_hc invoked, gpa = 1e433000, npages  = 95, enc = 1
[   56.469215] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 0
[   56.469227] page_enc_status_hc invoked, gpa = 1e4e7000, npages  = 1, enc = 0
[   56.469242] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 1
[   56.469255] page_enc_status_hc invoked, gpa = 1e4e7000, npages  = 1, enc = 1
[   56.469958] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 0
[   56.470384] page_enc_status_hc invoked, gpa = 1d402000, npages  = 100, enc = 0
[   56.479326] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 1
[   56.479926] page_enc_status_hc invoked, gpa = 1d402000, npages  = 100, enc = 1
[   56.480118] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 0
[   56.480201] page_enc_status_hc invoked, gpa = 1d402000, npages  = 100, enc = 0
[   56.487883] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 1
[   56.488475] page_enc_status_hc invoked, gpa = 1d402000, npages  = 100, enc = 1
[   56.488612] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 0
[   56.488695] page_enc_status_hc invoked, gpa = 1d402000, npages  = 100, enc = 0
[   56.496240] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 1
[   56.496545] page_enc_status_hc invoked, gpa = 1d402000, npages  = 100, enc = 1
[   56.496679] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 0
[   56.496715] page_enc_status_hc invoked, gpa = 1e463000, npages  = 65, enc = 0
[   56.499440] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 1
[   56.499500] page_enc_status_hc invoked, gpa = 1e463000, npages  = 65, enc = 1
[   56.654279] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.654292] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.654308] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.654322] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.654338] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.654348] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.654363] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.654376] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.654392] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.654402] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.654416] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.654429] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.654445] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.654455] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.654469] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.654482] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.654498] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.654508] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.654523] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.654536] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.654552] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.654562] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.654577] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.654589] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.654605] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.654615] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.654630] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.654642] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.654658] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.654669] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.654683] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.654696] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.654711] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.654722] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.654736] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.654750] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.654777] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.654793] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.654808] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.654822] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.654838] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.654849] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.654864] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.654877] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.654893] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.654904] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.654919] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.654932] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.654948] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.654960] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.654976] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.654989] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.655006] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.655016] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.655031] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.655045] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.655062] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.655072] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.655087] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.655100] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.655116] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.655128] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.655143] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.655156] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.655172] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.655183] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.655198] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.655212] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.655228] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.655239] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.655253] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.655267] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.655283] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.655294] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.655308] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.655322] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.655338] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.655349] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.655364] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.655377] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.655394] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.655404] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.655419] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.655433] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.655449] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.655460] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.655475] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.655488] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.655505] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.655515] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.655530] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.655544] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.655560] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.655571] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.655585] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.655599] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.655615] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.655626] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.655641] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.655654] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.655671] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.655682] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.655697] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.655711] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.655727] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.655738] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.655753] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.655780] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.655802] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.655812] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.655828] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.655841] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.655857] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.655868] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.655883] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.655897] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.655914] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.655925] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.655940] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.655953] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.655969] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.655980] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.655994] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.656008] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.656024] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.656035] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.656049] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.656063] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.656079] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.656090] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.656105] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.656119] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.656136] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.656146] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.656161] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.656174] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.656191] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.656201] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.656216] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.656229] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.656247] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.656258] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.656273] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.656286] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.656303] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.656313] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.656330] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.656343] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.656360] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.656370] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.656385] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.656398] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.656415] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.656425] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.656440] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.656453] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.656470] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.656481] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.656495] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.656509] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.656525] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.656536] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.656551] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.656565] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.656581] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.656592] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.656607] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.656620] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.656637] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.656647] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.656662] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.656675] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.656692] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.656702] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.656717] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.656731] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.656747] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.656769] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.656787] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.656806] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.656823] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.656834] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.656848] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.656861] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.656878] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.656889] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.656906] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.656919] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.656936] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.656947] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.656962] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.656975] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.656991] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.657002] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.657017] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.657031] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.657047] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.657058] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.657073] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.657089] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.657110] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.657121] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.657136] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.657149] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.657166] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.657176] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.657191] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.657204] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.657221] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.657231] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.657246] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.657260] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.657277] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.657287] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.657303] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.657318] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.657334] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.657345] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.657360] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.657373] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.657389] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.657400] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.657415] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.657428] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.657444] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.657455] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.657470] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.657484] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.657501] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.657511] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.657526] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.657539] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.657556] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.657566] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.657581] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.657594] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.657611] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.657622] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.657637] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.657650] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.657666] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.657676] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.657691] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.657705] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.657722] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.657732] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.657747] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.657768] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.657788] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.657798] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.657813] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.657826] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.657842] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.657853] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.657868] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.657881] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.657898] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.657908] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.657924] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.657937] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.658157] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.658168] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.658184] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.658197] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.658213] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.658224] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.658239] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.658254] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.658271] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.658283] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.658298] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.658311] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.658328] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.658338] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.658353] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.658366] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.658383] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.658393] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.658408] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.658422] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.658438] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.658449] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.658464] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.658477] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.658495] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.658506] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.658521] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.658534] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.658551] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.658561] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.658576] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.658589] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.658606] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.658616] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.658632] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.658645] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.658662] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.658672] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.658687] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.658700] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.658717] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.658727] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.658742] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.658768] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.658789] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.658800] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.658815] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.658828] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.658845] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.658856] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.658873] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.658887] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.658904] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.658914] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.658929] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.658942] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.658959] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.658969] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.658984] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.658997] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.659014] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.659024] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.659039] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.659052] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.659069] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.659080] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.659095] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.659108] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.659125] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.659136] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.659151] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.659177] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.659194] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.659206] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.659221] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.659234] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.659251] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.659261] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.659276] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.659289] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.659307] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.659317] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.659332] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.659346] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.659362] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.659373] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.659388] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.659401] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.659417] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.659428] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.659443] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.659456] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.659472] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.659483] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.659498] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.659511] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.659527] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.659538] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.659553] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.659567] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.659583] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.659594] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.659609] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.659622] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.659638] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.659649] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.659664] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.659678] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.659695] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.659706] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.659721] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.659734] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.659751] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.659769] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.659792] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.659810] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.659827] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.659838] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.659852] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.659866] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.659884] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.659897] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.659912] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.659925] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.659942] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.659952] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.659967] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.659980] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.659998] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.660010] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.660026] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.660039] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.660056] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.660066] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.660081] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.660095] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.660112] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.660122] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.660137] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.660150] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.660167] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.660177] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.660192] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.660205] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.660222] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.660232] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.660247] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.660261] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.660277] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.660288] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.660303] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.660316] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.660333] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.660343] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.660358] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.660371] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.660388] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.660399] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.660414] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.660427] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.660444] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.660454] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.660471] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.660484] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.660501] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.660511] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.660526] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.660539] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.660555] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.660566] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.660581] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.660594] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.660611] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.660621] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.660636] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.660649] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.660665] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.660676] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.660691] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.660705] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.660721] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.660733] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.660748] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.660770] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.660792] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.660806] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.660821] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.660834] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.660852] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.660862] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.660877] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.660890] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.660907] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.660917] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.660933] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.660946] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.660963] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.660973] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.660988] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.661001] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.661017] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.661028] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.661043] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.661056] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.661072] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.661083] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.661100] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.661115] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.661132] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.661142] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.661158] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.661171] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.661188] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.661198] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.661213] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.661226] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.661243] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.661254] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.661270] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.661283] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.661301] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.661312] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.661326] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.661340] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.661356] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.661367] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.661382] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.661395] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.661412] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.661423] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.661437] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.661451] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.661467] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.661478] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.661492] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.661506] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.661522] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.661533] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.661547] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.661561] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.661577] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.661589] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.661604] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.661618] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.661635] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.661645] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.661660] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.661673] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.661690] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.661700] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.661715] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.661728] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.661744] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.661769] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.661790] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.661807] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.661824] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 0
[   56.661835] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 0
[   56.661850] page_enc_status_hc invoked, gpa = 1e49c000, npages  = 1, enc = 1
[   56.661863] page_enc_status_hc invoked, gpa = 1e4aa000, npages  = 1, enc = 1
[   56.841131] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.841145] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.841162] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.841175] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.841195] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.841205] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.841220] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.841233] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.841249] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.841259] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.841274] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.841286] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.841303] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.841313] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.841327] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.841340] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.841356] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.841366] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.841381] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.841393] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.841410] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.841420] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.841434] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.841447] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.841463] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.841473] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.841488] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.841500] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.841517] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.841527] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.841541] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.841554] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.841570] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.841580] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.841595] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.841608] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.841624] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.841634] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.841649] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.841661] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.841677] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.841688] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.841702] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.841714] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.841731] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.841741] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.841774] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.841793] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.841812] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.841822] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.841837] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.841851] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.841869] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.841883] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.841903] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.841916] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.841932] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.841943] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.841958] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.841972] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.841989] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.842000] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.842015] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.842028] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.842046] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.842057] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.842072] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.842085] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.842102] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.842113] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.842128] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.842141] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.842158] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.842170] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.842185] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.842198] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.842215] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.842226] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.842240] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.842253] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.842270] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.842281] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.842296] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.842309] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.842326] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.842337] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.842352] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.842365] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.842382] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.842392] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.842407] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.842420] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.842437] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.842447] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.842462] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.842475] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.842492] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.842503] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.842518] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.842531] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.842548] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.842559] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.842573] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.842586] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.842603] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.842614] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.842629] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.842643] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.842660] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.842670] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.842685] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.842698] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.842715] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.842725] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.842740] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.842761] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.842779] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.842795] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.842811] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.842824] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.842841] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.842852] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.842867] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.842879] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.842896] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.842908] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.842923] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.842936] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.842953] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.842964] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.842978] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.842991] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.843008] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.843019] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.843033] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.843046] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.843063] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.843074] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.843089] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.843101] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.843118] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.843129] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.843145] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.843158] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.843175] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.843187] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.843202] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.843215] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.843232] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.843242] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.843257] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.843269] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.843286] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.843297] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.843312] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.843324] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.843341] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.843352] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.843367] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.843380] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.843396] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.843408] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.843423] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.843436] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.843453] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.843464] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.843479] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.843492] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.843508] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.843519] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.843533] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.843546] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.843563] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.843574] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.843589] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.843601] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.843618] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.843629] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.843644] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.843658] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.843674] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.843685] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.843700] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.843713] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.843730] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.843740] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.843766] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.843781] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.843801] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.843813] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.843828] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.843841] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.843857] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.843869] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.843884] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.843897] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.843914] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.843924] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.843939] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.843952] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.843969] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.843980] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.843994] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.844007] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.844024] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.844035] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.844050] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.844062] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.844079] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 0
[   56.844090] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.844105] page_enc_status_hc invoked, gpa = 1d4dc000, npages  = 1, enc = 1
[   56.844118] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.851801] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.851814] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.851832] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.851846] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.855644] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.855656] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.855672] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.855686] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.855702] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.855712] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.855727] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.855740] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.855767] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.855783] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.855799] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.855812] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.855829] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.855840] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.855855] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.855868] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.855884] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.855895] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.855910] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.855923] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.855940] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.855951] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.855966] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.855979] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.855996] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.856006] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.856021] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.856035] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.856051] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.856062] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.856077] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.856091] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.856109] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.856119] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.856135] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.856148] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.856165] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.856175] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.856190] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.856203] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.856220] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.856231] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.856245] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.856259] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.856275] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.856286] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.856301] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.856314] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.856331] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.856341] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.856357] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.856370] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.856387] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.856397] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.856414] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.856427] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.856443] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.856454] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.856469] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.856483] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.856499] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.856510] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.856525] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.856538] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.856555] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.856565] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.856580] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.856594] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.856610] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.856621] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.856636] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.856650] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.856666] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.856676] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.856691] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.856704] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.856721] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.856732] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.856746] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.856767] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.856787] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.856801] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.856819] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.856833] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.856849] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.856860] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.856875] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.856888] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.856904] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.856915] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.856930] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.856943] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.856960] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.856970] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.856985] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.856998] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.857015] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.857025] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.857041] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.857054] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.857071] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.857081] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.857096] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.857109] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.857126] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.857136] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.857151] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.857166] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.857183] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.857193] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.857208] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.857222] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.857238] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.857249] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.857266] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.857279] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.857296] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.857307] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.857322] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.857335] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.857351] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.857362] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.857377] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.857390] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.857407] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.857417] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.857432] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.857446] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.857463] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.857473] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.857489] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.857502] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.857519] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.857530] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.857544] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.857558] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.857574] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.857586] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.857601] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.857614] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.857631] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.857641] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.857657] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.857670] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.857687] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.857697] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.857712] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.857726] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.857743] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.857766] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.857785] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.857801] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.857817] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.857828] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.857843] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.857856] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.857873] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.857883] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.857898] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.857911] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.857928] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.857939] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.857954] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.857967] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.857984] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.857996] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.858010] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.858024] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.858040] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.858051] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.858065] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.858078] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.858095] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.858106] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.858120] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.858134] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.858150] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.858161] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.858176] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.858189] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.858206] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.858217] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.858231] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.858244] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.858261] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.858272] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.858287] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.858300] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.858316] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.858327] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.858342] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.858355] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.858372] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.858382] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.858397] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.858411] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.858633] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.858644] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.858660] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.858673] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.858898] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.858909] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.858924] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.858937] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.858954] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.858965] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.858979] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.858993] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.859009] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.859021] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.859036] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.859049] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.859066] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.859076] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.859091] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.859104] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.859121] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.859132] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.859147] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.859165] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.859225] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.859235] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.859250] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.859264] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.859280] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.859291] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.859306] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.859319] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.859337] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.859347] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.859362] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.859375] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.859392] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.859402] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.859418] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.859431] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.859448] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.859458] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.859473] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.859487] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.859503] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.859514] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.859529] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.859542] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.859558] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.859569] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.859584] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.859597] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.859614] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.859624] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.859639] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.859652] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.859887] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.859898] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.859914] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.859928] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.860148] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.860159] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.860173] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.860186] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.860203] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.860213] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.860227] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.860240] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.860256] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.860266] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.860281] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.860293] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.860309] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.860320] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.860334] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.860347] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.860363] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.860373] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.860388] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.860401] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.860417] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.860427] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.860441] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.860454] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.860470] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.860480] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.860495] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.860507] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.860524] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.860534] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.860548] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.860561] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.860577] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.860587] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.860602] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.860615] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.860631] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.860641] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.860656] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.860668] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.860684] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.860695] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.860709] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.860722] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.860738] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.860748] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.860774] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.860788] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.860805] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.860815] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.860831] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.860844] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.860861] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.860871] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.860886] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.860900] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.860917] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.860927] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.860942] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.860956] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.860972] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.860983] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.860998] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.861011] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.861028] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.861038] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 0
[   56.861053] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.861067] page_enc_status_hc invoked, gpa = 1d4db000, npages  = 1, enc = 1
[   56.861288] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.861360] page_enc_status_hc invoked, gpa = 1d405000, npages  = 20, enc = 0
[   56.861385] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.861475] page_enc_status_hc invoked, gpa = 1d405000, npages  = 20, enc = 1
[   56.866913] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.866925] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.866941] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.866954] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.866970] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.866980] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.866995] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.867008] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.867023] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.867034] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.867048] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.867061] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.867077] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.867087] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.867102] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.867114] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.867130] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.867140] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.867155] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.867168] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.867184] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.867195] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.867209] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.867222] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.867238] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.867248] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.867263] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.867275] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.867291] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.867301] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.867316] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.867328] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.867344] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.867355] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.867369] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.867382] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.867398] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.867409] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.867423] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.867436] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.867452] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.867462] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.867476] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.867489] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.867505] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.867515] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.867530] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.867542] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.867558] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.867568] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.867583] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.867596] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.867612] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.867622] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.867637] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.867649] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.867665] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.867676] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.867690] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.867703] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.867718] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.867729] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.867743] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.867772] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.867795] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.867810] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.867825] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.867839] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.867855] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.867867] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.867882] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.867896] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.867914] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.867925] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.867940] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.867953] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.867969] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.867981] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.867996] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.868010] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.868027] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.868038] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.868053] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.868066] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.868083] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.868093] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.868108] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.868121] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.868138] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.868149] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.868164] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.868176] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.868192] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.868203] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.868218] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.868231] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.868248] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.868258] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.868273] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.868286] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.868303] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.868314] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.868328] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.868342] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.868358] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.868369] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.868384] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.868396] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.868412] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.868422] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.868437] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.868450] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.868466] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.868477] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.868492] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.868505] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.868523] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.868534] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.868549] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.868562] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.868578] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.868589] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.868604] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.868617] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.868633] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.868644] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.868659] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.868672] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.868688] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.868699] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.868714] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.868727] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.868745] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.868766] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.868784] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.868799] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.868815] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.868826] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.868843] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.868856] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.868873] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.868884] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.868898] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.868911] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.868928] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.868938] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.868954] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.868967] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.868983] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.868994] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.869009] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.869022] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.869039] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.869049] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.869064] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.869077] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.869093] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.869104] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.869119] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.869132] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.869149] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.869164] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.869189] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.869203] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.869219] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.869230] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.869245] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.869258] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.869274] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.869285] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.869300] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.869313] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.869330] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.869341] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.869355] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.869368] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.869385] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.869396] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.869411] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.869424] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.869441] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.869453] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.869468] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.869481] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.869497] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.869508] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.869523] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.869536] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.869552] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.869563] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.869578] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.869591] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.869607] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.869618] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.869633] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.869647] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.869874] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.869886] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.869901] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.869914] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.873160] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.873172] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.873188] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.873201] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.873217] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.873227] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.873242] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.873254] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.873270] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.873281] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.873295] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.873308] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.873324] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.873334] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.873348] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.873361] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.873377] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.873387] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.873402] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.873415] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.873431] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.873441] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.873456] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.873468] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.873484] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.873495] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.873509] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.873522] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.873538] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.873548] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.873562] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.873575] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.873591] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.873601] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.873616] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.873629] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.873645] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.873655] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.873670] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.873682] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.873698] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.873708] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.873723] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.873735] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.873751] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.873780] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.873798] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.873811] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.873828] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.873839] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.873854] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.873868] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.873884] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.873895] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.873910] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.873923] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.873940] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.873951] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.873965] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.873979] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.873995] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.874006] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.874023] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.874036] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.874053] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.874065] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.874080] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.874094] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.874110] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.874121] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.874136] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.874150] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.874166] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.874177] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.874192] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.874206] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.874223] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.874233] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.874248] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.874261] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.874279] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.874289] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.874304] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.874318] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.874334] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.874345] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.874360] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.874373] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.874389] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.874400] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.874415] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.874428] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.874445] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.874456] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.874471] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.874484] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.874501] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.874512] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.874527] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.874540] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.874557] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.874568] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.874583] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.874596] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.874613] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.874623] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.874639] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.874652] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.874668] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.874679] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.874694] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.874707] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.874723] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.874734] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.874750] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.874773] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.874792] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.874802] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.874817] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.874830] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.874847] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.874858] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.874873] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.874886] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.874904] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.874920] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.874935] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.874948] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.874965] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.874975] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.874991] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.875004] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.875020] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.875031] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.875046] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.875059] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.875075] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.875086] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.875101] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.875114] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.875130] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.875141] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.875156] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.875169] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.875186] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.875198] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.875213] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.875226] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.875243] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.875254] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.875269] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.875282] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.875298] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.875309] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.875324] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.875337] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.875353] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.875364] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.875379] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.875392] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.875409] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.875420] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.875435] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.875448] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.875465] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.875476] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.875491] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.875504] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.875520] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.875532] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.875547] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.875560] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.875576] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.875587] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.875602] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.875615] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.875633] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.875643] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.875658] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.875672] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.875688] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.875699] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.875714] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.875727] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.875743] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.875764] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.875781] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.875799] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.875816] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.875828] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.875843] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.875856] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.875872] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.875883] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.875900] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.875914] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   56.876134] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 0
[   56.876145] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 0
[   56.876161] page_enc_status_hc invoked, gpa = 1d4ab000, npages  = 1, enc = 1
[   56.876174] page_enc_status_hc invoked, gpa = 1e400000, npages  = 1, enc = 1
[   57.052642] page_enc_status_hc invoked, gpa = 1d49b000, npages  = 1, enc = 0
[   57.052656] page_enc_status_hc invoked, gpa = 1d4aa000, npages  = 1, enc = 0
[   57.052673] page_enc_status_hc invoked, gpa = 1d49b000, npages  = 1, enc = 1
[   57.052687] page_enc_status_hc invoked, gpa = 1d4aa000, npages  = 1, enc = 1
[   57.055227] page_enc_status_hc invoked, gpa = 1d49a000, npages  = 2, enc = 0
[   57.330735] page_enc_status_hc invoked, gpa = 1d418000, npages  = 1, enc = 0
[   57.330749] page_enc_status_hc invoked, gpa = 1d416000, npages  = 1, enc = 0
[   57.330786] page_enc_status_hc invoked, gpa = 1d418000, npages  = 1, enc = 1
[   57.330801] page_enc_status_hc invoked, gpa = 1d416000, npages  = 1, enc = 1
[   57.331019] page_enc_status_hc invoked, gpa = 1d418000, npages  = 1, enc = 0
[   57.331030] page_enc_status_hc invoked, gpa = 1d416000, npages  = 1, enc = 0
[   57.331046] page_enc_status_hc invoked, gpa = 1d418000, npages  = 1, enc = 1
[   57.331060] page_enc_status_hc invoked, gpa = 1d416000, npages  = 1, enc = 1
[   57.331265] page_enc_status_hc invoked, gpa = 1d418000, npages  = 1, enc = 0
[   57.331275] page_enc_status_hc invoked, gpa = 1d416000, npages  = 1, enc = 0
[   57.331290] page_enc_status_hc invoked, gpa = 1d418000, npages  = 1, enc = 1
[   57.331304] page_enc_status_hc invoked, gpa = 1d416000, npages  = 1, enc = 1
[   57.337557] page_enc_status_hc invoked, gpa = 1d419000, npages  = 1, enc = 0
[   57.337699] page_enc_status_hc invoked, gpa = 1d419000, npages  = 1, enc = 1
[   57.349555] page_enc_status_hc invoked, gpa = 1d49a000, npages  = 2, enc = 1

As can be observed here, all guest MMIO ranges are initially setup as shared, and those are all contigious guest page ranges.

After that the encryption status hypercalls are invoked when DMA gets triggered during disk i/o while booting the guest ... here again the guest page ranges are contigious, though mostly single page is touched and a lot of page re-use is observed.

So a range-based list/structure will be a "good" fit for such usage scenarios.

Thanks,
Ashish

#11 lore raw

Re: [PATCH v2 1/9] KVM: x86: Add AMD SEV specific Hypercall3

Quoted 103 lines · * Ashish Kalra ([email protected]) wrote:
Quoted 83 lines · Ashish Kalra

Hello All,

Quoted 68 lines · Brijesh Singh
Quoted 51 lines · Steve Rutherford
Quoted 34 lines · Sean Christopherson
Quoted 19 lines · Paolo Bonzini
Quoted 18 lines · Sean Christopherson
Quoted 10 lines · Ashish Kalra

From: Brijesh Singh <[email protected]>

KVM hypercall framework relies on alternative framework to patch the VMCALL -> VMMCALL on AMD platform. If a hypercall is made before apply_alternative() is called then it defaults to VMCALL. The approach works fine on non SEV guest. A VMCALL would causes #UD, and hypervisor will be able to decode the instruction and do the right things. But when SEV is active, guest memory is encrypted with guest key and hypervisor will not be able to decode the instruction bytes.

Quoted 2 lines

Add SEV specific hypercall3, it unconditionally uses VMMCALL. The hypercall will be used by the SEV guest to notify encrypted pages to the hypervisor.

What if we invert KVM_HYPERCALL and X86_FEATURE_VMMCALL to default to VMMCALL and opt into VMCALL? It's a synthetic feature flag either way, and I don't think there are any existing KVM hypercalls that happen before alternatives are patched, i.e. it'll be a nop for sane kernel builds.

Quoted 4 lines
Quoted 2 lines

I'm also skeptical that a KVM specific hypercall is the right approach for the encryption behavior, but I'll take that up in the patches later in the series.

Do you think that it's the guest that should "donate" memory for the bitmap instead?

No. Two things I'd like to explore:

1. Making the hypercall to announce/request private vs. shared common across hypervisors (KVM, Hyper-V, VMware, etc...) and technologies (SEV-* and TDX). I'm concerned that we'll end up with multiple hypercalls that do more or less the same thing, e.g. KVM+SEV, Hyper-V+SEV, TDX, etc... Maybe it's a pipe dream, but I'd like to at least explore options before shoving in KVM- only hypercalls.

Quoted 6 lines

2. Tracking shared memory via a list of ranges instead of a using bitmap to track all of guest memory. For most use cases, the vast majority of guest memory will be private, most ranges will be 2mb+, and conversions between private and shared will be uncommon events, i.e. the overhead to walk and split/merge list entries is hopefully not a big concern. I suspect a list would consume far less memory, hopefully without impacting performance.

For a fancier data structure, I'd suggest an interval tree. Linux already has an rbtree-based interval tree implementation, which would likely work, and would probably assuage any performance concerns.

Something like this would not be worth doing unless most of the shared pages were physically contiguous. A sample Ubuntu 20.04 VM on GCP had 60ish discontiguous shared regions. This is by no means a thorough search, but it's suggestive. If this is typical, then the bitmap would be far less efficient than most any interval-based data structure.

Quoted 4 lines

You'd have to allow userspace to upper bound the number of intervals (similar to the maximum bitmap size), to prevent host OOMs due to malicious guests. There's something nice about the guest donating memory for this, since that would eliminate the OOM risk.

Tracking the list of ranges may not be bad idea, especially if we use the some kind of rbtree-based data structure to update the ranges. It will certainly be better than bitmap which grows based on the guest memory size and as you guys see in the practice most of the pages will be guest private. I am not sure if guest donating a memory will cover all the cases, e.g what if we do a memory hotplug (increase the guest ram from 2GB to 64GB), will donated memory range will be enough to store the metadata.

Quoted 1 line

.

With reference to internal discussions regarding the above, i am going to look into specific items as listed below :

1). "hypercall" related :
a). Explore the SEV-SNP page change request structure (included in GHCB), see if there is something common there than can be re-used for SEV/SEV-ES page encryption status hypercalls.
b). Explore if there is any common hypercall framework i can use in Linux/KVM.

Quoted 6 lines

2). related to the "backing" data structure - explore using a range-based list or something like rbtree-based interval tree data structure (as mentioned by Steve above) to replace the current bitmap based implementation.

I do agree that a range-based list or an interval tree data structure is a really good "logical" fit for the guest page encryption status tracking.

We can only keep track of the guest unencrypted shared pages in the range(s) list (which will keep the data structure quite compact) and all the guest private/encrypted memory does not really need any tracking in the list, anything not in the list will be encrypted/private.

Also looking at a more "practical" use case, here is the current log of page encryption status hypercalls when booting a linux guest :

...

<snip>

Quoted 4 lines
[   56.146336] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.146351] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.147261] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.147271] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0

....

Quoted 4 lines
[   56.180730] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.180741] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.180768] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.180782] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1

....

Quoted 4 lines
[   56.197110] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.197120] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.197136] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.197148] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1

....

Quoted 4 lines
[   56.222679] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.222691] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.222707] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.222720] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1

....

Quoted 4 lines
[   56.313747] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.313771] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.313789] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.313803] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1

....

Quoted 6 lines
[   56.459276] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 0
[   56.459428] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 1
[   56.460037] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 1
[   56.460216] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 0
[   56.460299] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 0
[   56.460448] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 1

....

Quoted 10 lines

As can be observed here, all guest MMIO ranges are initially setup as shared, and those are all contigious guest page ranges.

After that the encryption status hypercalls are invoked when DMA gets triggered during disk i/o while booting the guest ... here again the guest page ranges are contigious, though mostly single page is touched and a lot of page re-use is observed.

So a range-based list/structure will be a "good" fit for such usage scenarios.

It seems surprisingly common to flick the same pages back and forth between encrypted and clear for quite a while; why is this?

Dave

Quoted 3 lines

Thanks,
Ashish

Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
#13 lore raw

Re: [PATCH v2 1/9] KVM: x86: Add AMD SEV specific Hypercall3

In reply to a message not in the archive
Quoted 163 lines · * Kalra, Ashish ([email protected]) wrote:

Hello Dave,

On Dec 18, 2020, at 1:40 PM, Dr. David Alan Gilbert <[email protected]> wrote:

* Ashish Kalra ([email protected]) wrote:
On Fri, Dec 11, 2020 at 10:55:42PM +0000, Ashish Kalra wrote:
Hello All,

On Tue, Dec 08, 2020 at 10:29:05AM -0600, Brijesh Singh wrote:

On 12/7/20 9:09 PM, Steve Rutherford wrote:
On Mon, Dec 7, 2020 at 12:42 PM Sean Christopherson <[email protected]> wrote:
On Sun, Dec 06, 2020, Paolo Bonzini wrote:
On 03/12/20 01:34, Sean Christopherson wrote:
On Tue, Dec 01, 2020, Ashish Kalra wrote:
From: Brijesh Singh <[email protected]>

KVM hypercall framework relies on alternative framework to patch the VMCALL -> VMMCALL on AMD platform. If a hypercall is made before apply_alternative() is called then it defaults to VMCALL. The approach works fine on non SEV guest. A VMCALL would causes #UD, and hypervisor will be able to decode the instruction and do the right things. But when SEV is active, guest memory is encrypted with guest key and hypervisor will not be able to decode the instruction bytes.

Add SEV specific hypercall3, it unconditionally uses VMMCALL. The hypercall will be used by the SEV guest to notify encrypted pages to the hypervisor. What if we invert KVM_HYPERCALL and X86_FEATURE_VMMCALL to default to VMMCALL and opt into VMCALL? It's a synthetic feature flag either way, and I don't think there are any existing KVM hypercalls that happen before alternatives are patched, i.e. it'll be a nop for sane kernel builds.

I'm also skeptical that a KVM specific hypercall is the right approach for the encryption behavior, but I'll take that up in the patches later in the series. Do you think that it's the guest that should "donate" memory for the bitmap instead?
No. Two things I'd like to explore:

1. Making the hypercall to announce/request private vs. shared common across hypervisors (KVM, Hyper-V, VMware, etc...) and technologies (SEV-* and TDX). I'm concerned that we'll end up with multiple hypercalls that do more or less the same thing, e.g. KVM+SEV, Hyper-V+SEV, TDX, etc... Maybe it's a pipe dream, but I'd like to at least explore options before shoving in KVM- only hypercalls.

2. Tracking shared memory via a list of ranges instead of a using bitmap to track all of guest memory. For most use cases, the vast majority of guest memory will be private, most ranges will be 2mb+, and conversions between private and shared will be uncommon events, i.e. the overhead to walk and split/merge list entries is hopefully not a big concern. I suspect a list would consume far less memory, hopefully without impacting performance.
For a fancier data structure, I'd suggest an interval tree. Linux already has an rbtree-based interval tree implementation, which would likely work, and would probably assuage any performance concerns.

Something like this would not be worth doing unless most of the shared pages were physically contiguous. A sample Ubuntu 20.04 VM on GCP had 60ish discontiguous shared regions. This is by no means a thorough search, but it's suggestive. If this is typical, then the bitmap would be far less efficient than most any interval-based data structure.

You'd have to allow userspace to upper bound the number of intervals (similar to the maximum bitmap size), to prevent host OOMs due to malicious guests. There's something nice about the guest donating memory for this, since that would eliminate the OOM risk.

Tracking the list of ranges may not be bad idea, especially if we use the some kind of rbtree-based data structure to update the ranges. It will certainly be better than bitmap which grows based on the guest memory size and as you guys see in the practice most of the pages will be guest private. I am not sure if guest donating a memory will cover all the cases, e.g what if we do a memory hotplug (increase the guest ram from 2GB to 64GB), will donated memory range will be enough to store the metadata.

.

With reference to internal discussions regarding the above, i am going to look into specific items as listed below :

1). "hypercall" related :
a). Explore the SEV-SNP page change request structure (included in GHCB), see if there is something common there than can be re-used for SEV/SEV-ES page encryption status hypercalls.
b). Explore if there is any common hypercall framework i can use in Linux/KVM.

2). related to the "backing" data structure - explore using a range-based list or something like rbtree-based interval tree data structure (as mentioned by Steve above) to replace the current bitmap based implementation.

I do agree that a range-based list or an interval tree data structure is a really good "logical" fit for the guest page encryption status tracking.

We can only keep track of the guest unencrypted shared pages in the range(s) list (which will keep the data structure quite compact) and all the guest private/encrypted memory does not really need any tracking in the list, anything not in the list will be encrypted/private.

Also looking at a more "practical" use case, here is the current log of page encryption status hypercalls when booting a linux guest :

...

<snip>

[   56.146336] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.146351] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.147261] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.147271] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0

....

[   56.180730] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.180741] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.180768] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.180782] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1

....

[   56.197110] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.197120] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.197136] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.197148] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1

....

[   56.222679] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.222691] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.222707] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.222720] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1

....

[   56.313747] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.313771] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.313789] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.313803] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
....
[   56.459276] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 0
[   56.459428] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 1
[   56.460037] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 1
[   56.460216] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 0
[   56.460299] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 0
[   56.460448] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 1

....

As can be observed here, all guest MMIO ranges are initially setup as shared, and those are all contigious guest page ranges.

After that the encryption status hypercalls are invoked when DMA gets triggered during disk i/o while booting the guest ... here again the guest page ranges are contigious, though mostly single page is touched and a lot of page re-use is observed.

So a range-based list/structure will be a "good" fit for such usage scenarios.

It seems surprisingly common to flick the same pages back and forth between encrypted and clear for quite a while; why is this?

Quoted 6 lines

dma_alloc_coherent()'s will allocate pages and then call set_decrypted() on them and then at dma_free_coherent(), set_encrypted() is called on the pages to be freed. So these observations in the logs where a lot of single 4K pages are seeing C-bit transitions and corresponding hypercalls are the ones associated with dma_alloc_coherent().

It makes me wonder if it might be worth teaching it to hold onto those DMA pages somewhere until it needs them for something else and avoid the extra hypercalls; just something to think about.

Dave

Quoted 7 lines

Thanks,
Ashish

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
#14 lore raw

Re: [PATCH v2 1/9] KVM: x86: Add AMD SEV specific Hypercall3

Quoted 171 lines · Dr. David Alan Gilbert
Quoted 163 lines · * Kalra, Ashish ([email protected]) wrote:

Hello Dave,

On Dec 18, 2020, at 1:40 PM, Dr. David Alan Gilbert <[email protected]> wrote:

* Ashish Kalra ([email protected]) wrote:
On Fri, Dec 11, 2020 at 10:55:42PM +0000, Ashish Kalra wrote:
Hello All,

On Tue, Dec 08, 2020 at 10:29:05AM -0600, Brijesh Singh wrote:

On 12/7/20 9:09 PM, Steve Rutherford wrote:
On Mon, Dec 7, 2020 at 12:42 PM Sean Christopherson <[email protected]> wrote:
On Sun, Dec 06, 2020, Paolo Bonzini wrote:
On 03/12/20 01:34, Sean Christopherson wrote:
On Tue, Dec 01, 2020, Ashish Kalra wrote:
From: Brijesh Singh <[email protected]>

KVM hypercall framework relies on alternative framework to patch the VMCALL -> VMMCALL on AMD platform. If a hypercall is made before apply_alternative() is called then it defaults to VMCALL. The approach works fine on non SEV guest. A VMCALL would causes #UD, and hypervisor will be able to decode the instruction and do the right things. But when SEV is active, guest memory is encrypted with guest key and hypervisor will not be able to decode the instruction bytes.

Add SEV specific hypercall3, it unconditionally uses VMMCALL. The hypercall will be used by the SEV guest to notify encrypted pages to the hypervisor. What if we invert KVM_HYPERCALL and X86_FEATURE_VMMCALL to default to VMMCALL and opt into VMCALL? It's a synthetic feature flag either way, and I don't think there are any existing KVM hypercalls that happen before alternatives are patched, i.e. it'll be a nop for sane kernel builds.

I'm also skeptical that a KVM specific hypercall is the right approach for the encryption behavior, but I'll take that up in the patches later in the series. Do you think that it's the guest that should "donate" memory for the bitmap instead?
No. Two things I'd like to explore:

1. Making the hypercall to announce/request private vs. shared common across hypervisors (KVM, Hyper-V, VMware, etc...) and technologies (SEV-* and TDX). I'm concerned that we'll end up with multiple hypercalls that do more or less the same thing, e.g. KVM+SEV, Hyper-V+SEV, TDX, etc... Maybe it's a pipe dream, but I'd like to at least explore options before shoving in KVM- only hypercalls.

2. Tracking shared memory via a list of ranges instead of a using bitmap to track all of guest memory. For most use cases, the vast majority of guest memory will be private, most ranges will be 2mb+, and conversions between private and shared will be uncommon events, i.e. the overhead to walk and split/merge list entries is hopefully not a big concern. I suspect a list would consume far less memory, hopefully without impacting performance.
For a fancier data structure, I'd suggest an interval tree. Linux already has an rbtree-based interval tree implementation, which would likely work, and would probably assuage any performance concerns.

Something like this would not be worth doing unless most of the shared pages were physically contiguous. A sample Ubuntu 20.04 VM on GCP had 60ish discontiguous shared regions. This is by no means a thorough search, but it's suggestive. If this is typical, then the bitmap would be far less efficient than most any interval-based data structure.

You'd have to allow userspace to upper bound the number of intervals (similar to the maximum bitmap size), to prevent host OOMs due to malicious guests. There's something nice about the guest donating memory for this, since that would eliminate the OOM risk.

Tracking the list of ranges may not be bad idea, especially if we use the some kind of rbtree-based data structure to update the ranges. It will certainly be better than bitmap which grows based on the guest memory size and as you guys see in the practice most of the pages will be guest private. I am not sure if guest donating a memory will cover all the cases, e.g what if we do a memory hotplug (increase the guest ram from 2GB to 64GB), will donated memory range will be enough to store the metadata.

.

With reference to internal discussions regarding the above, i am going to look into specific items as listed below :

1). "hypercall" related :
a). Explore the SEV-SNP page change request structure (included in GHCB), see if there is something common there than can be re-used for SEV/SEV-ES page encryption status hypercalls.
b). Explore if there is any common hypercall framework i can use in Linux/KVM.

2). related to the "backing" data structure - explore using a range-based list or something like rbtree-based interval tree data structure (as mentioned by Steve above) to replace the current bitmap based implementation.

I do agree that a range-based list or an interval tree data structure is a really good "logical" fit for the guest page encryption status tracking.

We can only keep track of the guest unencrypted shared pages in the range(s) list (which will keep the data structure quite compact) and all the guest private/encrypted memory does not really need any tracking in the list, anything not in the list will be encrypted/private.

Also looking at a more "practical" use case, here is the current log of page encryption status hypercalls when booting a linux guest :

...

<snip>

[   56.146336] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.146351] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.147261] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.147271] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0

....

[   56.180730] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.180741] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.180768] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.180782] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1

....

[   56.197110] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.197120] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.197136] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.197148] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1

....

[   56.222679] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.222691] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.222707] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.222720] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1

....

[   56.313747] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.313771] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.313789] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.313803] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
....
[   56.459276] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 0
[   56.459428] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 1
[   56.460037] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 1
[   56.460216] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 0
[   56.460299] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 0
[   56.460448] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 1

....

As can be observed here, all guest MMIO ranges are initially setup as shared, and those are all contigious guest page ranges.

After that the encryption status hypercalls are invoked when DMA gets triggered during disk i/o while booting the guest ... here again the guest page ranges are contigious, though mostly single page is touched and a lot of page re-use is observed.

So a range-based list/structure will be a "good" fit for such usage scenarios.

It seems surprisingly common to flick the same pages back and forth between encrypted and clear for quite a while; why is this?

Quoted 6 lines

dma_alloc_coherent()'s will allocate pages and then call set_decrypted() on them and then at dma_free_coherent(), set_encrypted() is called on the pages to be freed. So these observations in the logs where a lot of single 4K pages are seeing C-bit transitions and corresponding hypercalls are the ones associated with dma_alloc_coherent().

Quoted 5 lines

It makes me wonder if it might be worth teaching it to hold onto those DMA pages somewhere until it needs them for something else and avoid the extra hypercalls; just something to think about.

Dave

Following up on this discussion and looking at the hypercall logs and DMA usage scenarios on the SEV, I have the following additional observations and comments :

It is mostly the Guest MMIO regions setup as un-encrypted by uefi/edk2 initially, which will be the "static" nodes in the backing data structure for page encryption status. These will be like 15-20 nodes/entries.

Drivers doing DMA allocations using GFP_ATOMIC will be fetching DMA buffers from the pre-allocated unencrypted atomic pool, hence it will be a "static" node added at kernel startup.

As we see with the logs, almost all runtime C-bit transitions and corresponding hypercalls will be from DMA I/O and dma_alloc_coherent/dma_free_coherent calls, these will be using 4K/single pages and mostly fragmented ranges, so if we use a "rbtree" based interval tree then there will be a lot of tree insertions and deletions (dma_alloc_coherent followed with a dma_free_coherent), so this will lead to a lot of expensive tree rotations and re-balancing, compared to much less complex and faster linked list node insertions and deletions (if we use a list based structure to represent these interval ranges).

Also as the static nodes in the structure will be quite limited (all the above DMA I/O added ranges will simply be inserted and removed), so a linked list lookup won't be too expensive compared to a tree lookup. In other words, this be a fixed size list.

Looking at the above, I am now more inclined to use a list based structure to represent the page encryption status.

Looking fwd. to any comments/feedback/thoughts on the above.

Thanks,
Ashish

#15 lore raw

Re: [PATCH v2 1/9] KVM: x86: Add AMD SEV specific Hypercall3

Avoiding an rbtree for such a small (but unstable) list seems correct.

For the unencrypted region list strategy, the only questions that I have are fairly secondary.
- How should the kernel upper bound the size of the list in the face of malicious guests, but still support large guests? (Something similar to the size provided in the bitmap API would work).
- What serialization format should be used for the ioctl API? (Usermode could send down a pointer to a user region and a size. The kernel could then populate that with an array of structs containing bases and limits for unencrypted regions.)
- How will the kernel tag a guest as having exceeded its maximum list size, in order to indicate that the list is now incomplete? (Track a poison bit, and send it up when getting the serialized list of regions).

In my view, there are two main competitors to this strategy:
- (Existing) Bitmap API
- A guest memory donation based model

The existing bitmap API avoids any issues with growing too large, since it's size is predictable.

To elaborate on the memory donation based model, the guest could put an encryption status data structure into unencrypted guest memory, and then use a hypercall to inform the host where the base of that structure is located. The main advantage of this is that it side steps any issues around malicious guests causing large allocations.

The unencrypted region list seems very practical. It's biggest advantage over the bitmap is how cheap it will be to pass the structure up from the kernel. A memory donation based model could achieve similar performance, but with some additional complexity.

Does anyone view the memory donation model as worth the complexity? Does anyone think the simplicity of the bitmap is a better tradeoff compared to an unencrypted region list?
Or have other ideas that are not mentioned here?

Quoted 201 lines · Ashish Kalra
Quoted 171 lines · Dr. David Alan Gilbert
Quoted 163 lines · * Kalra, Ashish ([email protected]) wrote:

Hello Dave,

On Dec 18, 2020, at 1:40 PM, Dr. David Alan Gilbert <[email protected]> wrote:

* Ashish Kalra ([email protected]) wrote:
On Fri, Dec 11, 2020 at 10:55:42PM +0000, Ashish Kalra wrote:
Hello All,

On Tue, Dec 08, 2020 at 10:29:05AM -0600, Brijesh Singh wrote:

On 12/7/20 9:09 PM, Steve Rutherford wrote:
On Mon, Dec 7, 2020 at 12:42 PM Sean Christopherson <[email protected]> wrote:
On Sun, Dec 06, 2020, Paolo Bonzini wrote:
On 03/12/20 01:34, Sean Christopherson wrote:
On Tue, Dec 01, 2020, Ashish Kalra wrote:
From: Brijesh Singh <[email protected]>

KVM hypercall framework relies on alternative framework to patch the VMCALL -> VMMCALL on AMD platform. If a hypercall is made before apply_alternative() is called then it defaults to VMCALL. The approach works fine on non SEV guest. A VMCALL would causes #UD, and hypervisor will be able to decode the instruction and do the right things. But when SEV is active, guest memory is encrypted with guest key and hypervisor will not be able to decode the instruction bytes.

Add SEV specific hypercall3, it unconditionally uses VMMCALL. The hypercall will be used by the SEV guest to notify encrypted pages to the hypervisor. What if we invert KVM_HYPERCALL and X86_FEATURE_VMMCALL to default to VMMCALL and opt into VMCALL? It's a synthetic feature flag either way, and I don't think there are any existing KVM hypercalls that happen before alternatives are patched, i.e. it'll be a nop for sane kernel builds.

I'm also skeptical that a KVM specific hypercall is the right approach for the encryption behavior, but I'll take that up in the patches later in the series. Do you think that it's the guest that should "donate" memory for the bitmap instead?
No. Two things I'd like to explore:

1. Making the hypercall to announce/request private vs. shared common across hypervisors (KVM, Hyper-V, VMware, etc...) and technologies (SEV-* and TDX). I'm concerned that we'll end up with multiple hypercalls that do more or less the same thing, e.g. KVM+SEV, Hyper-V+SEV, TDX, etc... Maybe it's a pipe dream, but I'd like to at least explore options before shoving in KVM- only hypercalls.

2. Tracking shared memory via a list of ranges instead of a using bitmap to track all of guest memory. For most use cases, the vast majority of guest memory will be private, most ranges will be 2mb+, and conversions between private and shared will be uncommon events, i.e. the overhead to walk and split/merge list entries is hopefully not a big concern. I suspect a list would consume far less memory, hopefully without impacting performance.
For a fancier data structure, I'd suggest an interval tree. Linux already has an rbtree-based interval tree implementation, which would likely work, and would probably assuage any performance concerns.

Something like this would not be worth doing unless most of the shared pages were physically contiguous. A sample Ubuntu 20.04 VM on GCP had 60ish discontiguous shared regions. This is by no means a thorough search, but it's suggestive. If this is typical, then the bitmap would be far less efficient than most any interval-based data structure.

You'd have to allow userspace to upper bound the number of intervals (similar to the maximum bitmap size), to prevent host OOMs due to malicious guests. There's something nice about the guest donating memory for this, since that would eliminate the OOM risk.

Tracking the list of ranges may not be bad idea, especially if we use the some kind of rbtree-based data structure to update the ranges. It will certainly be better than bitmap which grows based on the guest memory size and as you guys see in the practice most of the pages will be guest private. I am not sure if guest donating a memory will cover all the cases, e.g what if we do a memory hotplug (increase the guest ram from 2GB to 64GB), will donated memory range will be enough to store the metadata.

.

With reference to internal discussions regarding the above, i am going to look into specific items as listed below :

1). "hypercall" related :
a). Explore the SEV-SNP page change request structure (included in GHCB), see if there is something common there than can be re-used for SEV/SEV-ES page encryption status hypercalls.
b). Explore if there is any common hypercall framework i can use in Linux/KVM.

2). related to the "backing" data structure - explore using a range-based list or something like rbtree-based interval tree data structure (as mentioned by Steve above) to replace the current bitmap based implementation.

I do agree that a range-based list or an interval tree data structure is a really good "logical" fit for the guest page encryption status tracking.

We can only keep track of the guest unencrypted shared pages in the range(s) list (which will keep the data structure quite compact) and all the guest private/encrypted memory does not really need any tracking in the list, anything not in the list will be encrypted/private.

Also looking at a more "practical" use case, here is the current log of page encryption status hypercalls when booting a linux guest :

...

<snip>

[   56.146336] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.146351] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.147261] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.147271] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0

....

[   56.180730] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.180741] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.180768] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.180782] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1

....

[   56.197110] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.197120] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.197136] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.197148] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1

....

[   56.222679] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.222691] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.222707] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.222720] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1

....

[   56.313747] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.313771] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.313789] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.313803] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
....
[   56.459276] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 0
[   56.459428] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 1
[   56.460037] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 1
[   56.460216] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 0
[   56.460299] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 0
[   56.460448] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 1

....

As can be observed here, all guest MMIO ranges are initially setup as shared, and those are all contigious guest page ranges.

After that the encryption status hypercalls are invoked when DMA gets triggered during disk i/o while booting the guest ... here again the guest page ranges are contigious, though mostly single page is touched and a lot of page re-use is observed.

So a range-based list/structure will be a "good" fit for such usage scenarios.

It seems surprisingly common to flick the same pages back and forth between encrypted and clear for quite a while; why is this?

Quoted 6 lines

dma_alloc_coherent()'s will allocate pages and then call set_decrypted() on them and then at dma_free_coherent(), set_encrypted() is called on the pages to be freed. So these observations in the logs where a lot of single 4K pages are seeing C-bit transitions and corresponding hypercalls are the ones associated with dma_alloc_coherent().

Quoted 5 lines

It makes me wonder if it might be worth teaching it to hold onto those DMA pages somewhere until it needs them for something else and avoid the extra hypercalls; just something to think about.

Dave

Following up on this discussion and looking at the hypercall logs and DMA usage scenarios on the SEV, I have the following additional observations and comments :

It is mostly the Guest MMIO regions setup as un-encrypted by uefi/edk2 initially, which will be the "static" nodes in the backing data structure for page encryption status. These will be like 15-20 nodes/entries.

Drivers doing DMA allocations using GFP_ATOMIC will be fetching DMA buffers from the pre-allocated unencrypted atomic pool, hence it will be a "static" node added at kernel startup.

As we see with the logs, almost all runtime C-bit transitions and corresponding hypercalls will be from DMA I/O and dma_alloc_coherent/dma_free_coherent calls, these will be using 4K/single pages and mostly fragmented ranges, so if we use a "rbtree" based interval tree then there will be a lot of tree insertions and deletions (dma_alloc_coherent followed with a dma_free_coherent), so this will lead to a lot of expensive tree rotations and re-balancing, compared to much less complex and faster linked list node insertions and deletions (if we use a list based structure to represent these interval ranges).

Also as the static nodes in the structure will be quite limited (all the above DMA I/O added ranges will simply be inserted and removed), so a linked list lookup won't be too expensive compared to a tree lookup. In other words, this be a fixed size list.

Looking at the above, I am now more inclined to use a list based structure to represent the page encryption status.

Looking fwd. to any comments/feedback/thoughts on the above.

Thanks,
Ashish

#16 lore raw

Re: [PATCH v2 1/9] KVM: x86: Add AMD SEV specific Hypercall3

Hello Steve,

My thoughts here ...

Quoted 2 lines · Steve Rutherford

Avoiding an rbtree for such a small (but unstable) list seems correct.

I agree.

Quoted 5 lines

For the unencrypted region list strategy, the only questions that I have are fairly secondary.
- How should the kernel upper bound the size of the list in the face of malicious guests, but still support large guests? (Something similar to the size provided in the bitmap API would work).

Please note that in our current implementation, we don't do any bitmap resize based on guest page encryption status hypercall, the max. bitmap size is computed dynamically by walking the KVM memslots and finding the maximum guest physical address size.

So, malicious guests cannot do large bitmap allocations using the hypercalls.

Quoted 9 lines

- What serialization format should be used for the ioctl API? (Usermode could send down a pointer to a user region and a size. The kernel could then populate that with an array of structs containing bases and limits for unencrypted regions.)
- How will the kernel tag a guest as having exceeded its maximum list size, in order to indicate that the list is now incomplete? (Track a poison bit, and send it up when getting the serialized list of regions).

With reference to the serialization concerns with active live migration and simultaneous page encryption bitmap updates, please note that in our current VMM implementation, after each memory region migration cycle we pause live migration, re-sync the page encryption bitmap, XOR it to the last synced bitmap, and then re-transfer any modified pages accordingly.

I have a prototype implementation for this in Qemu, which seems to work fine.

Quoted 7 lines

In my view, there are two main competitors to this strategy:
- (Existing) Bitmap API
- A guest memory donation based model

The existing bitmap API avoids any issues with growing too large, since it's size is predictable.

Yes, as i mentioned above, it's size is predictable and cannot grow too large.

Quoted 6 lines

To elaborate on the memory donation based model, the guest could put an encryption status data structure into unencrypted guest memory, and then use a hypercall to inform the host where the base of that structure is located. The main advantage of this is that it side steps any issues around malicious guests causing large allocations.

Quoted 8 lines

The unencrypted region list seems very practical. It's biggest advantage over the bitmap is how cheap it will be to pass the structure up from the kernel. A memory donation based model could achieve similar performance, but with some additional complexity.

Does anyone view the memory donation model as worth the complexity? Does anyone think the simplicity of the bitmap is a better tradeoff compared to an unencrypted region list?

One advantage in sticking with the bitmap is that it maps very nicely to the dirty bitmap page tracking logic in KVM/Qemu. The way Brijesh designed and implemented it is very similar to dirty page bitmap tracking and syncing between KVM and Qemu. The same logic is re-used for the page encryption bitmap which means quite mininal changes and code resuse in Qemu.

Any changes to the backing data structure, will require additional mapping logic to be added to Qemu.

This is one advantage in keeping the bitmap logic.

Thanks,
Ashish

Quoted 205 lines

Or have other ideas that are not mentioned here?

Quoted 201 lines · Ashish Kalra
Quoted 171 lines · Dr. David Alan Gilbert
Quoted 163 lines · * Kalra, Ashish ([email protected]) wrote:

Hello Dave,

On Dec 18, 2020, at 1:40 PM, Dr. David Alan Gilbert <[email protected]> wrote:

* Ashish Kalra ([email protected]) wrote:
On Fri, Dec 11, 2020 at 10:55:42PM +0000, Ashish Kalra wrote:
Hello All,

On Tue, Dec 08, 2020 at 10:29:05AM -0600, Brijesh Singh wrote:

On 12/7/20 9:09 PM, Steve Rutherford wrote:
On Mon, Dec 7, 2020 at 12:42 PM Sean Christopherson <[email protected]> wrote:
On Sun, Dec 06, 2020, Paolo Bonzini wrote:
On 03/12/20 01:34, Sean Christopherson wrote:
On Tue, Dec 01, 2020, Ashish Kalra wrote:
From: Brijesh Singh <[email protected]>

KVM hypercall framework relies on alternative framework to patch the VMCALL -> VMMCALL on AMD platform. If a hypercall is made before apply_alternative() is called then it defaults to VMCALL. The approach works fine on non SEV guest. A VMCALL would causes #UD, and hypervisor will be able to decode the instruction and do the right things. But when SEV is active, guest memory is encrypted with guest key and hypervisor will not be able to decode the instruction bytes.

Add SEV specific hypercall3, it unconditionally uses VMMCALL. The hypercall will be used by the SEV guest to notify encrypted pages to the hypervisor. What if we invert KVM_HYPERCALL and X86_FEATURE_VMMCALL to default to VMMCALL and opt into VMCALL? It's a synthetic feature flag either way, and I don't think there are any existing KVM hypercalls that happen before alternatives are patched, i.e. it'll be a nop for sane kernel builds.

I'm also skeptical that a KVM specific hypercall is the right approach for the encryption behavior, but I'll take that up in the patches later in the series. Do you think that it's the guest that should "donate" memory for the bitmap instead?
No. Two things I'd like to explore:

1. Making the hypercall to announce/request private vs. shared common across hypervisors (KVM, Hyper-V, VMware, etc...) and technologies (SEV-* and TDX). I'm concerned that we'll end up with multiple hypercalls that do more or less the same thing, e.g. KVM+SEV, Hyper-V+SEV, TDX, etc... Maybe it's a pipe dream, but I'd like to at least explore options before shoving in KVM- only hypercalls.

2. Tracking shared memory via a list of ranges instead of a using bitmap to track all of guest memory. For most use cases, the vast majority of guest memory will be private, most ranges will be 2mb+, and conversions between private and shared will be uncommon events, i.e. the overhead to walk and split/merge list entries is hopefully not a big concern. I suspect a list would consume far less memory, hopefully without impacting performance.
For a fancier data structure, I'd suggest an interval tree. Linux already has an rbtree-based interval tree implementation, which would likely work, and would probably assuage any performance concerns.

Something like this would not be worth doing unless most of the shared pages were physically contiguous. A sample Ubuntu 20.04 VM on GCP had 60ish discontiguous shared regions. This is by no means a thorough search, but it's suggestive. If this is typical, then the bitmap would be far less efficient than most any interval-based data structure.

You'd have to allow userspace to upper bound the number of intervals (similar to the maximum bitmap size), to prevent host OOMs due to malicious guests. There's something nice about the guest donating memory for this, since that would eliminate the OOM risk.

Tracking the list of ranges may not be bad idea, especially if we use the some kind of rbtree-based data structure to update the ranges. It will certainly be better than bitmap which grows based on the guest memory size and as you guys see in the practice most of the pages will be guest private. I am not sure if guest donating a memory will cover all the cases, e.g what if we do a memory hotplug (increase the guest ram from 2GB to 64GB), will donated memory range will be enough to store the metadata.

.

With reference to internal discussions regarding the above, i am going to look into specific items as listed below :

1). "hypercall" related :
a). Explore the SEV-SNP page change request structure (included in GHCB), see if there is something common there than can be re-used for SEV/SEV-ES page encryption status hypercalls.
b). Explore if there is any common hypercall framework i can use in Linux/KVM.

2). related to the "backing" data structure - explore using a range-based list or something like rbtree-based interval tree data structure (as mentioned by Steve above) to replace the current bitmap based implementation.

I do agree that a range-based list or an interval tree data structure is a really good "logical" fit for the guest page encryption status tracking.

We can only keep track of the guest unencrypted shared pages in the range(s) list (which will keep the data structure quite compact) and all the guest private/encrypted memory does not really need any tracking in the list, anything not in the list will be encrypted/private.

Also looking at a more "practical" use case, here is the current log of page encryption status hypercalls when booting a linux guest :

...

<snip>

[   56.146336] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.146351] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.147261] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.147271] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0

....

[   56.180730] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.180741] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.180768] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.180782] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1

....

[   56.197110] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.197120] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.197136] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.197148] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1

....

[   56.222679] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.222691] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.222707] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.222720] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1

....

[   56.313747] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.313771] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.313789] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.313803] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
....
[   56.459276] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 0
[   56.459428] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 1
[   56.460037] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 1
[   56.460216] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 0
[   56.460299] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 0
[   56.460448] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 1

....

As can be observed here, all guest MMIO ranges are initially setup as shared, and those are all contigious guest page ranges.

After that the encryption status hypercalls are invoked when DMA gets triggered during disk i/o while booting the guest ... here again the guest page ranges are contigious, though mostly single page is touched and a lot of page re-use is observed.

So a range-based list/structure will be a "good" fit for such usage scenarios.

It seems surprisingly common to flick the same pages back and forth between encrypted and clear for quite a while; why is this?

Quoted 6 lines

dma_alloc_coherent()'s will allocate pages and then call set_decrypted() on them and then at dma_free_coherent(), set_encrypted() is called on the pages to be freed. So these observations in the logs where a lot of single 4K pages are seeing C-bit transitions and corresponding hypercalls are the ones associated with dma_alloc_coherent().

Quoted 5 lines

It makes me wonder if it might be worth teaching it to hold onto those DMA pages somewhere until it needs them for something else and avoid the extra hypercalls; just something to think about.

Dave

Following up on this discussion and looking at the hypercall logs and DMA usage scenarios on the SEV, I have the following additional observations and comments :

It is mostly the Guest MMIO regions setup as un-encrypted by uefi/edk2 initially, which will be the "static" nodes in the backing data structure for page encryption status. These will be like 15-20 nodes/entries.

Drivers doing DMA allocations using GFP_ATOMIC will be fetching DMA buffers from the pre-allocated unencrypted atomic pool, hence it will be a "static" node added at kernel startup.

As we see with the logs, almost all runtime C-bit transitions and corresponding hypercalls will be from DMA I/O and dma_alloc_coherent/dma_free_coherent calls, these will be using 4K/single pages and mostly fragmented ranges, so if we use a "rbtree" based interval tree then there will be a lot of tree insertions and deletions (dma_alloc_coherent followed with a dma_free_coherent), so this will lead to a lot of expensive tree rotations and re-balancing, compared to much less complex and faster linked list node insertions and deletions (if we use a list based structure to represent these interval ranges).

Also as the static nodes in the structure will be quite limited (all the above DMA I/O added ranges will simply be inserted and removed), so a linked list lookup won't be too expensive compared to a tree lookup. In other words, this be a fixed size list.

Looking at the above, I am now more inclined to use a list based structure to represent the page encryption status.

Looking fwd. to any comments/feedback/thoughts on the above.

Thanks,
Ashish

#17 lore raw

Re: [PATCH v2 1/9] KVM: x86: Add AMD SEV specific Hypercall3

Hello Steve,

Sorry, i realized later that i replied to this email with regard to the current bitmap implementation and not the unencrpyted region list strategy.

I am now looking at your thoughts/questions with regard to the unencrypted region list strategy and will reply to them accordingly.

Thanks,
Ashish

Quoted 290 lines · Ashish Kalra

Hello Steve,

My thoughts here ...

Quoted 2 lines · Steve Rutherford

Avoiding an rbtree for such a small (but unstable) list seems correct.

I agree.

Quoted 5 lines

For the unencrypted region list strategy, the only questions that I have are fairly secondary.
- How should the kernel upper bound the size of the list in the face of malicious guests, but still support large guests? (Something similar to the size provided in the bitmap API would work).

Please note that in our current implementation, we don't do any bitmap resize based on guest page encryption status hypercall, the max. bitmap size is computed dynamically by walking the KVM memslots and finding the maximum guest physical address size.

So, malicious guests cannot do large bitmap allocations using the hypercalls.

Quoted 9 lines

- What serialization format should be used for the ioctl API? (Usermode could send down a pointer to a user region and a size. The kernel could then populate that with an array of structs containing bases and limits for unencrypted regions.)
- How will the kernel tag a guest as having exceeded its maximum list size, in order to indicate that the list is now incomplete? (Track a poison bit, and send it up when getting the serialized list of regions).

With reference to the serialization concerns with active live migration and simultaneous page encryption bitmap updates, please note that in our current VMM implementation, after each memory region migration cycle we pause live migration, re-sync the page encryption bitmap, XOR it to the last synced bitmap, and then re-transfer any modified pages accordingly.

I have a prototype implementation for this in Qemu, which seems to work fine.

Quoted 7 lines

In my view, there are two main competitors to this strategy:
- (Existing) Bitmap API
- A guest memory donation based model

The existing bitmap API avoids any issues with growing too large, since it's size is predictable.

Yes, as i mentioned above, it's size is predictable and cannot grow too large.

Quoted 6 lines

To elaborate on the memory donation based model, the guest could put an encryption status data structure into unencrypted guest memory, and then use a hypercall to inform the host where the base of that structure is located. The main advantage of this is that it side steps any issues around malicious guests causing large allocations.

Quoted 8 lines

The unencrypted region list seems very practical. It's biggest advantage over the bitmap is how cheap it will be to pass the structure up from the kernel. A memory donation based model could achieve similar performance, but with some additional complexity.

Does anyone view the memory donation model as worth the complexity? Does anyone think the simplicity of the bitmap is a better tradeoff compared to an unencrypted region list?

One advantage in sticking with the bitmap is that it maps very nicely to the dirty bitmap page tracking logic in KVM/Qemu. The way Brijesh designed and implemented it is very similar to dirty page bitmap tracking and syncing between KVM and Qemu. The same logic is re-used for the page encryption bitmap which means quite mininal changes and code resuse in Qemu.

Any changes to the backing data structure, will require additional mapping logic to be added to Qemu.

This is one advantage in keeping the bitmap logic.

Thanks,
Ashish

Quoted 205 lines

Or have other ideas that are not mentioned here?

Quoted 201 lines · Ashish Kalra
Quoted 171 lines · Dr. David Alan Gilbert
Quoted 163 lines · * Kalra, Ashish ([email protected]) wrote:

Hello Dave,

On Dec 18, 2020, at 1:40 PM, Dr. David Alan Gilbert <[email protected]> wrote:

* Ashish Kalra ([email protected]) wrote:
On Fri, Dec 11, 2020 at 10:55:42PM +0000, Ashish Kalra wrote:
Hello All,

On Tue, Dec 08, 2020 at 10:29:05AM -0600, Brijesh Singh wrote:

On 12/7/20 9:09 PM, Steve Rutherford wrote:
On Mon, Dec 7, 2020 at 12:42 PM Sean Christopherson <[email protected]> wrote:
On Sun, Dec 06, 2020, Paolo Bonzini wrote:
On 03/12/20 01:34, Sean Christopherson wrote:
On Tue, Dec 01, 2020, Ashish Kalra wrote:
From: Brijesh Singh <[email protected]>

KVM hypercall framework relies on alternative framework to patch the VMCALL -> VMMCALL on AMD platform. If a hypercall is made before apply_alternative() is called then it defaults to VMCALL. The approach works fine on non SEV guest. A VMCALL would causes #UD, and hypervisor will be able to decode the instruction and do the right things. But when SEV is active, guest memory is encrypted with guest key and hypervisor will not be able to decode the instruction bytes.

Add SEV specific hypercall3, it unconditionally uses VMMCALL. The hypercall will be used by the SEV guest to notify encrypted pages to the hypervisor. What if we invert KVM_HYPERCALL and X86_FEATURE_VMMCALL to default to VMMCALL and opt into VMCALL? It's a synthetic feature flag either way, and I don't think there are any existing KVM hypercalls that happen before alternatives are patched, i.e. it'll be a nop for sane kernel builds.

I'm also skeptical that a KVM specific hypercall is the right approach for the encryption behavior, but I'll take that up in the patches later in the series. Do you think that it's the guest that should "donate" memory for the bitmap instead?
No. Two things I'd like to explore:

1. Making the hypercall to announce/request private vs. shared common across hypervisors (KVM, Hyper-V, VMware, etc...) and technologies (SEV-* and TDX). I'm concerned that we'll end up with multiple hypercalls that do more or less the same thing, e.g. KVM+SEV, Hyper-V+SEV, TDX, etc... Maybe it's a pipe dream, but I'd like to at least explore options before shoving in KVM- only hypercalls.

2. Tracking shared memory via a list of ranges instead of a using bitmap to track all of guest memory. For most use cases, the vast majority of guest memory will be private, most ranges will be 2mb+, and conversions between private and shared will be uncommon events, i.e. the overhead to walk and split/merge list entries is hopefully not a big concern. I suspect a list would consume far less memory, hopefully without impacting performance.
For a fancier data structure, I'd suggest an interval tree. Linux already has an rbtree-based interval tree implementation, which would likely work, and would probably assuage any performance concerns.

Something like this would not be worth doing unless most of the shared pages were physically contiguous. A sample Ubuntu 20.04 VM on GCP had 60ish discontiguous shared regions. This is by no means a thorough search, but it's suggestive. If this is typical, then the bitmap would be far less efficient than most any interval-based data structure.

You'd have to allow userspace to upper bound the number of intervals (similar to the maximum bitmap size), to prevent host OOMs due to malicious guests. There's something nice about the guest donating memory for this, since that would eliminate the OOM risk.

Tracking the list of ranges may not be bad idea, especially if we use the some kind of rbtree-based data structure to update the ranges. It will certainly be better than bitmap which grows based on the guest memory size and as you guys see in the practice most of the pages will be guest private. I am not sure if guest donating a memory will cover all the cases, e.g what if we do a memory hotplug (increase the guest ram from 2GB to 64GB), will donated memory range will be enough to store the metadata.

.

With reference to internal discussions regarding the above, i am going to look into specific items as listed below :

1). "hypercall" related :
a). Explore the SEV-SNP page change request structure (included in GHCB), see if there is something common there than can be re-used for SEV/SEV-ES page encryption status hypercalls.
b). Explore if there is any common hypercall framework i can use in Linux/KVM.

2). related to the "backing" data structure - explore using a range-based list or something like rbtree-based interval tree data structure (as mentioned by Steve above) to replace the current bitmap based implementation.

I do agree that a range-based list or an interval tree data structure is a really good "logical" fit for the guest page encryption status tracking.

We can only keep track of the guest unencrypted shared pages in the range(s) list (which will keep the data structure quite compact) and all the guest private/encrypted memory does not really need any tracking in the list, anything not in the list will be encrypted/private.

Also looking at a more "practical" use case, here is the current log of page encryption status hypercalls when booting a linux guest :

...

<snip>

[   56.146336] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.146351] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.147261] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.147271] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0

....

[   56.180730] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.180741] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.180768] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.180782] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1

....

[   56.197110] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.197120] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.197136] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.197148] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1

....

[   56.222679] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.222691] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.222707] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.222720] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1

....

[   56.313747] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.313771] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.313789] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.313803] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
....
[   56.459276] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 0
[   56.459428] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 1
[   56.460037] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 1
[   56.460216] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 0
[   56.460299] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 0
[   56.460448] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 1

....

As can be observed here, all guest MMIO ranges are initially setup as shared, and those are all contigious guest page ranges.

After that the encryption status hypercalls are invoked when DMA gets triggered during disk i/o while booting the guest ... here again the guest page ranges are contigious, though mostly single page is touched and a lot of page re-use is observed.

So a range-based list/structure will be a "good" fit for such usage scenarios.

It seems surprisingly common to flick the same pages back and forth between encrypted and clear for quite a while; why is this?

Quoted 6 lines

dma_alloc_coherent()'s will allocate pages and then call set_decrypted() on them and then at dma_free_coherent(), set_encrypted() is called on the pages to be freed. So these observations in the logs where a lot of single 4K pages are seeing C-bit transitions and corresponding hypercalls are the ones associated with dma_alloc_coherent().

Quoted 5 lines

It makes me wonder if it might be worth teaching it to hold onto those DMA pages somewhere until it needs them for something else and avoid the extra hypercalls; just something to think about.

Dave

Following up on this discussion and looking at the hypercall logs and DMA usage scenarios on the SEV, I have the following additional observations and comments :

It is mostly the Guest MMIO regions setup as un-encrypted by uefi/edk2 initially, which will be the "static" nodes in the backing data structure for page encryption status. These will be like 15-20 nodes/entries.

Drivers doing DMA allocations using GFP_ATOMIC will be fetching DMA buffers from the pre-allocated unencrypted atomic pool, hence it will be a "static" node added at kernel startup.

As we see with the logs, almost all runtime C-bit transitions and corresponding hypercalls will be from DMA I/O and dma_alloc_coherent/dma_free_coherent calls, these will be using 4K/single pages and mostly fragmented ranges, so if we use a "rbtree" based interval tree then there will be a lot of tree insertions and deletions (dma_alloc_coherent followed with a dma_free_coherent), so this will lead to a lot of expensive tree rotations and re-balancing, compared to much less complex and faster linked list node insertions and deletions (if we use a list based structure to represent these interval ranges).

Also as the static nodes in the structure will be quite limited (all the above DMA I/O added ranges will simply be inserted and removed), so a linked list lookup won't be too expensive compared to a tree lookup. In other words, this be a fixed size list.

Looking at the above, I am now more inclined to use a list based structure to represent the page encryption status.

Looking fwd. to any comments/feedback/thoughts on the above.

Thanks,
Ashish

#20 lore raw

Re: [PATCH v2 1/9] KVM: x86: Add AMD SEV specific Hypercall3

In reply to Steve Rutherford

Hello Steve,

Quoted 7 lines · Steve Rutherford

Avoiding an rbtree for such a small (but unstable) list seems correct.

For the unencrypted region list strategy, the only questions that I have are fairly secondary.
- How should the kernel upper bound the size of the list in the face of malicious guests, but still support large guests? (Something similar to the size provided in the bitmap API would work).

I am thinking of another scenario, where a malicious guest can make infinite/repetetive hypercalls and DOS attack the host.

But probably this is a more generic issue, this can be done by any guest and under any hypervisor, i don't know what kind of mitigations exist for such a scenario ?

Potentially, the guest memory donation model can handle such an attack, because in this model, the hypervisor will expect only one hypercall, any repetetive hypercalls can make the hypervisor disable the guest ?

Thanks,
Ashish

Quoted 235 lines

- What serialization format should be used for the ioctl API? (Usermode could send down a pointer to a user region and a size. The kernel could then populate that with an array of structs containing bases and limits for unencrypted regions.)
- How will the kernel tag a guest as having exceeded its maximum list size, in order to indicate that the list is now incomplete? (Track a poison bit, and send it up when getting the serialized list of regions).

In my view, there are two main competitors to this strategy:
- (Existing) Bitmap API
- A guest memory donation based model

The existing bitmap API avoids any issues with growing too large, since it's size is predictable.

To elaborate on the memory donation based model, the guest could put an encryption status data structure into unencrypted guest memory, and then use a hypercall to inform the host where the base of that structure is located. The main advantage of this is that it side steps any issues around malicious guests causing large allocations.

The unencrypted region list seems very practical. It's biggest advantage over the bitmap is how cheap it will be to pass the structure up from the kernel. A memory donation based model could achieve similar performance, but with some additional complexity.

Does anyone view the memory donation model as worth the complexity? Does anyone think the simplicity of the bitmap is a better tradeoff compared to an unencrypted region list?
Or have other ideas that are not mentioned here?

Quoted 201 lines · Ashish Kalra
Quoted 171 lines · Dr. David Alan Gilbert
Quoted 163 lines · * Kalra, Ashish ([email protected]) wrote:

Hello Dave,

On Dec 18, 2020, at 1:40 PM, Dr. David Alan Gilbert <[email protected]> wrote:

* Ashish Kalra ([email protected]) wrote:
On Fri, Dec 11, 2020 at 10:55:42PM +0000, Ashish Kalra wrote:
Hello All,

On Tue, Dec 08, 2020 at 10:29:05AM -0600, Brijesh Singh wrote:

On 12/7/20 9:09 PM, Steve Rutherford wrote:
On Mon, Dec 7, 2020 at 12:42 PM Sean Christopherson <[email protected]> wrote:
On Sun, Dec 06, 2020, Paolo Bonzini wrote:
On 03/12/20 01:34, Sean Christopherson wrote:
On Tue, Dec 01, 2020, Ashish Kalra wrote:
From: Brijesh Singh <[email protected]>

KVM hypercall framework relies on alternative framework to patch the VMCALL -> VMMCALL on AMD platform. If a hypercall is made before apply_alternative() is called then it defaults to VMCALL. The approach works fine on non SEV guest. A VMCALL would causes #UD, and hypervisor will be able to decode the instruction and do the right things. But when SEV is active, guest memory is encrypted with guest key and hypervisor will not be able to decode the instruction bytes.

Add SEV specific hypercall3, it unconditionally uses VMMCALL. The hypercall will be used by the SEV guest to notify encrypted pages to the hypervisor. What if we invert KVM_HYPERCALL and X86_FEATURE_VMMCALL to default to VMMCALL and opt into VMCALL? It's a synthetic feature flag either way, and I don't think there are any existing KVM hypercalls that happen before alternatives are patched, i.e. it'll be a nop for sane kernel builds.

I'm also skeptical that a KVM specific hypercall is the right approach for the encryption behavior, but I'll take that up in the patches later in the series. Do you think that it's the guest that should "donate" memory for the bitmap instead?
No. Two things I'd like to explore:

1. Making the hypercall to announce/request private vs. shared common across hypervisors (KVM, Hyper-V, VMware, etc...) and technologies (SEV-* and TDX). I'm concerned that we'll end up with multiple hypercalls that do more or less the same thing, e.g. KVM+SEV, Hyper-V+SEV, TDX, etc... Maybe it's a pipe dream, but I'd like to at least explore options before shoving in KVM- only hypercalls.

2. Tracking shared memory via a list of ranges instead of a using bitmap to track all of guest memory. For most use cases, the vast majority of guest memory will be private, most ranges will be 2mb+, and conversions between private and shared will be uncommon events, i.e. the overhead to walk and split/merge list entries is hopefully not a big concern. I suspect a list would consume far less memory, hopefully without impacting performance.
For a fancier data structure, I'd suggest an interval tree. Linux already has an rbtree-based interval tree implementation, which would likely work, and would probably assuage any performance concerns.

Something like this would not be worth doing unless most of the shared pages were physically contiguous. A sample Ubuntu 20.04 VM on GCP had 60ish discontiguous shared regions. This is by no means a thorough search, but it's suggestive. If this is typical, then the bitmap would be far less efficient than most any interval-based data structure.

You'd have to allow userspace to upper bound the number of intervals (similar to the maximum bitmap size), to prevent host OOMs due to malicious guests. There's something nice about the guest donating memory for this, since that would eliminate the OOM risk.

Tracking the list of ranges may not be bad idea, especially if we use the some kind of rbtree-based data structure to update the ranges. It will certainly be better than bitmap which grows based on the guest memory size and as you guys see in the practice most of the pages will be guest private. I am not sure if guest donating a memory will cover all the cases, e.g what if we do a memory hotplug (increase the guest ram from 2GB to 64GB), will donated memory range will be enough to store the metadata.

.

With reference to internal discussions regarding the above, i am going to look into specific items as listed below :

1). "hypercall" related :
a). Explore the SEV-SNP page change request structure (included in GHCB), see if there is something common there than can be re-used for SEV/SEV-ES page encryption status hypercalls.
b). Explore if there is any common hypercall framework i can use in Linux/KVM.

2). related to the "backing" data structure - explore using a range-based list or something like rbtree-based interval tree data structure (as mentioned by Steve above) to replace the current bitmap based implementation.

I do agree that a range-based list or an interval tree data structure is a really good "logical" fit for the guest page encryption status tracking.

We can only keep track of the guest unencrypted shared pages in the range(s) list (which will keep the data structure quite compact) and all the guest private/encrypted memory does not really need any tracking in the list, anything not in the list will be encrypted/private.

Also looking at a more "practical" use case, here is the current log of page encryption status hypercalls when booting a linux guest :

...

<snip>

[   56.146336] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 1
[   56.146351] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.147261] page_enc_status_hc invoked, gpa = 1f00e000, npages  = 1, enc = 0
[   56.147271] page_enc_status_hc invoked, gpa = 1f018000, npages  = 1, enc = 0

....

[   56.180730] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 0
[   56.180741] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 0
[   56.180768] page_enc_status_hc invoked, gpa = 1f008000, npages  = 1, enc = 1
[   56.180782] page_enc_status_hc invoked, gpa = 1f006000, npages  = 1, enc = 1

....

[   56.197110] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 0
[   56.197120] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 0
[   56.197136] page_enc_status_hc invoked, gpa = 1f007000, npages  = 1, enc = 1
[   56.197148] page_enc_status_hc invoked, gpa = 1f005000, npages  = 1, enc = 1

....

[   56.222679] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 0
[   56.222691] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 0
[   56.222707] page_enc_status_hc invoked, gpa = 1e83b000, npages  = 1, enc = 1
[   56.222720] page_enc_status_hc invoked, gpa = 1e839000, npages  = 1, enc = 1

....

[   56.313747] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 0
[   56.313771] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 0
[   56.313789] page_enc_status_hc invoked, gpa = 1e5eb000, npages  = 1, enc = 1
[   56.313803] page_enc_status_hc invoked, gpa = 1e5e9000, npages  = 1, enc = 1
....
[   56.459276] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 0
[   56.459428] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 1
[   56.460037] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 1
[   56.460216] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 0
[   56.460299] page_enc_status_hc invoked, gpa = 1d767000, npages  = 100, enc = 0
[   56.460448] page_enc_status_hc invoked, gpa = 1e501000, npages  = 1, enc = 1

....

As can be observed here, all guest MMIO ranges are initially setup as shared, and those are all contigious guest page ranges.

After that the encryption status hypercalls are invoked when DMA gets triggered during disk i/o while booting the guest ... here again the guest page ranges are contigious, though mostly single page is touched and a lot of page re-use is observed.

So a range-based list/structure will be a "good" fit for such usage scenarios.

It seems surprisingly common to flick the same pages back and forth between encrypted and clear for quite a while; why is this?

Quoted 6 lines

dma_alloc_coherent()'s will allocate pages and then call set_decrypted() on them and then at dma_free_coherent(), set_encrypted() is called on the pages to be freed. So these observations in the logs where a lot of single 4K pages are seeing C-bit transitions and corresponding hypercalls are the ones associated with dma_alloc_coherent().

Quoted 5 lines

It makes me wonder if it might be worth teaching it to hold onto those DMA pages somewhere until it needs them for something else and avoid the extra hypercalls; just something to think about.

Dave

Following up on this discussion and looking at the hypercall logs and DMA usage scenarios on the SEV, I have the following additional observations and comments :

It is mostly the Guest MMIO regions setup as un-encrypted by uefi/edk2 initially, which will be the "static" nodes in the backing data structure for page encryption status. These will be like 15-20 nodes/entries.

Drivers doing DMA allocations using GFP_ATOMIC will be fetching DMA buffers from the pre-allocated unencrypted atomic pool, hence it will be a "static" node added at kernel startup.

As we see with the logs, almost all runtime C-bit transitions and corresponding hypercalls will be from DMA I/O and dma_alloc_coherent/dma_free_coherent calls, these will be using 4K/single pages and mostly fragmented ranges, so if we use a "rbtree" based interval tree then there will be a lot of tree insertions and deletions (dma_alloc_coherent followed with a dma_free_coherent), so this will lead to a lot of expensive tree rotations and re-balancing, compared to much less complex and faster linked list node insertions and deletions (if we use a list based structure to represent these interval ranges).

Also as the static nodes in the structure will be quite limited (all the above DMA I/O added ranges will simply be inserted and removed), so a linked list lookup won't be too expensive compared to a tree lookup. In other words, this be a fixed size list.

Looking at the above, I am now more inclined to use a list based structure to represent the page encryption status.

Looking fwd. to any comments/feedback/thoughts on the above.

Thanks,
Ashish

#21 lore raw

Re: [PATCH v2 1/9] KVM: x86: Add AMD SEV specific Hypercall3

Quoted 14 lines · Ashish Kalra

Hello Steve,

Quoted 7 lines · Steve Rutherford

Avoiding an rbtree for such a small (but unstable) list seems correct.

For the unencrypted region list strategy, the only questions that I have are fairly secondary.
- How should the kernel upper bound the size of the list in the face of malicious guests, but still support large guests? (Something similar to the size provided in the bitmap API would work).

I am thinking of another scenario, where a malicious guest can make infinite/repetetive hypercalls and DOS attack the host.

Quoted 7 lines

But probably this is a more generic issue, this can be done by any guest and under any hypervisor, i don't know what kind of mitigations exist for such a scenario ?

Potentially, the guest memory donation model can handle such an attack, because in this model, the hypervisor will expect only one hypercall, any repetetive hypercalls can make the hypervisor disable the guest ?

KVM doesn't need to explicitly bound its tracking structures, it just needs to use GFP_KERNEL_ACCOUNT when allocating kernel memory for the structures so that the memory will be accounted to the task/process/VM. Shadow MMU pages are the only exception that comes to mind; they're still accounted properly, but KVM also explicitly limits them for a variety of reasons.

The size of the list will naturally be bounded by the size of the guest; and assuming KVM proactively merges adjancent regions, that upper bound is probably reasonably low compared to other allocations, e.g. the aforementioned MMU pages.

And, using a list means a malicious guest will get automatically throttled as the latency of walking the list (to merge/delete existing entries) will increase with the size of the list.

#22 lore raw

Re: [PATCH v2 1/9] KVM: x86: Add AMD SEV specific Hypercall3

Quoted 29 lines · Sean Christopherson
Quoted 14 lines · Ashish Kalra

Hello Steve,

Quoted 7 lines · Steve Rutherford

Avoiding an rbtree for such a small (but unstable) list seems correct.

For the unencrypted region list strategy, the only questions that I have are fairly secondary.
- How should the kernel upper bound the size of the list in the face of malicious guests, but still support large guests? (Something similar to the size provided in the bitmap API would work).

I am thinking of another scenario, where a malicious guest can make infinite/repetetive hypercalls and DOS attack the host.

Quoted 7 lines

But probably this is a more generic issue, this can be done by any guest and under any hypervisor, i don't know what kind of mitigations exist for such a scenario ?

Potentially, the guest memory donation model can handle such an attack, because in this model, the hypervisor will expect only one hypercall, any repetetive hypercalls can make the hypervisor disable the guest ?

KVM doesn't need to explicitly bound its tracking structures, it just needs to use GFP_KERNEL_ACCOUNT when allocating kernel memory for the structures so that the memory will be accounted to the task/process/VM. Shadow MMU pages are the only exception that comes to mind; they're still accounted properly, but KVM also explicitly limits them for a variety of reasons.

Quoted 7 lines

The size of the list will naturally be bounded by the size of the guest; and assuming KVM proactively merges adjancent regions, that upper bound is probably reasonably low compared to other allocations, e.g. the aforementioned MMU pages.

And, using a list means a malicious guest will get automatically throttled as the latency of walking the list (to merge/delete existing entries) will increase with the size of the list.

Just to add here, potentially there won't be any proactive merging/deletion of existing entries, as the only static entries will be initial guest MMIO regions, which are contigious guest PA ranges but not necessarily adjacent.

After that, as discussed before, almost all entries will be due to DMA I/O with respect to dma_alloc_coherent/dma_free_coherent, and all these entries will be temporary as these DMA buffers are marked un-encrypted and immediately marked encrypted as soon as DMA I/O is completed, so it makes no sense to do merging of temporary entries that will be deleted from the list immediately after being added to it.

Thanks,
Ashish

#23 lore raw

Re: [PATCH v2 1/9] KVM: x86: Add AMD SEV specific Hypercall3

Quoted 34 lines · Ashish Kalra
Quoted 33 lines · Sean Christopherson
Quoted 14 lines · Ashish Kalra

Hello Steve,

Quoted 7 lines · Steve Rutherford

Avoiding an rbtree for such a small (but unstable) list seems correct.

For the unencrypted region list strategy, the only questions that I have are fairly secondary.
- How should the kernel upper bound the size of the list in the face of malicious guests, but still support large guests? (Something similar to the size provided in the bitmap API would work).

I am thinking of another scenario, where a malicious guest can make infinite/repetetive hypercalls and DOS attack the host.

Quoted 7 lines

But probably this is a more generic issue, this can be done by any guest and under any hypervisor, i don't know what kind of mitigations exist for such a scenario ?

Potentially, the guest memory donation model can handle such an attack, because in this model, the hypervisor will expect only one hypercall, any repetetive hypercalls can make the hypervisor disable the guest ?

KVM doesn't need to explicitly bound its tracking structures, it just needs to use GFP_KERNEL_ACCOUNT when allocating kernel memory for the structures so that the memory will be accounted to the task/process/VM. Shadow MMU pages are the only exception that comes to mind; they're still accounted properly, but KVM also explicitly limits them for a variety of reasons.

The size of the list will naturally be bounded by the size of the guest; and assuming KVM proactively merges adjancent regions, that upper bound is probably reasonably low compared to other allocations, e.g. the aforementioned MMU pages.

Quoted 8 lines
Quoted 3 lines

And, using a list means a malicious guest will get automatically throttled as the latency of walking the list (to merge/delete existing entries) will increase with the size of the list.

Just to add here, potentially there won't be any proactive merging/deletion of existing entries, as the only static entries will be initial guest MMIO regions, which are contigious guest PA ranges but not necessarily adjacent.

My point was that, if the guest is malicious, eventually there will be adjacent entries, e.g. the worst case scenario is that the encrypted status changes on every 4k page. Anyways, not really all that important, I mostly thinking out loud :-)

#18 lore raw

Re: [PATCH v2 1/9] KVM: x86: Add AMD SEV specific Hypercall3

In reply to Ashish Kalra
Quoted 54 lines · Ashish Kalra
Quoted 53 lines

Hello Steve,

My thoughts here ...

Quoted 2 lines · Steve Rutherford

Avoiding an rbtree for such a small (but unstable) list seems correct.

I agree.

Quoted 27 lines

For the unencrypted region list strategy, the only questions that I have are fairly secondary.
- How should the kernel upper bound the size of the list in the face of malicious guests, but still support large guests? (Something similar to the size provided in the bitmap API would work).
- What serialization format should be used for the ioctl API? (Usermode could send down a pointer to a user region and a size. The kernel could then populate that with an array of structs containing bases and limits for unencrypted regions.)
- How will the kernel tag a guest as having exceeded its maximum list size, in order to indicate that the list is now incomplete? (Track a poison bit, and send it up when getting the serialized list of regions).

In my view, there are two main competitors to this strategy:
- (Existing) Bitmap API
- A guest memory donation based model

The existing bitmap API avoids any issues with growing too large, since it's size is predictable.

To elaborate on the memory donation based model, the guest could put an encryption status data structure into unencrypted guest memory, and then use a hypercall to inform the host where the base of that structure is located. The main advantage of this is that it side steps any issues around malicious guests causing large allocations.

Quoted 8 lines

The unencrypted region list seems very practical. It's biggest advantage over the bitmap is how cheap it will be to pass the structure up from the kernel. A memory donation based model could achieve similar performance, but with some additional complexity.

Does anyone view the memory donation model as worth the complexity? Does anyone think the simplicity of the bitmap is a better tradeoff compared to an unencrypted region list?

One advantage in sticking with the bitmap is that it maps very nicely to the dirty bitmap page tracking logic in KVM/Qemu. The way Brijesh designed and implemented it is very similar to dirty page bitmap tracking and syncing between KVM and Qemu. The same logic is re-used for the page encryption bitmap which means quite mininal changes and code resuse in Qemu.

Quoted 5 lines
Quoted 5 lines

Any changes to the backing data structure, will require additional mapping logic to be added to Qemu.

This is one advantage in keeping the bitmap logic.

So if nobody is in favor of keeping the (current) bitmap logic, we will move to the unencrypted region list approach.

Thanks,
Ashish

#24 lore raw

Re: [PATCH v2 1/9] KVM: x86: Add AMD SEV specific Hypercall3

In reply to Sean Christopherson

Supporting merging of consecutive entries (or not) is less important to get right since it doesn't change any of the APIs. If someone runs into performance issues, they can loop back and fix this then. I'm slightly concerned with the behavior for overlapping regions. I also have slight concerns with how we handle re-encrypting small chunks of larger unencrypted regions. I don't think we've seen these in practice, but nothing precludes them afaik.

Quoted 45 lines · Sean Christopherson
Quoted 34 lines · Ashish Kalra
Quoted 33 lines · Sean Christopherson
Quoted 14 lines · Ashish Kalra

Hello Steve,

Quoted 7 lines · Steve Rutherford

Avoiding an rbtree for such a small (but unstable) list seems correct.

For the unencrypted region list strategy, the only questions that I have are fairly secondary.
- How should the kernel upper bound the size of the list in the face of malicious guests, but still support large guests? (Something similar to the size provided in the bitmap API would work).

I am thinking of another scenario, where a malicious guest can make infinite/repetetive hypercalls and DOS attack the host.

Quoted 7 lines

But probably this is a more generic issue, this can be done by any guest and under any hypervisor, i don't know what kind of mitigations exist for such a scenario ?

Potentially, the guest memory donation model can handle such an attack, because in this model, the hypervisor will expect only one hypercall, any repetetive hypercalls can make the hypervisor disable the guest ?

KVM doesn't need to explicitly bound its tracking structures, it just needs to use GFP_KERNEL_ACCOUNT when allocating kernel memory for the structures so that the memory will be accounted to the task/process/VM. Shadow MMU pages are the only exception that comes to mind; they're still accounted properly, but KVM also explicitly limits them for a variety of reasons.

The size of the list will naturally be bounded by the size of the guest; and assuming KVM proactively merges adjancent regions, that upper bound is probably reasonably low compared to other allocations, e.g. the aforementioned MMU pages.

Quoted 8 lines
Quoted 3 lines

And, using a list means a malicious guest will get automatically throttled as the latency of walking the list (to merge/delete existing entries) will increase with the size of the list.

Just to add here, potentially there won't be any proactive merging/deletion of existing entries, as the only static entries will be initial guest MMIO regions, which are contigious guest PA ranges but not necessarily adjacent.

Quoted 4 lines

My point was that, if the guest is malicious, eventually there will be adjacent entries, e.g. the worst case scenario is that the encrypted status changes on every 4k page. Anyways, not really all that important, I mostly thinking out loud :-)

Agreed. Tagging this with GFP_KERNEL_ACCOUNT means we don't need to upper bound the number of pages. I now don't think there is any unusual DoS potential here. Perhaps, if the guest tries really hard to make a massive list, they could get a softlockup on the host. Not sure how important that is to fix.

#19 lore raw

Re: [PATCH v2 1/9] KVM: x86: Add AMD SEV specific Hypercall3

In reply to Ashish Kalra
Quoted 55 lines · Ashish Kalra
Quoted 55 lines · Ashish Kalra
Quoted 53 lines

Hello Steve,

My thoughts here ...

Quoted 2 lines · Steve Rutherford

Avoiding an rbtree for such a small (but unstable) list seems correct.

I agree.

Quoted 27 lines

For the unencrypted region list strategy, the only questions that I have are fairly secondary.
- How should the kernel upper bound the size of the list in the face of malicious guests, but still support large guests? (Something similar to the size provided in the bitmap API would work).
- What serialization format should be used for the ioctl API? (Usermode could send down a pointer to a user region and a size. The kernel could then populate that with an array of structs containing bases and limits for unencrypted regions.)
- How will the kernel tag a guest as having exceeded its maximum list size, in order to indicate that the list is now incomplete? (Track a poison bit, and send it up when getting the serialized list of regions).

In my view, there are two main competitors to this strategy:
- (Existing) Bitmap API
- A guest memory donation based model

The existing bitmap API avoids any issues with growing too large, since it's size is predictable.

To elaborate on the memory donation based model, the guest could put an encryption status data structure into unencrypted guest memory, and then use a hypercall to inform the host where the base of that structure is located. The main advantage of this is that it side steps any issues around malicious guests causing large allocations.

Quoted 8 lines

The unencrypted region list seems very practical. It's biggest advantage over the bitmap is how cheap it will be to pass the structure up from the kernel. A memory donation based model could achieve similar performance, but with some additional complexity.

Does anyone view the memory donation model as worth the complexity? Does anyone think the simplicity of the bitmap is a better tradeoff compared to an unencrypted region list?

One advantage in sticking with the bitmap is that it maps very nicely to the dirty bitmap page tracking logic in KVM/Qemu. The way Brijesh designed and implemented it is very similar to dirty page bitmap tracking and syncing between KVM and Qemu. The same logic is re-used for the page encryption bitmap which means quite mininal changes and code resuse in Qemu.

Quoted 8 lines
Quoted 5 lines
Quoted 5 lines

Any changes to the backing data structure, will require additional mapping logic to be added to Qemu.

This is one advantage in keeping the bitmap logic.

So if nobody is in favor of keeping the (current) bitmap logic, we will move to the unencrypted region list approach.

Sounds good to me.

Quoted 3 lines

Thanks,
Ashish

#25 lore raw

Re: [PATCH v2 1/9] KVM: x86: Add AMD SEV specific Hypercall3

In reply to Steve Rutherford
Quoted 4 lines · Steve Rutherford

Supporting merging of consecutive entries (or not) is less important to get right since it doesn't change any of the APIs. If someone runs into performance issues, they can loop back and fix this then. I'm slightly concerned with the behavior for overlapping regions.

I'm assuming merging entries will be a near-trivial effort once everything else is implemented, e.g. KVM will need to traverse the list to remove entries when address are converted back to private. Piling on merging functionality at that point should not be all that hard, especially if the list is sorted and entries are merged on insertion.