Skip to contentSkip to conversation

vfio-pci/migration: Dirty logging of the Memory BAR region?

RFC In review Zenghui Yu 2 participants 3 messages Nov 15, 2020 – Nov 16, 2020 KVM arm64

lore
Share & tools
lore

Zenghui Yu

author reviewer
#0 lore raw

RFC 1/1 vfio-pci/migration: Dirty logging of the Memory BAR region?

Hi folks,

While trying the new vfio-pci migration on my arm64 server, I noticed an error at the very beginning:

qemu-system-aarch64: kvm_set_user_memory_region: KVM_SET_USER_MEMORY_REGION failed, slot=5, start=0x8000000000, size=0x100000: Invalid argument

The reason is that we've registered the Memory BAR region as ram device region (mr->ram_device is set) and tried to track dirty pages of this region during migration. QEMU tries to request tracking of it (via kvm memory listener's log_start() callback) whilst KVM/arm64 clearly refuses to do so [1]:

Quoted 8 lines

int kvm_arch_prepare_memory_region(struct kvm *kvm, ...)

{
		/* IO region dirty page logging not allowed */
		if (memslot->flags & KVM_MEM_LOG_DIRTY_PAGES) {
			ret = -EINVAL;
			goto out;
		}
}

If I understand things correctly, the Memory BAR region generally contains internal device-specific registers which shouldn't (and can't) be tracked by QEMU's dirty logging mechanism. I've patched QEMU with the following diff and it seems work for me, but all of these still require for comments.

softmmu/memory.c +4 -1
index 71951fe4dc..0958db1a08 100644
@@ -1806,7 +1806,10 @@ bool memory_region_is_ram_device(MemoryRegion *mr)
 uint8_t memory_region_get_dirty_log_mask(MemoryRegion *mr)
 {
     uint8_t mask = mr->dirty_log_mask;
-    if (global_dirty_log && (mr->ram_block || memory_region_is_iommu(mr))) {
+    RAMBlock *rb = mr->ram_block;
+
+    if (global_dirty_log && ((rb && qemu_ram_is_migratable(rb)) ||
+                             memory_region_is_iommu(mr))) {
         mask |= (1 << DIRTY_MEMORY_MIGRATION);
     }
     return mask;

Thanks,
Zenghui

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/kvm/mmu.c
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

#1 lore raw
Quoted 16 lines · Zenghui Yu
softmmu/memory.c +4 -1
index 71951fe4dc..0958db1a08 100644
@@ -1806,7 +1806,10 @@ bool memory_region_is_ram_device(MemoryRegion *mr)
  uint8_t memory_region_get_dirty_log_mask(MemoryRegion *mr)
  {
      uint8_t mask = mr->dirty_log_mask;
-    if (global_dirty_log && (mr->ram_block || 
 memory_region_is_iommu(mr))) {
+    RAMBlock *rb = mr->ram_block;
+
+    if (global_dirty_log && ((rb && qemu_ram_is_migratable(rb)) ||
+                             memory_region_is_iommu(mr))) {
          mask |= (1 << DIRTY_MEMORY_MIGRATION);
      }
Quoted 1 line

     return mask;

Yes, this makes sense. Please send it as a patch, thanks!

Paolo

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

author reviewer
#2 lore raw
Quoted 12 lines · Paolo Bonzini
Quoted 11 lines · Zenghui Yu
softmmu/memory.c +1 -1
index 71951fe4dc..0958db1a08 100644
@@ -1806,7 +1806,10 @@ bool memory_region_is_ram_device(MemoryRegion *mr)
 uint8_t memory_region_get_dirty_log_mask(MemoryRegion *mr)
 {
     uint8_t mask = mr->dirty_log_mask;
-    if (global_dirty_log && (mr->ram_block || 
 memory_region_is_iommu(mr))) {
+    RAMBlock *rb = mr->ram_block;
Quoted 8 lines
Quoted 6 lines
softmmu/memory.c +3 -0
+
+    if (global_dirty_log && ((rb && qemu_ram_is_migratable(rb)) ||
+                             memory_region_is_iommu(mr))) {
         mask |= (1 << DIRTY_MEMORY_MIGRATION);
     }
     return mask;

Yes, this makes sense. Please send it as a patch, thanks!

Sure, I'm going to write a commit message for it. Thanks for your review.

Zenghui
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm