vfio-pci/migration: Dirty logging of the Memory BAR region?
Zenghui Yu
reviewerRFC 1/1 vfio-pci/migration: Dirty logging of the Memory BAR region?
Hi folks,
While trying the new vfio-pci migration on my arm64 server, I noticed an error at the very beginning:
qemu-system-aarch64: kvm_set_user_memory_region: KVM_SET_USER_MEMORY_REGION failed, slot=5, start=0x8000000000, size=0x100000: Invalid argument
The reason is that we've registered the Memory BAR region as ram device region (mr->ram_device is set) and tried to track dirty pages of this region during migration. QEMU tries to request tracking of it (via kvm memory listener's log_start() callback) whilst KVM/arm64 clearly refuses to do so [1]:
Quoted 8 lines
int
kvm_arch_prepare_memory_region(struct kvm *kvm, ...){ /* IO region dirty page logging not allowed */ if (memslot->flags & KVM_MEM_LOG_DIRTY_PAGES) { ret = -EINVAL; goto out; } }
If I understand things correctly, the Memory BAR region generally contains internal device-specific registers which shouldn't (and can't) be tracked by QEMU's dirty logging mechanism. I've patched QEMU with the following diff and it seems work for me, but all of these still require for comments.
softmmu/memory.c +4 -1
@@ -1806,7 +1806,10 @@ bool memory_region_is_ram_device(MemoryRegion *mr)uint8_t memory_region_get_dirty_log_mask (MemoryRegion *mr) {uint8_t mask = mr->dirty_log_mask; -if (global_dirty_log && (mr->ram_block ||memory_region_is_iommu (mr))) { +RAMBlock *rb = mr->ram_block; + +if (global_dirty_log && ((rb &&qemu_ram_is_migratable (rb)) || +memory_region_is_iommu (mr))) { mask |= (1 <<DIRTY_MEMORY_MIGRATION ); }return mask;
Thanks,
Zenghui
[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/kvm/mmu.c
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Quoted 16 lines · Zenghui Yu
softmmu/memory.c +4
-1@@ -1806,7 +1806,10 @@ bool memory_region_is_ram_device(MemoryRegion *mr) uint8_t memory_region_get_dirty_log_mask(MemoryRegion *mr) { uint8_t mask = mr->dirty_log_mask; - if (global_dirty_log && (mr->ram_block || memory_region_is_iommu(mr))) { + RAMBlock *rb = mr->ram_block; + + if (global_dirty_log && ((rb && qemu_ram_is_migratable(rb)) || + memory_region_is_iommu(mr))) { mask |= (1 << DIRTY_MEMORY_MIGRATION); }
Quoted 1 line
return mask;
Yes, this makes sense. Please send it as a patch, thanks!
Paolo
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Zenghui Yu
reviewerQuoted 12 lines · Paolo Bonzini
Quoted 11 lines · Zenghui Yu
softmmu/memory.c +1
-1@@ -1806,7 +1806,10 @@ bool memory_region_is_ram_device(MemoryRegion *mr)uint8_t memory_region_get_dirty_log_mask (MemoryRegion *mr) {uint8_t mask = mr->dirty_log_mask; -if (global_dirty_log && (mr->ram_block || memory_region_is_iommu(mr))) { +RAMBlock *rb = mr->ram_block;
Quoted 8 lines
Quoted 6 lines
softmmu/memory.c +3
-0+ +if (global_dirty_log && ((rb &&qemu_ram_is_migratable (rb)) || +memory_region_is_iommu (mr))) { mask |= (1 <<DIRTY_MEMORY_MIGRATION ); }return mask;Yes, this makes sense. Please send it as a patch, thanks!
Sure, I'm going to write a commit message for it. Thanks for your review.
Zenghui
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm