dirty page tracking in kvm/qemu -- page faults inevitable?
Hi,
I've got an issue where we're hitting major performance penalties while doing live migration, and it seems like it might be due to page faults triggering hypervisor exits, and then we get stuck waiting for the iothread lock which is held by the qemu dirty page scanning code.
Accordingly, I'm trying to figure out the actual mechanism whereby dirty pages are tracked in qemu/kvm. I've got an Ivy Bridge CPU, a 3.4 kernel on the host, and qemu 1.4.
Looking at the qemu code, it seems to be calling down into kvm to get the dirty page information.
Looking at kvm, most of what I read seems to be doing the usual "mark it read-only and then when we take a page fault mark it as dirty" trick.
However, I read something about Intel EPT having hardware support for tracking dirty pages. It seems like this might avoid the need for a page fault, but might only be available on Haswell or later CPUs--is that correct? Is it supported in kvm? If so, when was support added?
Thanks,
Chris
P.S. Please CC me on reply, I'm not subscribed to the list.
Quoted 5 lines · Chris Friesen
Hi,
I've got an issue where we're hitting major performance penalties while doing live migration, and it seems like it might be due to page faults triggering hypervisor exits, and then we get stuck waiting for the iothread lock which is held by the qemu dirty page scanning code.
I am afraid that using dirty-bit instead of write-protection may cause the case even more worse for iothread-lock because we need to walk whole sptes to get dirty-set pages, however currently we only need to walk the page set in the bitmap.
Quoted 8 lines
Accordingly, I'm trying to figure out the actual mechanism whereby dirty pages are tracked in qemu/kvm. I've got an Ivy Bridge CPU, a 3.4 kernel on the host, and qemu 1.4.
Looking at the qemu code, it seems to be calling down into kvm to get the dirty page information.
Looking at kvm, most of what I read seems to be doing the usual "mark it read-only and then when we take a page fault mark it as dirty" trick.
However, I read something about Intel EPT having hardware support for tracking dirty pages. It seems like this might avoid the need for a page fault, but might only be available on Haswell or later CPUs--is that correct? Is it supported in kvm? If so, when was support added?
Actually, i have implemented the prototype long time ago, maybe it's the time to benchmark it and post it out.
Quoted 11 lines · Xiao Guangrong
Quoted 5 lines · Chris Friesen
Hi,
I've got an issue where we're hitting major performance penalties while doing live migration, and it seems like it might be due to page faults triggering hypervisor exits, and then we get stuck waiting for the iothread lock which is held by the qemu dirty page scanning code.
I am afraid that using dirty-bit instead of write-protection may cause the case even more worse for iothread-lock because we need to walk whole sptes to get dirty-set pages, however currently we only need to walk the page set in the bitmap.
I found a document at
"http://ftp.software-sources.co.il/Processor_Architecture_Update-Bob_Valentine.pdf" which talks about the benefits of Haswell. One of the items reads:
"New Accessed and Dirty bits for Extended Page Tables (EPT) eliminates major cause of vmexits"
Is that accurate? If so, then it seems like it should allow for the VM to run without trying to exit the hypervisor, and as long as it just does in-memory operations it won't contend on the iothread lock.
Chris
Paolo Bonzini
maintainerIl 30/07/2014 09:41, Chris Friesen ha scritto:
Quoted 9 lines
Quoted 4 lines
I am afraid that using dirty-bit instead of write-protection may cause the case even more worse for iothread-lock because we need to walk whole sptes to get dirty-set pages, however currently we only need to walk the page set in the bitmap.
I found a document at "http://ftp.software-sources.co.il/Processor_Architecture_Update-Bob_Valentine.pdf" which talks about the benefits of Haswell. One of the items reads:
Quoted 6 lines
"New Accessed and Dirty bits for Extended Page Tables (EPT) eliminates major cause of vmexits"
Is that accurate? If so, then it seems like it should allow for the VM to run without trying to exit the hypervisor, and as long as it just does in-memory operations it won't contend on the iothread lock.
True, but:
1) the problem is fishing the information out of the page tables and passing it up to userspace. You have to process the whole EPT tree one page at a time, instead of doing it 64 bits at a time. Also, one source of bad performance is having to split all entries of the EPT page tables down to 4K, and you get that anyway.
2) You should not get to userspace simply for marking a page as locked. As you describe it, your problem seems to be contention between QEMU
threads, KVM is not involved.
3) What version of QEMU are you using? Things have been improving steadily, and we probably will get to using atomic operations instead of the iothread lock to protect the migration dirty bitmap.
Paolo
Quoted 8 lines · Paolo Bonzini
Il 30/07/2014 09:41, Chris Friesen ha scritto:
Quoted 7 lines
I found a document at "http://ftp.software-sources.co.il/Processor_Architecture_Update-Bob_Valentine.pdf" which talks about the benefits of Haswell. One of the items reads:
"New Accessed and Dirty bits for Extended Page Tables (EPT) eliminates major cause of vmexits"
Quoted 7 lines
Quoted 3 lines
Is that accurate? If so, then it seems like it should allow for the VM to run without trying to exit the hypervisor, and as long as it just does in-memory operations it won't contend on the iothread lock.
2) You should not get to userspace simply for marking a page as locked. As you describe it, your problem seems to be contention between QEMU
threads, KVM is not involved.
What about writing to a page where we're tracking dirty pages? Would that get back up to qemu or would that be handled entirely in the kvm kernel module?
I was assuming that it was due to the page faults since as far as I know the app in the VM is just doing packet processing from/to memory-mapped circular buffers--the qemu threads in question aren't doing "normal" I/O but something is causing them to try to acquire the iothread lock.
Quoted 3 lines
3) What version of QEMU are you using? Things have been improving steadily, and we probably will get to using atomic operations instead of the iothread lock to protect the migration dirty bitmap.
We're currently on 1.4.2. We're looking at trying out 1.7 to see if it's better, but we've got some local patches that would need to get ported.
Chris
Paolo Bonzini
maintainerIl 30/07/2014 18:02, Chris Friesen ha scritto:
Quoted 14 lines · Paolo Bonzini
Quoted 12 lines
Il 30/07/2014 09:41, Chris Friesen ha scritto:
Quoted 11 lines
I found a document at "http://ftp.software-sources.co.il/Processor_Architecture_Update-Bob_Valentine.pdf"
which talks about the benefits of Haswell. One of the items reads:
"New Accessed and Dirty bits for Extended Page Tables (EPT) eliminates major cause of vmexits"
Is that accurate? If so, then it seems like it should allow for the VM to run without trying to exit the hypervisor, and as long as it just does in-memory operations it won't contend on the iothread lock.
Quoted 7 lines
Quoted 3 lines
2) You should not get to userspace simply for marking a page as locked. As you describe it, your problem seems to be contention between QEMU
threads, KVM is not involved.What about writing to a page where we're tracking dirty pages? Would that get back up to qemu or would that be handled entirely in the kvm kernel module?
It's handle inside the kernel module.
Every now and then QEMU asks the kernel for the dirty pages and ORs the bitmap returned by KVM with its own. All this is done under the iothread lock.
Quoted 9 lines
I was assuming that it was due to the page faults since as far as I know the app in the VM is just doing packet processing from/to memory-mapped circular buffers--the qemu threads in question aren't doing "normal" I/O but something is causing them to try to acquire the iothread lock.
Quoted 3 lines
3) What version of QEMU are you using? Things have been improving steadily, and we probably will get to using atomic operations instead of the iothread lock to protect the migration dirty bitmap.
Quoted 5 lines
We're currently on 1.4.2. We're looking at trying out 1.7 to see if it's better, but we've got some local patches that would need to get ported.
From a quick "git describe" 2.0 is needed. The patches end at commit
ae2810c (memory: syncronize kvm bitmap using bitmaps operations, 2013-11-05).
Paolo