Moving dirty bitmaps to userspace - Double buffering approach
Hi, I would like to hear your comments about the following plan:
Moving dirty bitmaps to userspace
- Double buffering approach
especially I would be glad if I can hear some advice about how to keep the compatibility.
Thanks in advance,
Takuya
---
Overview:
Last time, I submitted a patch
"make get dirty log ioctl return the first dirty page's position"
http://www.spinics.net/lists/kvm/msg29724.html
and got some new better ideas from Avi.
As a result, I agreed to try to eliminate the bitmap allocation done in the x86 KVM every time when we execute get dirty log by using "double buffering" approach.
Here is my plan:
- move the dirty bitmap allocation to userspace
We allocate bitmaps in the userspace and register them by ioctl. Once a bitmap is registered, we do not touch it from userspace and let the kernel modify it directly until we switch to the next bitmap. We use "double buffering" at this switch point: userspace give the kernel a new bitmap by ioctl and the kernel switch the bitmap atomically to new one.
After succeeded in this switch, we can read the old bitmap freely in the userspace and free it if we want: needless to say we can also reuse it at the next switch.
- implementation details
Although it may be possible to touch the bitmap from the kernel side without doing kmap, I think kmapping the bitmap is better. So we may use the following functions paying enough attention to the preemption control.
- get_user_pages()
- kmap_atomic()
- compatibility issues
What I am facing now are the compatibility issues. We have to support both the userspace and kernel side bitmap allocations to let the current qemu and KVM work properly.
1. From the kernel side, we have to care bitmap allocations done in both the kvm_vm_ioctl_set_memory_region() and kvm_vm_ioctl_get_dirty_log().
2. From the userspace side, we have to check the new api's availability and determine which way we use, e.g. by using check extension ioctl.
The most problematic is 1, kernel side. We have to be able to know by which way current bitmap allocation is being done using flags or something. In the case of set memory region, we have to judge whether we allocate a bitmap, and if not we have to register a bitmap later by another api: set memory region is not restricted to the dirty log issues and need more care than get dirty log.
Are there any good ways to solve this kind of problems?
Quoted 25 lines · Takuya Yoshikawa
Hi, I would like to hear your comments about the following plan:
Moving dirty bitmaps to userspace
- Double buffering approachespecially I would be glad if I can hear some advice about how to keep the compatibility.
Thanks in advance,
Takuya---
Overview:Last time, I submitted a patch
"make get dirty log ioctl return the first dirty page's position"
http://www.spinics.net/lists/kvm/msg29724.html
and got some new better ideas from Avi.As a result, I agreed to try to eliminate the bitmap allocation done in the x86 KVM every time when we execute get dirty log by using "double buffering" approach.
[...]
Quoted 6 lines
Although it may be possible to touch the bitmap from the kernel side without doing kmap, I think kmapping the bitmap is better. So we may use the following functions paying enough attention to the preemption control.
-get_user_pages()
-kmap_atomic()
Although direct access is more difficult (you need to implement put_user_bit() or similar) I think it is worthwhile. get_user_pages_fast() is fast, but nowhere near as fast as put_user_bit() (or set_bit_user()), which can be just two instructions in the fast path.
Quoted 9 lines
- compatibility issues
What I am facing now are the compatibility issues. We have to support both the userspace and kernel side bitmap allocations to let the current qemu and KVM work properly.
1. From the kernel side, we have to care bitmap allocations done in both the
kvm_vm_ioctl_set_memory_region()andkvm_vm_ioctl_get_dirty_log().
One way to handle this is to call do_mmap() from the kernel, so that now the bitmap really lives in user space. This is a bit ugly but I think acceptable. We already do this for KVM_SET_MEMORY_REGION (which was replaced by KVM_SET_USER_MEMORY_REGION, which moved allocation to userspace).
Quoted 4 lines
2. From the userspace side, we have to check the new api's availability and determine which way we use, e.g. by using check extension ioctl.
Quoted 8 lines
The most problematic is 1, kernel side. We have to be able to know by which way current bitmap allocation is being done using flags or something. In the case of set memory region, we have to judge whether we allocate a bitmap, and if not we have to register a bitmap later by another api: set memory region is not restricted to the dirty log issues and need more care than get dirty log.
Are there any good ways to solve this kind of problems?
I believe that do_mmap() will simplify this.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 63 lines · Takuya Yoshikawa
Hi, I would like to hear your comments about the following plan:
Moving dirty bitmaps to userspace
- Double buffering approachespecially I would be glad if I can hear some advice about how to keep the compatibility.
Thanks in advance,
Takuya---
Overview:Last time, I submitted a patch
"make get dirty log ioctl return the first dirty page's position"
http://www.spinics.net/lists/kvm/msg29724.html
and got some new better ideas from Avi.As a result, I agreed to try to eliminate the bitmap allocation done in the x86 KVM every time when we execute get dirty log by using "double buffering" approach.
Here is my plan:
- move the dirty bitmap allocation to userspace
We allocate bitmaps in the userspace and register them by ioctl. Once a bitmap is registered, we do not touch it from userspace and let the kernel modify it directly until we switch to the next bitmap. We use "double buffering" at this switch point: userspace give the kernel a new bitmap by ioctl and the kernel switch the bitmap atomically to new one.
After succeeded in this switch, we can read the old bitmap freely in the userspace and free it if we want: needless to say we can also reuse it at the next switch.
- implementation details
Although it may be possible to touch the bitmap from the kernel side without doing kmap, I think kmapping the bitmap is better. So we may use the following functions paying enough attention to the preemption control.
-get_user_pages()
-kmap_atomic()- compatibility issues
What I am facing now are the compatibility issues. We have to support both the userspace and kernel side bitmap allocations to let the current qemu and KVM work properly.
1. From the kernel side, we have to care bitmap allocations done in both the
kvm_vm_ioctl_set_memory_region()andkvm_vm_ioctl_get_dirty_log().2. From the userspace side, we have to check the new api's availability and determine which way we use, e.g. by using check extension ioctl.
Quoted 8 lines
The most problematic is 1, kernel side. We have to be able to know by which way current bitmap allocation is being done using flags or something. In the case of set memory region, we have to judge whether we allocate a bitmap, and if not we have to register a bitmap later by another api: set memory region is not restricted to the dirty log issues and need more care than get dirty log.
Are there any good ways to solve this kind of problems?
You can introduce a new get_dirty_log ioctl that passes the address of the next bitmap in userspace, and use it (after pinning with get_user_pages), instead of vmalloc'ing.
Quoted 8 lines · Marcelo Tosatti
Quoted 3 lines
Are there any good ways to solve this kind of problems?
You can introduce a new
get_dirty_logioctl that passes the address of the next bitmap in userspace, and use it (after pinning withget_user_pages), instead of vmalloc'ing.
No pinning please, put_user_bit() or set_bit_user().
(can be implemented generically using get_user_pages() and kmap_atomic(), but x86 should get an optimized implementation)
Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
Quoted 8 lines · Avi Kivity
Quoted 7 lines · Marcelo Tosatti
Quoted 3 lines
Are there any good ways to solve this kind of problems?
You can introduce a new
get_dirty_logioctl that passes the address of the next bitmap in userspace, and use it (after pinning withget_user_pages), instead of vmalloc'ing.
Thank you for your advice!
Quoted 6 lines
Quoted 1 line
No pinning please,
put_user_bit()orset_bit_user().(can be implemented generically using
get_user_pages()andkmap_atomic(), but x86 should get an optimized implementation)
Given your advice last time, I started this with my colleague.
-- We were just talking about how to strugle with every architectures.
As your comment, we'll make the generic implementation with optimized one for x86 first.
Thanks
Takuya