Skip to content

RFC KVM, x86, ppc, asm-generic: moving dirty bitmaps to user space

Takuya Yoshikawa kvm kvm-ia64 kvm-ppc linux-arch linuxppc-dev lkml 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest12 patches 28 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/12KVM: x86: avoid unnecessary bitmap allocation when memslot is cleanarch/x86/kvm/x86.c+23 -140 replies
2/12KVM: introduce slot level dirty state managementarch/ia64/kvm/kvm-ia64.c arch/powerpc/kvm/book3s.c arch/x86/kvm/x86.c +2 more+19 -270 replies
3/12KVM: introduce wrapper functions to create and destroy dirty bitmapsvirt/kvm/kvm_main.c+20 -70 replies
4/12x86: introduce copy_in_user() for 32-bitarch/x86/include/asm/uaccess_32.h arch/x86/lib/usercopy_32.c+28 -00 replies
5/12x86: introduce __set_bit() like function for bitmaps in user spacearch/x86/include/asm/uaccess.h+39 -00 replies
6/12PPC: introduce copy_in_user() for 32-bitarch/powerpc/include/asm/uaccess.h+17 -00 replies
7/12PPC: introduce __set_bit() like function for bitmaps in user spacearch/powerpc/include/asm/uaccess.h+19 -02 replies
8/12asm-generic: bitops: introduce le bit offset macroinclude/asm-generic/bitops/le.h+4 -08 replies
9/12KVM: introduce a wrapper function of set_bit_user_non_atomic()arch/ia64/include/asm/kvm_host.h arch/powerpc/include/asm/kvm_host.h arch/s390/include/asm/kvm_host.h +1 more+22 -00 replies
10/12KVM: move dirty bitmaps to user spacearch/ia64/kvm/kvm-ia64.c arch/powerpc/kvm/book3s.c arch/x86/kvm/x86.c +2 more+82 -283 replies
11/12KVM: introduce new API for getting/switching dirty bitmapsDocumentation/kvm/api.txt arch/ia64/kvm/kvm-ia64.c arch/powerpc/kvm/book3s.c +4 more+220 -344 replies
12/12qemu-kvm: use new API for getting/switching dirty bitmapskvm/include/linux/kvm.h qemu-kvm.c qemu-kvm.h+85 -80 replies

Reviews trailers per patch

1 of 12 patches has a Reviewed-by or Acked-by.

PatchArnd Bergmann
1/12
2/12
3/12
4/12
5/12
6/12
7/12
8/12A-b
9/12
10/12
11/12
12/12

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v1 thread

Cover letter [RFC][PATCH 0/12] KVM, x86, ppc, asm-generic: moving dirty bitmaps to user space

Hi, sorry for sending from my personal account.
The following series are all from me:

From: Takuya Yoshikawa <[email protected]>

The 3rd version of "moving dirty bitmaps to user space".

Quoted 1 line

From this version, we add x86 and ppc and asm-generic people to CC lists.

[To KVM people]

Sorry for being late to reply your comments.

Avi,
- I've wrote an answer to your question in patch 5/12: drivers/vhost/vhost.c .

- I've considered to change the set_bit_user_non_atomic to an inline function, but did not change because the other helpers in the uaccess.h are written as macros. Anyway, I hope that x86 people will give us appropriate suggestions about this.

- I thought that documenting about making bitmaps 64-bit aligned will be written when we add an API to register user-allocated bitmaps. So probably in the next series.

Avi, Alex,
- Could you check the ia64 and ppc parts, please? I tried to keep the logical changes as small as possible.

I personally tried to build these with cross compilers. For ia64, I could check build success with my patch series. But book3s, even without my patch series, it failed with the following errors:

arch/powerpc/kvm/book3s_paired_singles.c: In function 'kvmppc_emulate_paired_single':
arch/powerpc/kvm/book3s_paired_singles.c:1289: error: the frame size of 2288 bytes is larger than 2048 bytes make[1]: *** [arch/powerpc/kvm/book3s_paired_singles.o] Error 1
make: *** [arch/powerpc/kvm] Error 2

About changelog: there are two main changes from the 2nd version:
1. I changed the treatment of clean slots (see patch 1/12). This was already applied today, thanks!
2. I changed the switch API. (see patch 11/12).

To show this API's advantage, I also did a test (see the end of this mail).

[To x86 people]

Hi, Thomas, Ingo, Peter,

Please review the patches 4,5/12. Because this is the first experience for me to send patches to x86, please tell me if this lacks anything.

[To ppc people]

Hi, Benjamin, Paul, Alex,

Please see the patches 6,7/12. I first say sorry for that I've not tested these yet. In that sense, these may not be in the quality for precise reviews. But I will be happy if you would give me any comments.

Alex, could you help me? Though I have a plan to get PPC box in the future, currently I cannot test these.

[To asm-generic people]

Hi, Arnd,

Please review the patch 8/12. This kind of macro is acceptable?

[Performance test]

We measured the tsc needed to the ioctl()s for getting dirty logs in kernel.

Test environment

AMD Phenom(tm) 9850 Quad-Core Processor with 8GB memory

1. GUI test (running Ubuntu guest in graphical mode)

sudo qemu-system-x86_64 -hda dirtylog_test.img -boot c -m 4192 -net ...

We show a relatively stable part to compare how much time is needed for the basic parts of dirty log ioctl.

get.org get.opt switch.opt

slots[7].len=32768          278379     66398     64024
slots[8].len=32768          181246       270       160
slots[7].len=32768          263961     64673     64494
slots[8].len=32768          181655       265       160
slots[7].len=32768          263736     64701     64610
slots[8].len=32768          182785       267       160
slots[7].len=32768          260925     65360     65042
slots[8].len=32768          182579       264       160
slots[7].len=32768          267823     65915     65682
slots[8].len=32768          186350       271       160

At a glance, we know our optimization improved significantly compared to the original get dirty log ioctl. This is true for both get.opt and switch.opt. This has a really big impact for the personal KVM users who drive KVM in GUI mode on their usual PCs.

Next, we notice that switch.opt improved a hundred nano seconds or so for these slots. Although this may sound a bit tiny improvement, we can feel this as a difference of GUI's responses like mouse reactions.

To feel the difference, please try GUI on your PC with our patch series!

2. Live-migration test (4GB guest, write loop with 1GB buf)

We also did a live-migration test.

get.org get.opt switch.opt

slots[0].len=655360 797383 261144 222181

slots[1].len=3757047808    2186721   1965244   1842824
slots[2].len=637534208     1433562   1012723   1031213
slots[3].len=131072         216858       331       331
slots[4].len=131072         121635       225       164
slots[5].len=131072         120863       356       164
slots[6].len=16777216       121746      1133       156
slots[7].len=32768          120415       230       278
slots[8].len=32768          120368       216       149
slots[0].len=655360         806497    194710    223582
slots[1].len=3757047808    2142922   1878025   1895369
slots[2].len=637534208     1386512   1021309   1000345
slots[3].len=131072         221118       459       296
slots[4].len=131072         121516       272       166
slots[5].len=131072         122652       244       173
slots[6].len=16777216       123226     99185       149
slots[7].len=32768          121803       457       505
slots[8].len=32768          121586       216       155
slots[0].len=655360         766113    211317    213179
slots[1].len=3757047808    2155662   1974790   1842361
slots[2].len=637534208     1481411   1020004   1031352
slots[3].len=131072         223100       351       295
slots[4].len=131072         122982       436       164
slots[5].len=131072         122100       300       503
slots[6].len=16777216       123653       779       151
slots[7].len=32768          122617       284       157
slots[8].len=32768          122737       253       149

For slots other than 0,1,2 we can see the similar improvement.

Considering the fact that switch.opt does not depend on the bitmap length except for kvm_mmu_slot_remove_write_access(), this is the cause of some usec to msec time consumption: there might be some context switches.

But note that this was done with the workload which dirtied the memory endlessly during the live-migration.

In usual workload, the number of dirty pages varies a lot for each iteration and we should gain really a lot for relatively clean cases.