kvm: Transcendent Memory (tmem) on KVM
cover kvm: Transcendent Memory (tmem) on KVM
Hi,
We are undergraduate engineering students of Maharashtra Academy of Engineering, Pune, India and we are working on a project entitled 'Transcendent Memory on KVM' as a part of our academics. The project members are:
1. Ashutosh Tripathi
2. Shreyas Mahure
3. Nishant Gulhane
4. Akshay Karle
---
Project Description:
What is Transcendent Memory(tmem in short)?
Transcendent Memory is a memory optimization technique for the virtualized environment. It collects the underutilized memory of the guests and the unassigned(fallow) memory of the host and places it into a central tmem pool. Indirect access to this pool is then provided to the guests. For further information on tmem, please refer the article on lwn by Dr. Dan Magenheimer:
http://lwn.net/Articles/454795/
Since kvm is one of the most popular hypervisors available, we decided to implement this technique for kvm.
---
kvm-tmem Patch details:
This patch adds appropriate shims at the guest that invokes the kvm hypercalls, and the host uses zcache pools to implement the required functions.
To enable tmem on the 'kvm host' add the boot parameter:
"kvmtmem"
And to enable tmem in the 'kvm guests' add the boot parameter:
"tmem"
The diffstat details for this patch are given below:
arch/x86/include/asm/kvm_host.h | 1 arch/x86/kvm/x86.c | 4 drivers/staging/zcache/Makefile | 2 drivers/staging/zcache/kvm-tmem.c | 356 +++++++++++++++++++++++++++++++++++ drivers/staging/zcache/kvm-tmem.h | 55 +++++ drivers/staging/zcache/zcache-main.c | 98 ++++++++- include/linux/kvm_para.h | 1 7 files changed, 508 insertions(+), 9 deletions(-)
We have already uploaded our work alongwith the 'Frontswap' submitted by Dan, on the following link:
https://github.com/akshaykarle/kvm-tmem
Any comments/feedback would be appreciated and will help us a lot with our work.
Regards,
Akshay
Quoted 22 lines · Akshay Karle
Hi,
We are undergraduate engineering students of Maharashtra Academy of Engineering, Pune, India and we are working on a project entitled 'Transcendent Memory on KVM' as a part of our academics. The project members are:
1. Ashutosh Tripathi
2. Shreyas Mahure
3. Nishant Gulhane
4. Akshay Karle---
Project Description:
What is Transcendent Memory(tmem in short)?
Transcendent Memory is a memory optimization technique for the virtualized environment. It collects the underutilized memory of the guests and the unassigned(fallow) memory of the host and places it into a central tmem pool. Indirect access to this pool is then provided to the guests. For further information on tmem, please refer the article on lwn by Dr. Dan Magenheimer:
http://lwn.net/Articles/454795/
Quoted 8 lines
Since kvm is one of the most popular hypervisors available, we decided to implement this technique for kvm.
---
kvm-tmem Patch details:
This patch adds appropriate shims at the guest that invokes the kvm hypercalls, and the host uses zcache pools to implement the required functions.
Great!
Quoted 16 lines
To enable tmem on the 'kvm host' add the boot parameter:
"kvmtmem"
And to enable tmem in the 'kvm guests' add the boot parameter:
"tmem"The diffstat details for this patch are given below:
arch/x86/include/asm/kvm_host.h | 1 arch/x86/kvm/x86.c | 4 drivers/staging/zcache/Makefile | 2 drivers/staging/zcache/kvm-tmem.c | 356 +++++++++++++++++++++++++++++++++++ drivers/staging/zcache/kvm-tmem.h | 55 +++++ drivers/staging/zcache/zcache-main.c | 98 ++++++++- include/linux/kvm_para.h | 1 7 files changed, 508 insertions(+), 9 deletions(-)
Quoted 5 lines
We have already uploaded our work alongwith the 'Frontswap' submitted by Dan, on the following link:
https://github.com/akshaykarle/kvm-tmemAny comments/feedback would be appreciated and will help us a lot with our work.
Great. Will do.
Quoted 3 lines
Regards,
Akshay
Quoted 21 lines
---
kvm-tmem Patch details:
This patch adds appropriate shims at the guest that invokes the kvm hypercalls, and the host uses zcache pools to implement the required functions.To enable tmem on the 'kvm host' add the boot parameter:
"kvmtmem"
And to enable tmem in the 'kvm guests' add the boot parameter:
"tmem"The diffstat details for this patch are given below:
arch/x86/include/asm/kvm_host.h | 1 arch/x86/kvm/x86.c | 4 drivers/staging/zcache/Makefile | 2 drivers/staging/zcache/kvm-tmem.c | 356 +++++++++++++++++++++++++++++++++++ drivers/staging/zcache/kvm-tmem.h | 55 +++++ drivers/staging/zcache/zcache-main.c | 98 ++++++++- include/linux/kvm_para.h | 1 7 files changed, 508 insertions(+), 9 deletions(-)
Quoted 3 lines
We have already uploaded our work alongwith the 'Frontswap' submitted by Dan, on the following link:
https://github.com/akshaykarle/kvm-tmem
Is there a way for these patches to be posted on LKML? It is rather difficult to copy-n-paste patches in emails and sending them. Or if you want to, you can email them directly to me. To do that use 'git send-email' and 'git format-patch' to prep the git commits into patches.
Also, the title says 'RFC 0/2' but I am not seing 1 or 2?
Quoted 22 lines · Akshay Karle
Hi,
We are undergraduate engineering students of Maharashtra Academy of Engineering, Pune, India and we are working on a project entitled 'Transcendent Memory on KVM' as a part of our academics. The project members are:
1. Ashutosh Tripathi
2. Shreyas Mahure
3. Nishant Gulhane
4. Akshay Karle---
Project Description:
What is Transcendent Memory(tmem in short)?
Transcendent Memory is a memory optimization technique for the virtualized environment. It collects the underutilized memory of the guests and the unassigned(fallow) memory of the host and places it into a central tmem pool. Indirect access to this pool is then provided to the guests. For further information on tmem, please refer the article on lwn by Dr. Dan Magenheimer:
http://lwn.net/Articles/454795/
Quoted 5 lines
Since kvm is one of the most popular hypervisors available, we decided to implement this technique for kvm.
Any comments/feedback would be appreciated and will help us a lot with our work.
One of the potential problems with tmem is reduction in performance when the cache hit rate is low, for example when streaming.
Can you test this by creating a large file, for example with
dd < /dev/urandom > file bs=1M count=100000
and then measuring the time to stream it, using
time dd < file > /dev/null
with and without the patch?
Should be done on a cleancache enabled guest filesystem backed by a virtio disk with cache=none.
It would be interesting to compare kvm_stat during the streaming, with and without the patch.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 26 lines
From: Avi Kivity [mailto:[email protected]]
Subject: Re: [RFC 0/2] kvm: Transcendent Memory (tmem) on KVMQuoted 10 lines · Akshay Karle
Hi,
We are undergraduate engineering students of Maharashtra Academy of Engineering, Pune, India and we are working on a project entitled 'Transcendent Memory on KVM' as a part of our academics.
Since kvm is one of the most popular hypervisors available, we decided to implement this technique for kvm.
Any comments/feedback would be appreciated and will help us a lot with our work.
One of the potential problems with tmem is reduction in performance when the cache hit rate is low, for example when streaming.
Can you test this by creating a large file, for example with
dd <
/dev/urandom> file bs=1M count=100000and then measuring the time to stream it, using
time dd < file >
/dev/null
Quoted 7 lines
with and without the patch?
Should be done on a cleancache enabled guest filesystem backed by a virtio disk with cache=none.
It would be interesting to compare
kvm_statduring the streaming, with and without the patch.
Hi Avi --
The "WasActive" patch (https://lkml.org/lkml/2012/1/25/300)
is intended to avoid the streaming situation you are creating here. It increases the "quality" of cached pages placed into zcache and should probably also be used on the guest-side stubs (and/or maybe the host-side zcache... I don't know KVM well enough to determine if that would work).
As Dave Hansen pointed out, the WasActive patch is not yet correct and, as akpm points out, pageflag bits are scarce on 32-bit systems, so it remains to be seen if the WasActive patch can be upstreamed. Or maybe there is a different way to achieve the same goal. But I wanted to let you know that the streaming issue is understood and needs to be resolved for some cleancache backends just as it was resolved in the core mm code.
The measurement you suggest would still be interesting even without the WasActive patch as it measures a "worst case".
Dan
Quoted 28 lines · Dan Magenheimer
Quoted 11 lines
One of the potential problems with tmem is reduction in performance when the cache hit rate is low, for example when streaming.
Can you test this by creating a large file, for example with
dd <
/dev/urandom> file bs=1M count=100000and then measuring the time to stream it, using
time dd < file >
/dev/nullQuoted 7 lines
with and without the patch?
Should be done on a cleancache enabled guest filesystem backed by a virtio disk with cache=none.
It would be interesting to compare
kvm_statduring the streaming, with and without the patch.Hi Avi --
The "WasActive" patch (https://lkml.org/lkml/2012/1/25/300)
is intended to avoid the streaming situation you are creating here. It increases the "quality" of cached pages placed into zcache and should probably also be used on the guest-side stubs (and/or maybe the host-side zcache... I don't know KVM well enough to determine if that would work).
Quoted 7 lines
As Dave Hansen pointed out, the WasActive patch is not yet correct and, as akpm points out, pageflag bits are scarce on 32-bit systems, so it remains to be seen if the WasActive patch can be upstreamed. Or maybe there is a different way to achieve the same goal. But I wanted to let you know that the streaming issue is understood and needs to be resolved for some cleancache backends just as it was resolved in the core mm code.
Nice. This takes care of the tail-end of the streaming (the more important one - since it always involves a cold copy). What about the other side? Won't the read code invoke cleancache_get_page() for every page? (this one is just a null hypercall, so it's cheaper, but still expensive).
Quoted 2 lines
The measurement you suggest would still be interesting even without the WasActive patch as it measures a "worst case".
It can provide the justification for that patch, yes.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 37 lines · Avi Kivity
Quoted 28 lines · Dan Magenheimer
Quoted 11 lines
One of the potential problems with tmem is reduction in performance when the cache hit rate is low, for example when streaming.
Can you test this by creating a large file, for example with
dd <
/dev/urandom> file bs=1M count=100000and then measuring the time to stream it, using
time dd < file >
/dev/nullQuoted 7 lines
with and without the patch?
Should be done on a cleancache enabled guest filesystem backed by a virtio disk with cache=none.
It would be interesting to compare
kvm_statduring the streaming, with and without the patch.Hi Avi --
The "WasActive" patch (https://lkml.org/lkml/2012/1/25/300)
is intended to avoid the streaming situation you are creating here. It increases the "quality" of cached pages placed into zcache and should probably also be used on the guest-side stubs (and/or maybe the host-side zcache... I don't know KVM well enough to determine if that would work).Quoted 7 lines
As Dave Hansen pointed out, the WasActive patch is not yet correct and, as akpm points out, pageflag bits are scarce on 32-bit systems, so it remains to be seen if the WasActive patch can be upstreamed. Or maybe there is a different way to achieve the same goal. But I wanted to let you know that the streaming issue is understood and needs to be resolved for some cleancache backends just as it was resolved in the core mm code.
Quoted 5 lines
Nice. This takes care of the tail-end of the streaming (the more important one - since it always involves a cold copy). What about the other side? Won't the read code invoke
cleancache_get_page()for every page? (this one is just a null hypercall, so it's cheaper, but still expensive).
That is something we should fix - I think it was mentioned in the frontswap email thread the need for batching and it certainly seems required as those hypercalls aren't that cheap.
Quoted 10 lines · Konrad Rzeszutek Wilk
Quoted 6 lines
Nice. This takes care of the tail-end of the streaming (the more important one - since it always involves a cold copy). What about the other side? Won't the read code invoke
cleancache_get_page()for every page? (this one is just a null hypercall, so it's cheaper, but still expensive).That is something we should fix - I think it was mentioned in the frontswap email thread the need for batching and it certainly seems required as those hypercalls aren't that cheap.
In fact when tmem was first proposed I asked for two changes - make it batchable, and make it asynchronous (so we can offload copies to a dma engine, etc). Of course that would have made tmem significantly more complicated.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 15 lines
From: Avi Kivity [mailto:[email protected]]
Sent: Thursday, March 15, 2012 12:11 PM
To: Konrad Rzeszutek Wilk
Cc: Dan Magenheimer; Akshay Karle; [email protected]; [email protected]; ashu tripathi; nishant gulhane; amarmore2006; Shreyas Mahure; mahesh mohan
Subject: Re: [RFC 0/2] kvm: Transcendent Memory (tmem) on KVMQuoted 7 lines · Konrad Rzeszutek Wilk
Quoted 6 lines
Nice. This takes care of the tail-end of the streaming (the more important one - since it always involves a cold copy). What about the other side? Won't the read code invoke
cleancache_get_page()for every page? (this one is just a null hypercall, so it's cheaper, but still expensive).
Quoted 8 lines
Quoted 3 lines
That is something we should fix - I think it was mentioned in the frontswap email thread the need for batching and it certainly seems required as those hypercalls aren't that cheap.
In fact when tmem was first proposed I asked for two changes - make it batchable, and make it asynchronous (so we can offload copies to a dma engine, etc). Of course that would have made tmem significantly more complicated.
(Sorry, I'm not typing fast enough to keep up with the thread...)
Hi Avi --
In case it wasn't clear from my last reply, RAMster shows that tmem CAN be used asynchronously... by making it more complicated, but without making the core kernel changes more complicated.
In RAMster, pages are locally cached (compressed using zcache) and then, depending on policy, a separate thread sends the pages to a remote machine. So the first part (compress and store locally) still must be synchronous, but the second part (transmit to another -- remote or possibly host? -- system) can be done asynchronously. The RAMster code has to handle all the race conditions, which is a pain but seems to work.
This is all working today in RAMster (which is in linux-next). Batching is still not implemented by any tmem backend, but RAMster demonstrates how the backend implementation COULD do batching without any additional core kernel changes. I.e. no changes necessary to frontswap or cleancache.
So, you see, I *was* listening. I just wasn't willing to fight the uphill battle of much more complexity in the core kernel for a capability that could be implemented differently.
That said, I still think it remains to be proven that reducing the number of hypercalls by 2x or 3x (or whatever the batching factor you choose) will make a noticeable performance difference. But if it does, batching can be done... and completely hidden in the backend.
(I hope Andrea is listening ;-)
Dan
Quoted 47 lines · Dan Magenheimer
Quoted 15 lines
From: Avi Kivity [mailto:[email protected]]
Sent: Thursday, March 15, 2012 12:11 PM
To: Konrad Rzeszutek Wilk
Cc: Dan Magenheimer; Akshay Karle; [email protected]; [email protected]; ashu tripathi; nishant gulhane; amarmore2006; Shreyas Mahure; mahesh mohan
Subject: Re: [RFC 0/2] kvm: Transcendent Memory (tmem) on KVMQuoted 7 lines · Konrad Rzeszutek Wilk
Quoted 6 lines
Nice. This takes care of the tail-end of the streaming (the more important one - since it always involves a cold copy). What about the other side? Won't the read code invoke
cleancache_get_page()for every page? (this one is just a null hypercall, so it's cheaper, but still expensive).Quoted 8 lines
Quoted 3 lines
That is something we should fix - I think it was mentioned in the frontswap email thread the need for batching and it certainly seems required as those hypercalls aren't that cheap.
In fact when tmem was first proposed I asked for two changes - make it batchable, and make it asynchronous (so we can offload copies to a dma engine, etc). Of course that would have made tmem significantly more complicated.
(Sorry, I'm not typing fast enough to keep up with the thread...)
Hi Avi --
In case it wasn't clear from my last reply, RAMster shows that tmem CAN be used asynchronously... by making it more complicated, but without making the core kernel changes more complicated.
In RAMster, pages are locally cached (compressed using zcache) and then, depending on policy, a separate thread sends the pages to a remote machine. So the first part (compress and store locally) still must be synchronous, but the second part (transmit to another -- remote or possibly host? -- system) can be done asynchronously. The RAMster code has to handle all the race conditions, which is a pain but seems to work.
This is all working today in RAMster (which is in linux-next). Batching is still not implemented by any tmem backend, but RAMster demonstrates how the backend implementation COULD do batching without any additional core kernel changes. I.e. no changes necessary to frontswap or cleancache.
Quoted 3 lines
So, you see, I *was* listening. I just wasn't willing to fight the uphill battle of much more complexity in the core kernel for a capability that could be implemented differently.
Dan, please stop this.
The frontswap work is going through me and my goal is to provide the batching and asynchronous option. It might take longer than anticipated b/c it might require redoing some of the code - that is OK. We can do this in steps too - first do the synchronous (as is right now in implementation) and then add on the batching and asynchrnous work. This means breaking the ABI/API, and I believe Avi would like the ABI be as much baked as possible so that he does not have to provide a v2 (or v3) of the tmem support in KVM.
I appreciate you having done that in RAMster but the "transmit" option is what we need to batch. Think of Scatter Gather DMA.
Quoted 4 lines
That said, I still think it remains to be proven that reducing the number of hypercalls by 2x or 3x (or whatever the batching factor you choose) will make a noticeable
I was thinking 32 - about the same number that we do in Xen with PV MMU upcalls. We also batch it there with multicalls.
Quoted 6 lines
performance difference. But if it does, batching can be done... and completely hidden in the backend.
(I hope Andrea is listening ;-)
Dan
Quoted 27 lines
From: Konrad Rzeszutek Wilk
Subject: Re: [RFC 0/2] kvm: Transcendent Memory (tmem) on KVMQuoted 17 lines · Avi Kivity
Quoted 8 lines · Dan Magenheimer
The "WasActive" patch (https://lkml.org/lkml/2012/1/25/300)
is intended to avoid the streaming situation you are creating here. It increases the "quality" of cached pages placed into zcache and should probably also be used on the guest-side stubs (and/or maybe the host-side zcache... I don't know KVM well enough to determine if that would work).Quoted 7 lines
As Dave Hansen pointed out, the WasActive patch is not yet correct and, as akpm points out, pageflag bits are scarce on 32-bit systems, so it remains to be seen if the WasActive patch can be upstreamed. Or maybe there is a different way to achieve the same goal. But I wanted to let you know that the streaming issue is understood and needs to be resolved for some cleancache backends just as it was resolved in the core mm code.
Quoted 5 lines
Nice. This takes care of the tail-end of the streaming (the more important one - since it always involves a cold copy). What about the other side? Won't the read code invoke
cleancache_get_page()for every page? (this one is just a null hypercall, so it's cheaper, but still expensive).
Quoted 3 lines
That is something we should fix - I think it was mentioned in the frontswap email thread the need for batching and it certainly seems required as those hypercalls aren't that cheap.
And exactly how expensive ARE hypercalls these days? On the first VT/SVN systems they were tens of thousands of cycles... now they are closer to sub-thousand are they not? (I remember seeing a graph of hypercall overhead dropping across generations of CPUs... anybody have a pointer to a public graph of this?)
One of my favorite papers these days is "When Poll is Better than Interrupt"
(http://static.usenix.org/events/fast12/tech/full_papers/Yang.pdf) which argues that wasting some CPU cycles doing a busy-wait is often more efficient than slogging through the Block I/O subsystem to set up and respond to an interrupt, if the device is fast enough. I wonder if the same might be true comparing hypercall overhead for tmem vs the path for KVM to get a page from the host via its normal path?
Ignoring that for now, if excessive hypercalls is a problem, a better solution than batching may be to modify the Maharashtra approach to be more like RAMster: Put zcache in the guest-side and treat the host like a "remote" system.
But let's wait for the Maharashta team to do some measurements first before we make any assumptions or change any designs...