stand-alone kvmtool
Hi,
as I found it increasingly inconvenient to use kvmtool[1] as part of a Linux repository, I decided to give it a go and make it a stand-alone project. So I filtered all the respective commits, adjusted the paths in there (while keeping authorship and commit date, of course) and then added the missing bits to let it compile without a kernel tree nearby. The result is now available on:
git://linux-arm.org/kvmtool.git
http://linux-arm.org/kvmtool.git
You can simply check it out, type make and use "./lkvm run" for a quick test. So far I briefly tested x86-64, arm and arm64, the later two were also cross-compiled. For sure there are rough edges in there (for instance copying a few non-uapi header files into), but I deem it worthy enough to get some public comments.
For me that also fixed some nasty warnings about libfdt, which now are gone due it using your system library version of it.
I also managed to get rid of the libc-i386-dev dependency when compiling for x86-64, but that still needs to be cleaned up and thus is not in the current HEAD.
I haven't got around to compile-test the other supported architectures, but supporting them should be as easy as copying over the uapi kvm.h header file (see the respective ARM commit). Contributions (and tests!) are welcome.
Please give it a go and tell me what you think. I don't want to fork the project, so I am happy if someone "official" picks it up.
Cheers,
Andre.
[1] https://github.com/penberg/linux-kvm/tree/master/tools/kvm
Hello Andre,
Quoted 11 lines · Andre Przywara
Hi,
as I found it increasingly inconvenient to use kvmtool[1] as part of a Linux repository, I decided to give it a go and make it a stand-alone project. So I filtered all the respective commits, adjusted the paths in there (while keeping authorship and commit date, of course) and then added the missing bits to let it compile without a kernel tree nearby. The result is now available on:
git://linux-arm.org/kvmtool.git
http://linux-arm.org/kvmtool.git
It builds fine on x86_64, but when I tried to crosscompile from x86_64 to AArch64,
I get in trouble because of libfdt: I have the aarch64 libs (static and shared), but how do I instruct the build system to get it from the right place?
Thanks,
Claudio
Quoted 28 lines
You can simply check it out, type make and use "./lkvm run" for a quick test. So far I briefly tested x86-64, arm and arm64, the later two were also cross-compiled. For sure there are rough edges in there (for instance copying a few non-uapi header files into), but I deem it worthy enough to get some public comments.
For me that also fixed some nasty warnings about libfdt, which now are gone due it using your system library version of it.
I also managed to get rid of the libc-i386-dev dependency when compiling for x86-64, but that still needs to be cleaned up and thus is not in the current HEAD.
I haven't got around to compile-test the other supported architectures, but supporting them should be as easy as copying over the uapikvm.hheader file (see the respective ARM commit). Contributions (and tests!) are welcome.Please give it a go and tell me what you think. I don't want to fork the project, so I am happy if someone "official" picks it up.
Cheers,
Andre.[1] https://github.com/penberg/linux-kvm/tree/master/tools/kvm
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Signature · 7 lines
Claudio Fontana Server Virtualization Architect Huawei Technologies Duesseldorf GmbH Riesstraße 25 - 80992 München office: +49 89 158834 4135 mobile: +49 15253060158
Ciao Claudio,
Quoted 12 lines · Claudio Fontana
Hello Andre,
Quoted 9 lines · Andre Przywara
Hi,
as I found it increasingly inconvenient to use kvmtool[1] as part of a Linux repository, I decided to give it a go and make it a stand-alone project. So I filtered all the respective commits, adjusted the paths in there (while keeping authorship and commit date, of course) and then added the missing bits to let it compile without a kernel tree nearby. The result is now available on:
Quoted 5 lines
Quoted 2 lines
git://linux-arm.org/kvmtool.git
http://linux-arm.org/kvmtool.gitIt builds fine on
x86_64, but when I tried to crosscompile fromx86_64to AArch64,
I get in trouble because of libfdt: I have the aarch64 libs (static and shared), but how do I instruct the build system to get it from the right place?
You have to install them into your cross-compiler's SYSROOT. Get the location of that by executing
$ ${CROSS_COMPILE}gcc -print-sysroot.
If it's just for libfdt, it's probably the easiest to copy them
manually, the header files into $SYSROOT/usr/include, the libraries into
$SYSROOT/usr/lib/aarch64-linux-gnu
That fixed it for me ;-)For a more robust approach you would use your distribution's packaging system to install the aarch64 package into $SYSROOT.
Cheers,
Andre.
Quoted 31 lines
Quoted 29 lines
You can simply check it out, type make and use "./lkvm run" for a quick test. So far I briefly tested x86-64, arm and arm64, the later two were also cross-compiled. For sure there are rough edges in there (for instance copying a few non-uapi header files into), but I deem it worthy enough to get some public comments.
For me that also fixed some nasty warnings about libfdt, which now are gone due it using your system library version of it.
I also managed to get rid of the libc-i386-dev dependency when compiling for x86-64, but that still needs to be cleaned up and thus is not in the current HEAD.
I haven't got around to compile-test the other supported architectures, but supporting them should be as easy as copying over the uapikvm.hheader file (see the respective ARM commit). Contributions (and tests!) are welcome.Please give it a go and tell me what you think. I don't want to fork the project, so I am happy if someone "official" picks it up.
Cheers,
Andre.[1] https://github.com/penberg/linux-kvm/tree/master/tools/kvm
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Quoted 21 lines · Andre Przywara
Ciao Claudio,
Quoted 12 lines · Claudio Fontana
Hello Andre,
Quoted 9 lines · Andre Przywara
Hi,
as I found it increasingly inconvenient to use kvmtool[1] as part of a Linux repository, I decided to give it a go and make it a stand-alone project. So I filtered all the respective commits, adjusted the paths in there (while keeping authorship and commit date, of course) and then added the missing bits to let it compile without a kernel tree nearby. The result is now available on:
Quoted 5 lines
Quoted 2 lines
git://linux-arm.org/kvmtool.git
http://linux-arm.org/kvmtool.gitIt builds fine on
x86_64, but when I tried to crosscompile fromx86_64to AArch64,
I get in trouble because of libfdt: I have the aarch64 libs (static and shared), but how do I instruct the build system to get it from the right place?
Quoted 7 lines
You have to install them into your cross-compiler's SYSROOT. Get the location of that by executing
$ ${CROSS_COMPILE}gcc -print-sysroot. If it's just for libfdt, it's probably the easiest to copy them manually, the header files into $SYSROOT/usr/include, the libraries into $SYSROOT/usr/lib/aarch64-linux-gnu That fixed it for me ;-)
Thanks!
Quoted 6 lines
For a more robust approach you would use your distribution's packaging system to install the aarch64 package into $SYSROOT.
Cheers,
Andre.
I still prefer to be forced to understand things so I actually prefer the manual route.
Ciao,
Claudio
Quoted 31 lines
Quoted 31 lines
Quoted 29 lines
You can simply check it out, type make and use "./lkvm run" for a quick test. So far I briefly tested x86-64, arm and arm64, the later two were also cross-compiled. For sure there are rough edges in there (for instance copying a few non-uapi header files into), but I deem it worthy enough to get some public comments.
For me that also fixed some nasty warnings about libfdt, which now are gone due it using your system library version of it.
I also managed to get rid of the libc-i386-dev dependency when compiling for x86-64, but that still needs to be cleaned up and thus is not in the current HEAD.
I haven't got around to compile-test the other supported architectures, but supporting them should be as easy as copying over the uapikvm.hheader file (see the respective ARM commit). Contributions (and tests!) are welcome.Please give it a go and tell me what you think. I don't want to fork the project, so I am happy if someone "official" picks it up.
Cheers,
Andre.[1] https://github.com/penberg/linux-kvm/tree/master/tools/kvm
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Hi Andre,
Thanks for doing this. Since it looks unlikely that kvmtool will ever be merged back into the kernel tree, it makes sense to cut the dependency in my opinion.
Quoted 25 lines · Andre Przywara
as I found it increasingly inconvenient to use kvmtool[1] as part of a Linux repository, I decided to give it a go and make it a stand-alone project. So I filtered all the respective commits, adjusted the paths in there (while keeping authorship and commit date, of course) and then added the missing bits to let it compile without a kernel tree nearby. The result is now available on:
git://linux-arm.org/kvmtool.git
http://linux-arm.org/kvmtool.gitYou can simply check it out, type make and use "./lkvm run" for a quick test. So far I briefly tested x86-64, arm and arm64, the later two were also cross-compiled. For sure there are rough edges in there (for instance copying a few non-uapi header files into), but I deem it worthy enough to get some public comments.
For me that also fixed some nasty warnings about libfdt, which now are gone due it using your system library version of it.
I also managed to get rid of the libc-i386-dev dependency when compiling for x86-64, but that still needs to be cleaned up and thus is not in the current HEAD.
I haven't got around to compile-test the other supported architectures, but supporting them should be as easy as copying over the uapikvm.hheader file (see the respective ARM commit). Contributions (and tests!) are welcome.
Quoted 2 lines
Please give it a go and tell me what you think. I don't want to fork the project, so I am happy if someone "official" picks it up.
In which case, it's probably best to post the patches for review rather than just point me at your git repo!
Will
Hi Will,
Quoted 32 lines · Will Deacon
Hi Andre,
Thanks for doing this. Since it looks unlikely that kvmtool will ever be merged back into the kernel tree, it makes sense to cut the dependency in my opinion.
Quoted 25 lines · Andre Przywara
as I found it increasingly inconvenient to use kvmtool[1] as part of a Linux repository, I decided to give it a go and make it a stand-alone project. So I filtered all the respective commits, adjusted the paths in there (while keeping authorship and commit date, of course) and then added the missing bits to let it compile without a kernel tree nearby. The result is now available on:
git://linux-arm.org/kvmtool.git
http://linux-arm.org/kvmtool.gitYou can simply check it out, type make and use "./lkvm run" for a quick test. So far I briefly tested x86-64, arm and arm64, the later two were also cross-compiled. For sure there are rough edges in there (for instance copying a few non-uapi header files into), but I deem it worthy enough to get some public comments.
For me that also fixed some nasty warnings about libfdt, which now are gone due it using your system library version of it.
I also managed to get rid of the libc-i386-dev dependency when compiling for x86-64, but that still needs to be cleaned up and thus is not in the current HEAD.
I haven't got around to compile-test the other supported architectures, but supporting them should be as easy as copying over the uapikvm.hheader file (see the respective ARM commit). Contributions (and tests!) are welcome.
Quoted 5 lines
Quoted 2 lines
Please give it a go and tell me what you think. I don't want to fork the project, so I am happy if someone "official" picks it up.
In which case, it's probably best to post the patches for review rather than just point me at your git repo!
Makes some sense, although part of the exercise was to get rid of the huge, now unneeded Linux kernel code base.
So this approach required a fresh repository, and due to the different paths there is no out-of-the-box patch compatibility between the two. Also I wanted to provide an easy way for people to give it a test.
So what I could do is to send the top-most patches against Pekka's github repository, which would eliminate the references to the kernel directory (at the cost of duplicating some files).
Once this is settled, acked and applied, one could try to create a new repository with the tools/kvm directory being the new root.
Let me know if that makes more sense and I will rework the patches to apply against the current upstream kvmtool.
Cheers,
Andre.
P.S. Although both approaches still provide the kvmtool patch history, they do not compile before the dependency cut patches. If that is an issue, one could think about injecting those new patches back into the repository time line. Admittedly that sounds scary, but would solve the problem.
Quoted 9 lines · Andre Przywara
Hi Will,
Quoted 6 lines · Will Deacon
Hi Andre,
Thanks for doing this. Since it looks unlikely that kvmtool will ever be merged back into the kernel tree, it makes sense to cut the dependency in my opinion.
<snip>
Quoted 6 lines
P.S. Although both approaches still provide the kvmtool patch history, they do not compile before the dependency cut patches. If that is an issue, one could think about injecting those new patches back into the repository time line. Admittedly that sounds scary, but would solve the problem.
If you can have it all it would be nice to preserve buildability all through your history for bisecting (and the moon on a stick please ;-)
Is the dependency on the kernel sources something that has been stable over the projects history or something that's been declining/increasing over time?
Signature · 5 lines
Alex Bennée _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Quoted 6 lines · Alex Bennée
If you can have it all it would be nice to preserve buildability all through your history for bisecting (and the moon on a stick please ;-)
Is the dependency on the kernel sources something that has been stable over the projects history or something that's been declining/increasing over time?
AFAICT, it's been mostly stable after the initial sweep to use kernel-provided data structures such as lists and rbtrees.
- Pekka
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Hi,
Quoted 3 lines · Will Deacon
Thanks for doing this. Since it looks unlikely that kvmtool will ever be merged back into the kernel tree, it makes sense to cut the dependency in my opinion.
I am certainly OK with a standalone repository which preserves the history. Will, would you like to take over the proposed new repository and put it somewhere on git.kernel.org, perhaps?
- Pekka
Quoted 1 line · Pekka Enberg
Hi,
Hi Pekka,
Sorry for the delay, I've been away from email for a few days.
Quoted 8 lines · Will Deacon
Quoted 3 lines
Thanks for doing this. Since it looks unlikely that kvmtool will ever be merged back into the kernel tree, it makes sense to cut the dependency in my opinion.
I am certainly OK with a standalone repository which preserves the history. Will, would you like to take over the proposed new repository and put it somewhere on git.kernel.org, perhaps?
Sure thing. I'll sync-up with Andre and reply back here when we've got something sensible.
One extra point that I don't think has been mentioned in this thread so far is that separating kvmtool from the kernel sources is likely a prerequisite for distribution packaging. Once we've got something sorted out, I'll poke some friendly local debian developers and see if they can't package it up for us.
Will
Quoted 8 lines · Pekka Enberg
Quoted 3 lines · Will Deacon
Thanks for doing this. Since it looks unlikely that kvmtool will ever be merged back into the kernel tree, it makes sense to cut the dependency in my opinion.
I am certainly OK with a standalone repository which preserves the history. Will, would you like to take over the proposed new repository and put it somewhere on git.kernel.org, perhaps?
Finally got around to this with Andre's help, so I'll post a separate mail advertising it.
Will
GREAT stuff. I'll be looking for it :-)
ron
Quoted 7 lines · Will Deacon
Quoted 2 lines · Pekka Enberg
Quoted 1 line · Will Deacon
Thanks for doing this. Since it looks unlikely that kvmtool will ever
be
Quoted 3 lines
Quoted 2 lines
merged back into the kernel tree, it makes sense to cut the dependency in my opinion.
Quoted 9 lines
Quoted 3 lines
I am certainly OK with a standalone repository which preserves the history. Will, would you like to take over the proposed new repository and put it somewhere on git.kernel.org, perhaps?
Finally got around to this with Andre's help, so I'll post a separate mail advertising it.
Will
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Quoted 11 lines · Andre Przywara
Hi,
as I found it increasingly inconvenient to use kvmtool[1] as part of a Linux repository, I decided to give it a go and make it a stand-alone project. So I filtered all the respective commits, adjusted the paths in there (while keeping authorship and commit date, of course) and then added the missing bits to let it compile without a kernel tree nearby. The result is now available on:
git://linux-arm.org/kvmtool.git
http://linux-arm.org/kvmtool.git
Hi Andre,
What inconvenience is caused by having it sit inside the kernel tree beyond an increased requirement in disk space?
Moving it out will make us lose all the new features and bug fixes we gain from using the kernel code directly rather than copying it once in a while.
With your suggestion we'll end up needing something that copies stuff from the kernel into that standalone tree, just like what qemu does.
Thanks,
Sasha
Hi Sasha,
thanks for taking a look!
Quoted 10 lines · Sasha Levin
Quoted 9 lines · Andre Przywara
Hi,
as I found it increasingly inconvenient to use kvmtool[1] as part of a Linux repository, I decided to give it a go and make it a stand-alone project. So I filtered all the respective commits, adjusted the paths in there (while keeping authorship and commit date, of course) and then added the missing bits to let it compile without a kernel tree nearby. The result is now available on:
Quoted 7 lines
Quoted 2 lines
git://linux-arm.org/kvmtool.git
http://linux-arm.org/kvmtool.gitHi Andre,
What inconvenience is caused by having it sit inside the kernel tree beyond an increased requirement in disk space?
Reduced disk space is admittedly one of the benefits of this exercise. Also it makes cloning a lot easier and would allow easier packaging.
Many of the issues we face here come from the fact that kvmtool lives in _a_ kernel repository, but it's not upstream. So we loose the benefit of joined kernel-userland development. In fact we have to do regular merges of mostly unrelated upstream kernel code into the branch to get it compiled with a new feature.
Also having a pure userland tool in the kernel repository sounds just wrong to me, especially as KVM has a nice API with compatibility features. There is a clear interface between the KVM kernel and the controlling userland, so they should not need to share code beyond the API defining header files. Having a shared code base lures people into breaking the interface.
Quoted 3 lines
Moving it out will make us lose all the new features and bug fixes we gain from using the kernel code directly rather than copying it once in a while.
Which code are you exactly thinking of?
Quoted 1 line
From the code I copied I don't see that rbtree or the Linux list
implementations for instance justify a common code base. If in dire need, one could setup alerts on the few code files copied to spot upstream bug fixes.
I see there is a slight drawback in this regard, but I think the benefits outweigh it.
Quoted 2 lines
With your suggestion we'll end up needing something that copies stuff from the kernel into that standalone tree, just like what qemu does.
While I see that copying is not the best solution, QEMU lives very well with it, doesn't it? With KVM's feature compatibility API and the kernel's "don't break userland" policy there should be no real problem. Also with the current situation we just replace "copy uapi header files" with "merge in upstream kernel code base", which is also manually triggered and much more ugly IMHO.
I agree that the whole argumentation would be much different if kvmtool would be upstream, but it is not and as Will pointed out will probably never be. So to make it's usage easier for the users and distribution package maintainers, I'd like to see it live on in a separate (kernel.org) repository.
I could imagine that the easier accessibility would make it more appealing to potential users (and packagers!)
Cheers,
Andre.
Quoted 2 lines · Sasha Levin
What inconvenience is caused by having it sit inside the kernel tree beyond an increased requirement in disk space?
I've come across this problem with the perf tools - luckily, the perf tools allow the source to be exported from the kernel tree, but it is far from a good solution.
The problem is when you're primarily cross-building the kernel on a system where you don't have the target libraries (because, eg, you're running in a build environment for multiple different target systems.) Having to build userspace tools in that scenario is a _major_ pita.
Yes, of course it's possible to pull the 1GB of kernel GIT respository down onto the target just to build some silly userspace tool, but when your rootfs lives on an 8GB SD card or a USB memory stick (as is the case with the ARM Juno 64-bit platform), and when the userspace tool somehow depends on the kernel source tree being configured, it really starts getting painful.
TBH, I don't much care provided there is a way to export a source tarball for the tool from the kernel (like perf does) which can then be transferred to the target and built there.
Signature · 2 lines
FTTC broadband for 0.8mile line: currently at 10.5Mbps down 400kbps up according to speedtest.net.
Hi,
On Thu, Feb 19, 2015 at 05:56:45AM -0500, Sasha Levin wrote:
[...]
Quoted 4 lines
What inconvenience is caused by having it sit inside the kernel tree beyond an increased requirement in disk space?
FWIW: I would prefer seeing this outside the kernel tree; I think it is slightly confusing to keep it as part of a non-upstream kernel repo and it is simpler to view git change logs etc. in gitweb for a stand-alone repo.
-Christoffer
I much prefer kvmtool to be standalone outside the kernel tree. I don't see any benefit in putting it in the kernel or putting the kernel in it.
I think Andre is doing the right thing here.
ron
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm