debugging windows guests
Hello,
I am researching KVM as a malware analysis platform and had some questions about debugging the guest OS. In my case I intend to use windows guests. So my questsions are as follows:
Questions:
1. What instrumentation facilities are their available?
2. Is it possible to extend the debugging interface so that debugging is more transparent to the guest OS? IE: there is still a limit of 4 HW breakpoints (which makes me wonder why a LIST is used for them...)
3. I'm not finding any published API for interfacing with KVM/KQEMU/QEMU at a low level, for example, for writing custom tracers, etc. Is there one? Or is there something similar?
Bugs:
1. I hit a bug w/ instruction logging using a RAM based temp folder. If I ran w/ the following command line:
(Version info: QEMU PC emulator version 0.10.50 (qemu-kvm-devel-88))
qemu-system-x86_64 -hda debian.img -enable-nesting -d in_asm
It would successfully log to the tmp log file, but obviously, KVM would be disabled.
If I use sudo, it won't log to the file, is this a known issue?
2. -enable-nesting on AMD hardware using a xen guest OS causes xen to GPF somewhere in svm_cpu_up. Is nesting supposed to work w/ Xen based guests?
Quoted 11 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Raindog
Sent: Tuesday, December 15, 2009 2:25 AM
To: [email protected]
Subject: debugging windows guestsHello,
Quoted 7 lines
I am researching KVM as a malware analysis platform and had some questions about debugging the guest OS. In my case I intend to use windows guests. So my questsions are as follows:
Questions:
1. What instrumentation facilities are their available?
[YV] http://www.linux-kvm.org/page/WindowsGuestDrivers/GuestDebugging
Quoted 34 lines
2. Is it possible to extend the debugging interface so that debugging is
more transparent to the guest OS? IE: there is still a limit of 4 HW breakpoints (which makes me wonder why a LIST is used for them...)3. I'm not finding any published API for interfacing with KVM/KQEMU/QEMU
at a low level, for example, for writing custom tracers, etc. Is there one? Or is there something similar?Bugs:
1. I hit a bug w/ instruction logging using a RAM based temp folder. If I ran w/ the following command line:
(Version info: QEMU PC emulator version 0.10.50 (qemu-kvm-devel-88))qemu-system-x86_64 -hda debian.img -enable-nesting -d
in_asmIt would successfully log to the tmp log file, but obviously, KVM would be disabled.
If I use sudo, it won't log to the file, is this a known issue?
2. -enable-nesting on AMD hardware using a xen guest OS causes xen to GPF somewhere in
svm_cpu_up. Is nesting supposed to work w/ Xen based guests?--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 6 lines · Raindog
Hello,
I am researching KVM as a malware analysis platform and had some questions about debugging the guest OS. In my case I intend to use windows guests. So my questsions are as follows:
Quoted 7 lines
Questions:
1. What instrumentation facilities are their available?
2. Is it possible to extend the debugging interface so that debugging is more transparent to the guest OS? IE: there is still a limit of 4 HW breakpoints (which makes me wonder why a LIST is used for them...)
In accelerated KVM mode, the x86 architecture restricts us to 4 break- or watchpoints that can be active at the same time. If you switch to emulation mode, there are no such limits. Actually, I just made use of this for debugging a subtle stack corruption in a guest, and I had more than 70 watchpoints active at the same time. It's just "slightly" slower than KVM...
Quoted 4 lines
3. I'm not finding any published API for interfacing with KVM/KQEMU/QEMU at a low level, for example, for writing custom tracers, etc. Is there one? Or is there something similar?
KVM provides tracepoints for the Linux ftrace framework, see related documentation of the kernel. If you extend your guest to issue certain events that the hypervisor sees and traces (e.g. writes to pseudo I/O ports), you can also trace things inside the guest that are otherwise invisible to the host. I once hacked up an ad-hoc tracing by means of hypercalls (required some kvm patching). That also worked from guest userspace - and revealed that even more hypercalls could be called that way (that's fixed in KVM now).
Quoted 9 lines
Bugs:
1. I hit a bug w/ instruction logging using a RAM based temp folder. If I ran w/ the following command line:
(Version info: QEMU PC emulator version 0.10.50 (qemu-kvm-devel-88))qemu-system-x86_64 -hda debian.img -enable-nesting -d
in_asm
-d only works in emulation mode as it relies on dynamic code translation (TCG). For qemu-kvm, you need to switch to emulation via -no-kvm (for upstream QEMU, it's the other way around).
Quoted 9 lines
It would successfully log to the tmp log file, but obviously, KVM would be disabled.
If I use sudo, it won't log to the file, is this a known issue?
2. -enable-nesting on AMD hardware using a xen guest OS causes xen to GPF somewhere in
svm_cpu_up. Is nesting supposed to work w/ Xen based guests?
If your host kernel or kvm-kmod is not 2.6.32 based, update first. A lot of nested SVM fixes went in recently. If it still fails, put Alex (Graf) and Joerg (Roedel) on CC.
Jan
Quoted 15 lines · Jan Kiszka
Quoted 6 lines · Raindog
Hello,
I am researching KVM as a malware analysis platform and had some questions about debugging the guest OS. In my case I intend to use windows guests. So my questsions are as follows:
Quoted 7 lines
Questions:
1. What instrumentation facilities are their available?
2. Is it possible to extend the debugging interface so that debugging is more transparent to the guest OS? IE: there is still a limit of 4 HW breakpoints (which makes me wonder why a LIST is used for them...)
Quoted 7 lines
In accelerated KVM mode, the x86 architecture restricts us to 4 break- or watchpoints that can be active at the same time. If you switch to emulation mode, there are no such limits. Actually, I just made use of this for debugging a subtle stack corruption in a guest, and I had more than 70 watchpoints active at the same time. It's just "slightly" slower than KVM...
Ok, is there anything special that needs to be done to enable additional watchpoints as they are being called? How are these set btw? Is it accomplished transparently through gdb? IE: if you set a watchpoint at a specific address, under emulation mode, they simulate HW bps in that no code is modified via the injection of an int 3?
Quoted 7 lines
Quoted 4 lines
3. I'm not finding any published API for interfacing with KVM/KQEMU/QEMU at a low level, for example, for writing custom tracers, etc. Is there one? Or is there something similar?
KVM provides tracepoints for the Linux ftrace framework, see related documentation of the kernel.
I found this http://lxr.linux.no/#linux+v2.6.27/Documentation/ftrace.txt but that can hardly be accused of being called documentation. I don't think something like this: http://www.pintool.org/tutorials/asplos08/slides/PinTutorial.pdf is unreasonable.
Quoted 1 line
If you extend your guest
Windows is by design not extensible.
Quoted 16 lines
to issue certain
events that the hypervisor sees and traces (e.g. writes to pseudo I/O ports), you can also trace things inside the guest that are otherwise invisible to the host. I once hacked up an ad-hoc tracing by means of hypercalls (required some kvm patching). That also worked from guest userspace - and revealed that even more hypercalls could be called that way (that's fixed in KVM now).Quoted 8 lines
Bugs:
1. I hit a bug w/ instruction logging using a RAM based temp folder. If I ran w/ the following command line:
(Version info: QEMU PC emulator version 0.10.50 (qemu-kvm-devel-88))
Quoted 6 lines
Quoted 1 line
qemu-system-x86_64 -hda debian.img -enable-nesting -d
in_asm-d only works in emulation mode as it relies on dynamic code translation (TCG). For qemu-kvm, you need to switch to emulation via -no-kvm (for upstream QEMU, it's the other way around).
Hence why running w/out admin rights enables the logging. I was confused because the logfile is still created w/ the kvm module disabled.
Quoted 6 lines
Quoted 6 lines
It would successfully log to the tmp log file, but obviously, KVM would be disabled.
If I use sudo, it won't log to the file, is this a known issue?
Quoted 8 lines
Quoted 3 lines
2. -enable-nesting on AMD hardware using a xen guest OS causes xen to GPF somewhere in
svm_cpu_up. Is nesting supposed to work w/ Xen based guests?If your host kernel or kvm-kmod is not 2.6.32 based, update first. A lot of nested SVM fixes went in recently. If it still fails, put Alex (Graf) and Joerg (Roedel) on CC.
Upgrading now...
Quoted 3 lines
Jan
Quoted 25 lines · Raindog
Quoted 16 lines · Jan Kiszka
Quoted 11 lines · Raindog
Hello,
I am researching KVM as a malware analysis platform and had some questions about debugging the guest OS. In my case I intend to use windows guests. So my questsions are as follows:
Questions:
1. What instrumentation facilities are their available?
2. Is it possible to extend the debugging interface so that
debugging is
Quoted 2 lines
more transparent to the guest OS? IE: there is still a limit of 4 HW breakpoints (which makes me wonder why a LIST is used for them...)
Quoted 7 lines
In accelerated KVM mode, the x86 architecture restricts us to 4 break- or watchpoints that can be active at the same time. If you switch to emulation mode, there are no such limits. Actually, I just made use of this for debugging a subtle stack corruption in a guest, and I had more than 70 watchpoints active at the same time. It's just "slightly" slower than KVM...
Quoted 5 lines
Ok, is there anything special that needs to be done to enable additional watchpoints as they are being called? How are these set btw? Is it accomplished transparently through gdb? IE: if you set a watchpoint at a specific address, under emulation mode, they simulate HW bps in that no code is modified via the injection of an int 3?
Yes, break- and watchpoints are transparent to the guest in emulation mode. In KVM mode, hardware breakpoints do not require int 3, but they are limited and the guest my notice that its own breakpoints have no effect as long as the host injected some.
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
3. I'm not finding any published API for interfacing with
KVM/KQEMU/QEMU
Quoted 2 lines
at a low level, for example, for writing custom tracers, etc. Is there one? Or is there something similar?
Quoted 8 lines
Quoted 2 lines
KVM provides tracepoints for the Linux ftrace framework, see related documentation of the kernel.
I found this http://lxr.linux.no/#linux+v2.6.27/Documentation/ftrace.txt
but that can hardly be accused of being called documentation. I don't think something like this:
http://www.pintool.org/tutorials/asplos08/slides/PinTutorial.pdf is unreasonable.
2.6.27 is too old anyway. There should be at least one LWN.net article on this, and also quite a few presentations and paper, just ask your favorite search engine.
Quoted 3 lines
Quoted 2 lines
If you extend your guest
Windows is by design not extensible.
Depends on where and how you want to hook into it. Of course, its kernel is out of reach. But if you are interested e.g. in marking specific I/O requests, you could write your own driver and hook into the stack.
Quoted 20 lines
Quoted 20 lines
to issue certain
events that the hypervisor sees and traces (e.g. writes to pseudo I/O ports), you can also trace things inside the guest that are otherwise invisible to the host. I once hacked up an ad-hoc tracing by means of hypercalls (required some kvm patching). That also worked from guest userspace - and revealed that even more hypercalls could be called that way (that's fixed in KVM now).Quoted 5 lines
Bugs:
1. I hit a bug w/ instruction logging using a RAM based temp
folder. If
Quoted 4 lines
I ran w/ the following command line:
(Version info: QEMU PC emulator version 0.10.50 (qemu-kvm-devel-88))qemu-system-x86_64 -hda debian.img -enable-nesting -d
in_asm
Quoted 7 lines
Quoted 4 lines
-d only works in emulation mode as it relies on dynamic code translation (TCG). For qemu-kvm, you need to switch to emulation via -no-kvm (for upstream QEMU, it's the other way around).
Hence why running w/out admin rights enables the logging. I was confused because the logfile is still created w/ the kvm module disabled.
It's created but remains empty for obvious reasons in KVM mode.
Jan
Quoted 27 lines · Jan Kiszka
Quoted 6 lines · Raindog
Hello,
I am researching KVM as a malware analysis platform and had some questions about debugging the guest OS. In my case I intend to use windows guests. So my questsions are as follows:
Quoted 7 lines
Questions:
1. What instrumentation facilities are their available?
2. Is it possible to extend the debugging interface so that debugging is more transparent to the guest OS? IE: there is still a limit of 4 HW breakpoints (which makes me wonder why a LIST is used for them...)
In accelerated KVM mode, the x86 architecture restricts us to 4 break- or watchpoints that can be active at the same time. If you switch to emulation mode, there are no such limits. Actually, I just made use of this for debugging a subtle stack corruption in a guest, and I had more than 70 watchpoints active at the same time. It's just "slightly" slower than KVM...
Quoted 4 lines
3. I'm not finding any published API for interfacing with KVM/KQEMU/QEMU at a low level, for example, for writing custom tracers, etc. Is there one? Or is there something similar?
Quoted 5 lines
KVM provides tracepoints for the Linux ftrace framework, see related documentation of the kernel. If you extend your guest to issue certain events that the hypervisor sees and traces (e.g. writes to pseudo I/O ports), you can also trace things inside the guest that are otherwise invisible to the host.
You can WRITE_PORT_BUFFER_UCHAR to com1/com2 port when you are in kernel mode.
Quoted 35 lines
I once hacked up an ad-hoc tracing by means of
hypercalls (required some kvm patching). That also worked from guest userspace - and revealed that even more hypercalls could be called that way (that's fixed in KVM now).Quoted 9 lines
Bugs:
1. I hit a bug w/ instruction logging using a RAM based temp folder. If I ran w/ the following command line:
(Version info: QEMU PC emulator version 0.10.50 (qemu-kvm-devel-88))qemu-system-x86_64 -hda debian.img -enable-nesting -d
in_asm-d only works in emulation mode as it relies on dynamic code translation (TCG). For qemu-kvm, you need to switch to emulation via -no-kvm (for upstream QEMU, it's the other way around).
Quoted 9 lines
It would successfully log to the tmp log file, but obviously, KVM would be disabled.
If I use sudo, it won't log to the file, is this a known issue?
2. -enable-nesting on AMD hardware using a xen guest OS causes xen to GPF somewhere in
svm_cpu_up. Is nesting supposed to work w/ Xen based guests?If your host kernel or kvm-kmod is not 2.6.32 based, update first. A lot of nested SVM fixes went in recently. If it still fails, put Alex (Graf) and Joerg (Roedel) on CC.
Jan
Am 16.12.2009 um 09:14 schrieb Vadim Rozenfeld <[email protected]>:
Quoted 28 lines · Jan Kiszka
Quoted 27 lines
Quoted 6 lines · Raindog
Hello,
I am researching KVM as a malware analysis platform and had some questions about debugging the guest OS. In my case I intend to use windows guests. So my questsions are as follows:
Quoted 7 lines
Questions:
1. What instrumentation facilities are their available?
2. Is it possible to extend the debugging interface so that debugging is more transparent to the guest OS? IE: there is still a limit of 4 HW breakpoints (which makes me wonder why a LIST is used for them...)
In accelerated KVM mode, the x86 architecture restricts us to 4 break- or watchpoints that can be active at the same time. If you switch to emulation mode, there are no such limits. Actually, I just made use of this for debugging a subtle stack corruption in a guest, and I had more than 70 watchpoints active at the same time. It's just "slightly" slower than KVM...
Quoted 4 lines
3. I'm not finding any published API for interfacing with KVM/KQEMU/QEMU at a low level, for example, for writing custom tracers, etc. Is there one? Or is there something similar?
Quoted 40 lines
Quoted 5 lines
KVM provides tracepoints for the Linux ftrace framework, see related documentation of the kernel. If you extend your guest to issue certain events that the hypervisor sees and traces (e.g. writes to pseudo I/O ports), you can also trace things inside the guest that are otherwise invisible to the host.
You can
WRITE_PORT_BUFFER_UCHARto com1/com2 port when you are in kernel mode.Quoted 33 lines
I once hacked up an ad-hoc tracing by means of
hypercalls (required some kvm patching). That also worked from guest userspace - and revealed that even more hypercalls could be called that way (that's fixed in KVM now).Quoted 9 lines
Bugs:
1. I hit a bug w/ instruction logging using a RAM based temp folder. If I ran w/ the following command line:
(Version info: QEMU PC emulator version 0.10.50 (qemu-kvm-devel-88))qemu-system-x86_64 -hda debian.img -enable-nesting -d
in_asm-d only works in emulation mode as it relies on dynamic code translation
(TCG). For qemu-kvm, you need to switch to emulation via -no-kvm (for upstream QEMU, it's the other way around).Quoted 9 lines
It would successfully log to the tmp log file, but obviously, KVM would be disabled.
If I use sudo, it won't log to the file, is this a known issue?
2. -enable-nesting on AMD hardware using a xen guest OS causes xen to GPF somewhere in
svm_cpu_up. Is nesting supposed to work w/ Xen based guests?If your host kernel or kvm-kmod is not 2.6.32 based, update first. A lot of nested SVM fixes went in recently. If it still fails, put Alex (Graf) and Joerg (Roedel) on CC.
Also make sure you pass nested=1 to kvm-amd.ko.
Xen definitely worked for me, so you're probably just missing one of the many magic bits :-).
Alex
Quoted 15 lines · Jan Kiszka
Quoted 6 lines · Raindog
Hello,
I am researching KVM as a malware analysis platform and had some questions about debugging the guest OS. In my case I intend to use windows guests. So my questsions are as follows:
Quoted 7 lines
Questions:
1. What instrumentation facilities are their available?
2. Is it possible to extend the debugging interface so that debugging is more transparent to the guest OS? IE: there is still a limit of 4 HW breakpoints (which makes me wonder why a LIST is used for them...)
Quoted 7 lines
In accelerated KVM mode, the x86 architecture restricts us to 4 break- or watchpoints that can be active at the same time. If you switch to emulation mode, there are no such limits. Actually, I just made use of this for debugging a subtle stack corruption in a guest, and I had more than 70 watchpoints active at the same time. It's just "slightly" slower than KVM...
Are there any advantages over stock qemu if using kvm w/out the kernel module?
Quoted 2 lines · Raindog
Are there any advantages over stock qemu if using kvm w/out the kernel module?
No. qemu-kvm is not tested without kvm, so there may be disadvantages.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 6 lines · Avi Kivity
Quoted 2 lines · Raindog
Are there any advantages over stock qemu if using kvm w/out the kernel module?
No. qemu-kvm is not tested without kvm, so there may be disadvantages.
Does that then imply that svm emulation (--enable-nesting) is not well tested either?
Quoted 9 lines · Raindog
Quoted 6 lines · Avi Kivity
Quoted 2 lines · Raindog
Are there any advantages over stock qemu if using kvm w/out the kernel module?
No. qemu-kvm is not tested without kvm, so there may be disadvantages.
Does that then imply that svm emulation (--enable-nesting) is not well tested either?
It's an unrelated feature (but I wouldn't say it is heavily tested).
Signature · 1 line
error compiling committee.c: too many arguments to function