help? need to enable VT-d for 2.6.27
I'm in a bit of an odd situation. We run an embedded version of linux on many different boards with different vendor agreements so it's painful to upgrade kernel versions.
I've been asked to provide VT-d support for KVM on our kernel (primarily for mapping NIC VFs into the guests), and I'm trying to figure out my best option to get a working implementation.
Looking at the kvm git tree, kvm-77 seems to be based off 2.6.27. Should I look at porting that?
Would I be better off just scanning for anything VT-d related in 2.6.28 and backporting it?
Are there any better options? Any advice you could provide would be appreciated.
Thanks,
Chris
Signature · 5 lines
Chris Friesen Software Developer GENBAND [email protected] www.genband.com
Quoted 8 lines · Chris Friesen
I'm in a bit of an odd situation. We run an embedded version of linux on many different boards with different vendor agreements so it's painful to upgrade kernel versions.
I've been asked to provide VT-d support for KVM on our kernel (primarily for mapping NIC VFs into the guests), and I'm trying to figure out my best option to get a working implementation.
I've had pains with VF passthrough with as late as a Fedora 14 kernel (2.6.35). Ended up punting and moving to 2.6.39.
Quoted 3 lines
Looking at the kvm git tree, kvm-77 seems to be based off 2.6.27. Should I look at porting that?
By 2.6.27 do you mean WRL3 or something else?
David
Quoted 11 lines
Would I be better off just scanning for anything VT-d related in 2.6.28 and backporting it?
Are there any better options? Any advice you could provide would be appreciated.
Thanks,
Chris
Quoted 6 lines · David Ahern
Quoted 5 lines · Chris Friesen
I'm in a bit of an odd situation. We run an embedded version of linux on many different boards with different vendor agreements so it's painful to upgrade kernel versions.
Quoted 6 lines
Quoted 3 lines
I've been asked to provide VT-d support for KVM on our kernel (primarily for mapping NIC VFs into the guests), and I'm trying to figure out my best option to get a working implementation.
I've had pains with VF passthrough with as late as a Fedora 14 kernel (2.6.35). Ended up punting and moving to 2.6.39.
Ouch. Not good news, but not entirely unexpected. Are you doing anything complicated, or was this just basic functionality that wasn't working?
Quoted 4 lines
Quoted 2 lines
Looking at the kvm git tree, kvm-77 seems to be based off 2.6.27. Should I look at porting that?
By 2.6.27 do you mean WRL3 or something else?
Yes, that's correct. (Wind River Linux 3, for the non-embedded types out there.)
Chris
Signature · 5 lines
Chris Friesen Software Developer GENBAND [email protected] www.genband.com
Quoted 11 lines · Chris Friesen
Quoted 10 lines · David Ahern
Quoted 8 lines · Chris Friesen
I'm in a bit of an odd situation. We run an embedded version of linux on many different boards with different vendor agreements so it's painful to upgrade kernel versions.
I've been asked to provide VT-d support for KVM on our kernel (primarily for mapping NIC VFs into the guests), and I'm trying to figure out my best option to get a working implementation.
Quoted 6 lines
Quoted 2 lines
I've had pains with VF passthrough with as late as a Fedora 14 kernel (2.6.35). Ended up punting and moving to 2.6.39.
Ouch. Not good news, but not entirely unexpected. Are you doing anything complicated, or was this just basic functionality that wasn't working?
In my case I am setting the MAC address for the VFs in the host at boot; they are getting reset on passthrough so the VM driver creates a random one. A backport of commit a6b5ea3 fixes one aspect. Looking at the code for the igb driver I think commit 8fa7e0f fixes the other reset problem.
On the plus side I can tell you that the VF driver for the VM
http://sourceforge.net/projects/e1000/files/igbvf%20stable/
compiles cleaning against WRL3 and appears to work fine (limited testing with benchmarks).
kernel module wise I ran some basic KVM tests with kvm-kmod-2.6.32.27; I have not tried VF passthrough. This is on my radar as well, so I would be interested in what you learn.
David
Quoted 11 lines
Quoted 5 lines
Quoted 3 lines
Looking at the kvm git tree, kvm-77 seems to be based off 2.6.27. Should I look at porting that?
By 2.6.27 do you mean WRL3 or something else?
Yes, that's correct. (Wind River Linux 3, for the non-embedded types out there.)
Chris
Quoted 8 lines · David Ahern
In my case I am setting the MAC address for the VFs in the host at boot; they are getting reset on passthrough so the VM driver creates a random one. A backport of commit a6b5ea3 fixes one aspect. Looking at the code for the igb driver I think commit 8fa7e0f fixes the other reset problem.
On the plus side I can tell you that the VF driver for the VM
http://sourceforge.net/projects/e1000/files/igbvf%20stable/
Quoted 6 lines
compiles cleaning against WRL3 and appears to work fine (limited testing with benchmarks).
kernel module wise I ran some basic KVM tests with kvm-kmod-2.6.32.27; I have not tried VF passthrough. This is on my radar as well, so I would be interested in what you learn.
I downloaded the latest kvm-kmod release. It looks interesting, but it has some issues:
1) I use separate source and build output directories. kvm-kmod assumes that there is a "source" link in my build output directory pointing back to the source directory. This doesn't exist, so the build breaks.
2) If I create such a link, it builds a few files, and then gives the following:
/home/cfriesen/Download/kvm-kmod-3.0b/x86/x86.c: In function ‘vcpu_enter_guest’:/home/cfriesen/Download/kvm-kmod-3.0b/x86/x86.c:5539:26: error: ‘prof_on’ undeclared (first use in this function) /home/cfriesen/Download/kvm-kmod-3.0b/x86/x86.c:5539:26: note: each undeclared identifier is reported only once for each function it appears in
This appears to be due to the fact that KVM_PROFILING is defined unconditionally in my kernel, but "prof_on" is only defined if CONFIG_PROFILING is enabled. Commenting out that chunk of code in x86.c lets everything build.
I haven't tried actually loading it yet, that will be next.
Thanks for the pointer.
Chris
Signature · 5 lines
Chris Friesen Software Developer GENBAND [email protected] www.genband.com
Quoted 35 lines · Chris Friesen
Quoted 8 lines · David Ahern
In my case I am setting the MAC address for the VFs in the host at boot; they are getting reset on passthrough so the VM driver creates a random one. A backport of commit a6b5ea3 fixes one aspect. Looking at the code for the igb driver I think commit 8fa7e0f fixes the other reset problem.
On the plus side I can tell you that the VF driver for the VM
http://sourceforge.net/projects/e1000/files/igbvf%20stable/Quoted 6 lines
compiles cleaning against WRL3 and appears to work fine (limited testing with benchmarks).
kernel module wise I ran some basic KVM tests with kvm-kmod-2.6.32.27; I have not tried VF passthrough. This is on my radar as well, so I would be interested in what you learn.
I downloaded the latest kvm-kmod release. It looks interesting, but it has some issues:
1) I use separate source and build output directories. kvm-kmod assumes that there is a "source" link in my build output directory pointing back to the source directory. This doesn't exist, so the build breaks.
2) If I create such a link, it builds a few files, and then gives the following:
/home/cfriesen/Download/kvm-kmod-3.0b/x86/x86.c: In function ‘vcpu_enter_guest’:/home/cfriesen/Download/kvm-kmod-3.0b/x86/x86.c:5539:26: error:
‘prof_on’ undeclared (first use in this function)/home/cfriesen/Download/kvm-kmod-3.0b/x86/x86.c:5539:26: note: each undeclared identifier is reported only once for each function it appears in
Quoted 4 lines
This appears to be due to the fact that
KVM_PROFILINGis defined unconditionally in my kernel, but "prof_on" is only defined ifCONFIG_PROFILINGis enabled. Commenting out that chunk of code inx86.clets everything build.
I was a bit nervous trying kmod's from that much ahead of 2.6.27; that's why I picked the 2.6.32 train.
Also, the notes for adding the device assignment to Fedora 11 might come in handy:
http://fedoraproject.org/wiki/Features/KVM_PCI_Device_Assignment
I have not walked through the bug references yet, but Fedora 11 is not too far ahead of WRL3. When I backported perf to WRL3 I used Fedora 10 as a reference. For VT-d, Fedora 11 would be interesting to look at -- is the support complete in F-11 and if so take a look at its kernel source.
David
Quoted 7 lines
I haven't tried actually loading it yet, that will be next.
Thanks for the pointer.
Chris
Quoted 17 lines · Chris Friesen
Quoted 8 lines · David Ahern
In my case I am setting the MAC address for the VFs in the host at boot; they are getting reset on passthrough so the VM driver creates a random one. A backport of commit a6b5ea3 fixes one aspect. Looking at the code for the igb driver I think commit 8fa7e0f fixes the other reset problem.
On the plus side I can tell you that the VF driver for the VM
http://sourceforge.net/projects/e1000/files/igbvf%20stable/Quoted 6 lines
compiles cleaning against WRL3 and appears to work fine (limited testing with benchmarks).
kernel module wise I ran some basic KVM tests with kvm-kmod-2.6.32.27; I have not tried VF passthrough. This is on my radar as well, so I would be interested in what you learn.
Quoted 6 lines
I downloaded the latest kvm-kmod release. It looks interesting, but it has some issues:
1) I use separate source and build output directories. kvm-kmod assumes that there is a "source" link in my build output directory pointing back to the source directory. This doesn't exist, so the build breaks.
That's at least how recent vanilla kernels work. I think to remember that my old OpenSUSE 11.1 (2.6.27) used to work fine with kvm-kmod as well, and that was an out-of-tree setup too (but maybe a post-processed one).
Anyway, if you have some patch to fix this, it will always be welcome.
Quoted 12 lines
2) If I create such a link, it builds a few files, and then gives the following:
/home/cfriesen/Download/kvm-kmod-3.0b/x86/x86.c: In function ‘vcpu_enter_guest’:/home/cfriesen/Download/kvm-kmod-3.0b/x86/x86.c:5539:26: error:
‘prof_on’ undeclared (first use in this function)/home/cfriesen/Download/kvm-kmod-3.0b/x86/x86.c:5539:26: note: each undeclared identifier is reported only once for each function it appears in
Quoted 4 lines
This appears to be due to the fact that
KVM_PROFILINGis defined unconditionally in my kernel, but "prof_on" is only defined ifCONFIG_PROFILINGis enabled. Commenting out that chunk of code inx86.clets everything build.
If CONFIG_PROFILING is off, prof_on is defined to 0. But we may miss some include on older kernels. Does adding linux/profile.h to x86.c solve the issue as well?
Jan
Quoted 13 lines · Jan Kiszka
Quoted 11 lines · Chris Friesen
2) If I create such a link, it builds a few files, and then gives the following:
/home/cfriesen/Download/kvm-kmod-3.0b/x86/x86.c: In function ‘vcpu_enter_guest’:/home/cfriesen/Download/kvm-kmod-3.0b/x86/x86.c:5539:26: error:
‘prof_on’ undeclared (first use in this function)/home/cfriesen/Download/kvm-kmod-3.0b/x86/x86.c:5539:26: note: each undeclared identifier is reported only once for each function it appears in
Quoted 8 lines
Quoted 4 lines
This appears to be due to the fact that
KVM_PROFILINGis defined unconditionally in my kernel, but "prof_on" is only defined ifCONFIG_PROFILINGis enabled. Commenting out that chunk of code inx86.clets everything build.If
CONFIG_PROFILINGis off,prof_onis defined to 0. But we may miss some include on older kernels. Does addinglinux/profile.htox86.csolve the issue as well?
It would appear that I got the explanation wrong. CONFIG_PROFILING is enabled on my kernel. The problem is that our OS vendor has included a patch to convert prof_on to immediate values, which requires a change in x86.c.
Never mind, nothing to see here.
Chris
Signature · 5 lines
Chris Friesen Software Developer GENBAND [email protected] www.genband.com
[BUG] error compiling qemu-kvm-0.15.0 without vnc
In reply to Jan KiszkaHi,
I've run into another problem. I configured qemu-kvm-0.15.0 as:
./configure --target-list="i386-softmmu,x86_64-softmmu" --disable-sdl --disable-vnc --disable-curses
Building it, I get:
CC i386-softmmu/pcspk.o CC i386-softmmu/i8254.o CC i386-softmmu/i8254-kvm.o CC i386-softmmu/device-assignment.o LINK i386-softmmu/qemu
monitor.o: In function `add_graphics_client':/home/cfriesen/Download/qemu-kvm-0.15.0/monitor.c:1226: undefined reference to `vnc_display_add_client'
collect2: ld returned 1 exit status
make[1]: *** [qemu] Error 1
make: *** [subdir-i386-softmmu] Error 2
The following patch allowed it to compile, but I don't know enough about kvm to know if this is the proper thing to do.
Chris
Download/qemu-kvm-0.15.0/console.h +4 -1
@@ -372,15 +372,18 @@ void cocoa_display_init(DisplayState *dsvoid vnc_display_init (DisplayState *ds);void vnc_display_close (DisplayState *ds);int vnc_display_open (DisplayState *ds,const char *display); -void vnc_display_add_client (DisplayState *ds,int csock,int skipauth);int vnc_display_disable_login (DisplayState *ds);char *vnc_display_local_addr (DisplayState *ds);#ifdef CONFIG_VNC +void vnc_display_add_client (DisplayState *ds,int csock,int skipauth);int vnc_display_password (DisplayState *ds,const char *password);int vnc_display_pw_expire (DisplayState *ds,time_t expires);void do_info_vnc_print (Monitor *mon,const QObject *data);void do_info_vnc (Monitor *mon,QObject **ret_data);#else +static inline void vnc_display_add_client (DisplayState *ds,int csock,int skipauth) +{ +}static inline int vnc_display_password (DisplayState *ds,const char *password) {qerror_report (QERR_FEATURE_DISABLED ,"vnc" );
Signature · 5 lines
Chris Friesen Software Developer GENBAND [email protected] www.genband.com
Quoted 25 lines · Chris Friesen
Quoted 14 lines · Jan Kiszka
Quoted 12 lines · Chris Friesen
2) If I create such a link, it builds a few files, and then gives the following:
/home/cfriesen/Download/kvm-kmod-3.0b/x86/x86.c: In function ‘vcpu_enter_guest’:/home/cfriesen/Download/kvm-kmod-3.0b/x86/x86.c:5539:26: error:
‘prof_on’ undeclared (first use in this function)/home/cfriesen/Download/kvm-kmod-3.0b/x86/x86.c:5539:26: note: each undeclared identifier is reported only once for each function it appears inQuoted 8 lines
Quoted 4 lines
This appears to be due to the fact that
KVM_PROFILINGis defined unconditionally in my kernel, but "prof_on" is only defined ifCONFIG_PROFILINGis enabled. Commenting out that chunk of code inx86.clets everything build.If
CONFIG_PROFILINGis off,prof_onis defined to 0. But we may miss some include on older kernels. Does addinglinux/profile.htox86.csolve the issue as well?
Quoted 4 lines
It would appear that I got the explanation wrong.
CONFIG_PROFILINGis enabled on my kernel. The problem is that our OS vendor has included a patch to convertprof_onto immediate values, which requires a change inx86.c.
Ah, now I remember: LTTng fallouts. I think there was just some EXPORT_SYMBOL missing in the kernel, but my memory is weak.
Jan
Re: [BUG] error compiling qemu-kvm-0.15.0 without vnc
In reply to Chris FriesenQuoted 44 lines · Chris Friesen
Hi,
I've run into another problem. I configured qemu-kvm-0.15.0 as:
./configure --target-list="i386-softmmu,x86_64-softmmu" --disable-sdl --disable-vnc --disable-curses
Building it, I get:
CC i386-softmmu/pcspk.o CC i386-softmmu/i8254.o CC i386-softmmu/i8254-kvm.o CC i386-softmmu/device-assignment.o LINK i386-softmmu/qemumonitor.o: In function `add_graphics_client':
/home/cfriesen/Download/qemu-kvm-0.15.0/monitor.c:1226: undefined reference to `vnc_display_add_client'
collect2: ld returned 1 exit status
make[1]: *** [qemu] Error 1
make: *** [subdir-i386-softmmu] Error 2The following patch allowed it to compile, but I don't know enough about kvm to know if this is the proper thing to do.
Chris
Index:
cfriesen/Download/qemu-kvm-0.15.0/console.h
===================================================================Download/qemu-kvm-0.15.0/console.h +1
-1@@ -372,15 +372,18 @@ void cocoa_display_init(DisplayState *dsvoid vnc_display_init (DisplayState *ds);void vnc_display_close (DisplayState *ds);int vnc_display_open (DisplayState *ds,const char *display); -void vnc_display_add_client (DisplayState *ds,int csock,int skipauth);int vnc_display_disable_login (DisplayState *ds);char *vnc_display_local_addr (DisplayState *ds);#ifdef CONFIG_VNC +void vnc_display_add_client (DisplayState *ds,int csock,int skipauth);int vnc_display_password (DisplayState *ds,const char *password);
Download/qemu-kvm-0.15.0/console.h void cocoa_display_init(DisplayState *ds · line 389
Quoted 10 lines
Download/qemu-kvm-0.15.0/console.h +3
-0int vnc_display_pw_expire (DisplayState *ds,time_t expires);void do_info_vnc_print (Monitor *mon,const QObject *data);void do_info_vnc (Monitor *mon,QObject **ret_data);#else +static inline void vnc_display_add_client (DisplayState *ds,int csock,int skipauth) +{ +}static inline int vnc_display_password (DisplayState *ds,const char *password) {qerror_report (QERR_FEATURE_DISABLED ,"vnc" );
c62f6d1d76aea587556c85b6b7b5c44167006264 and 860341f60582959698d2e1d839a5b7a004a2d76f need to be applied to stable-0.15 of upstream QEMU.
Jan
Re: [BUG] error compiling qemu-kvm-0.15.0 without vnc
Am 31.08.2011 09:40, schrieb Jan Kiszka:
Quoted 51 lines · Chris Friesen
Quoted 50 lines
Hi,
I've run into another problem. I configured qemu-kvm-0.15.0 as:
./configure --target-list="i386-softmmu,x86_64-softmmu" --disable-sdl --disable-vnc --disable-curses
Building it, I get:
CC i386-softmmu/pcspk.o CC i386-softmmu/i8254.o CC i386-softmmu/i8254-kvm.o CC i386-softmmu/device-assignment.o LINK i386-softmmu/qemumonitor.o: In function `add_graphics_client':
/home/cfriesen/Download/qemu-kvm-0.15.0/monitor.c:1226: undefined reference to `vnc_display_add_client'
collect2: ld returned 1 exit status
make[1]: *** [qemu] Error 1
make: *** [subdir-i386-softmmu] Error 2The following patch allowed it to compile, but I don't know enough about kvm to know if this is the proper thing to do.
Chris
Index:
cfriesen/Download/qemu-kvm-0.15.0/console.h
===================================================================Download/qemu-kvm-0.15.0/console.h +4
-1@@ -372,15 +372,18 @@ void cocoa_display_init(DisplayState *dsvoid vnc_display_init (DisplayState *ds);void vnc_display_close (DisplayState *ds);int vnc_display_open (DisplayState *ds,const char *display); -void vnc_display_add_client (DisplayState *ds,int csock,int skipauth);int vnc_display_disable_login (DisplayState *ds);char *vnc_display_local_addr (DisplayState *ds);#ifdef CONFIG_VNC +void vnc_display_add_client (DisplayState *ds,int csock,int skipauth);int vnc_display_password (DisplayState *ds,const char *password);int vnc_display_pw_expire (DisplayState *ds,time_t expires);void do_info_vnc_print (Monitor *mon,const QObject *data);void do_info_vnc (Monitor *mon,QObject **ret_data);#else +static inline void vnc_display_add_client (DisplayState *ds,int csock,int skipauth) +{ +}static inline int vnc_display_password (DisplayState *ds,const char *password) {
Quoted 8 lines
Download/qemu-kvm-0.15.0/console.h void cocoa_display_init(DisplayState *ds · line 389
Quoted 4 lines
Download/qemu-kvm-0.15.0/console.h +0
-0qerror_report (QERR_FEATURE_DISABLED ,"vnc" );
c62f6d1d76aea587556c85b6b7b5c44167006264and860341f60582959698d2e1d839a5b7a004a2d76fneed to be applied to stable-0.15 of upstream QEMU.
So I guess Justin should be CCed?
Kevin
Re: [Qemu-devel] [BUG] error compiling qemu-kvm-0.15.0 without vnc
Quoted 10 lines · Kevin Wolf
Am 31.08.2011 09:40, schrieb Jan Kiszka:
Quoted 5 lines · Chris Friesen
c62f6d1d76aea587556c85b6b7b5c44167006264and860341f60582959698d2e1d839a5b7a004a2d76fneed to be applied to stable-0.15 of upstream QEMU.So I guess Justin should be CCed?
Kevin
Thanks for the CC, and pulled.
Justin