Skip to contentSkip to conversation

help? need to enable VT-d for 2.6.27

DISCUSSION Chris Friesen 5 participants 13 messages Aug 30, 2011 – Aug 31, 2011 KVM

lore
Share & tools
lore
#0 lore raw

I'm in a bit of an odd situation. We run an embedded version of linux on many different boards with different vendor agreements so it's painful to upgrade kernel versions.

I've been asked to provide VT-d support for KVM on our kernel (primarily for mapping NIC VFs into the guests), and I'm trying to figure out my best option to get a working implementation.

Looking at the kvm git tree, kvm-77 seems to be based off 2.6.27. Should I look at porting that?

Would I be better off just scanning for anything VT-d related in 2.6.28 and backporting it?

Are there any better options? Any advice you could provide would be appreciated.

Thanks,
Chris

Signature · 5 lines
Chris Friesen
Software Developer
GENBAND
[email protected]
www.genband.com
#1 lore raw
Quoted 8 lines · Chris Friesen

I'm in a bit of an odd situation. We run an embedded version of linux on many different boards with different vendor agreements so it's painful to upgrade kernel versions.

I've been asked to provide VT-d support for KVM on our kernel (primarily for mapping NIC VFs into the guests), and I'm trying to figure out my best option to get a working implementation.

I've had pains with VF passthrough with as late as a Fedora 14 kernel (2.6.35). Ended up punting and moving to 2.6.39.

Quoted 3 lines

Looking at the kvm git tree, kvm-77 seems to be based off 2.6.27. Should I look at porting that?

By 2.6.27 do you mean WRL3 or something else?

David

Quoted 11 lines

Would I be better off just scanning for anything VT-d related in 2.6.28 and backporting it?

Are there any better options? Any advice you could provide would be appreciated.

Thanks,
Chris

#2 lore raw
Quoted 6 lines · David Ahern
Quoted 5 lines · Chris Friesen

I'm in a bit of an odd situation. We run an embedded version of linux on many different boards with different vendor agreements so it's painful to upgrade kernel versions.

Quoted 6 lines
Quoted 3 lines

I've been asked to provide VT-d support for KVM on our kernel (primarily for mapping NIC VFs into the guests), and I'm trying to figure out my best option to get a working implementation.

I've had pains with VF passthrough with as late as a Fedora 14 kernel (2.6.35). Ended up punting and moving to 2.6.39.

Ouch. Not good news, but not entirely unexpected. Are you doing anything complicated, or was this just basic functionality that wasn't working?

Quoted 4 lines
Quoted 2 lines

Looking at the kvm git tree, kvm-77 seems to be based off 2.6.27. Should I look at porting that?

By 2.6.27 do you mean WRL3 or something else?

Yes, that's correct. (Wind River Linux 3, for the non-embedded types out there.)

Chris

Signature · 5 lines
Chris Friesen
Software Developer
GENBAND
[email protected]
www.genband.com
#3 lore raw
Quoted 11 lines · Chris Friesen
Quoted 10 lines · David Ahern
Quoted 8 lines · Chris Friesen

I'm in a bit of an odd situation. We run an embedded version of linux on many different boards with different vendor agreements so it's painful to upgrade kernel versions.

I've been asked to provide VT-d support for KVM on our kernel (primarily for mapping NIC VFs into the guests), and I'm trying to figure out my best option to get a working implementation.

Quoted 6 lines
Quoted 2 lines

I've had pains with VF passthrough with as late as a Fedora 14 kernel (2.6.35). Ended up punting and moving to 2.6.39.

Ouch. Not good news, but not entirely unexpected. Are you doing anything complicated, or was this just basic functionality that wasn't working?

In my case I am setting the MAC address for the VFs in the host at boot; they are getting reset on passthrough so the VM driver creates a random one. A backport of commit a6b5ea3 fixes one aspect. Looking at the code for the igb driver I think commit 8fa7e0f fixes the other reset problem.

On the plus side I can tell you that the VF driver for the VM
http://sourceforge.net/projects/e1000/files/igbvf%20stable/

compiles cleaning against WRL3 and appears to work fine (limited testing with benchmarks).

kernel module wise I ran some basic KVM tests with kvm-kmod-2.6.32.27; I have not tried VF passthrough. This is on my radar as well, so I would be interested in what you learn.

David

Quoted 11 lines
Quoted 5 lines
Quoted 3 lines

Looking at the kvm git tree, kvm-77 seems to be based off 2.6.27. Should I look at porting that?

By 2.6.27 do you mean WRL3 or something else?

Yes, that's correct. (Wind River Linux 3, for the non-embedded types out there.)

Chris

#4 lore raw
Quoted 8 lines · David Ahern

In my case I am setting the MAC address for the VFs in the host at boot; they are getting reset on passthrough so the VM driver creates a random one. A backport of commit a6b5ea3 fixes one aspect. Looking at the code for the igb driver I think commit 8fa7e0f fixes the other reset problem.

On the plus side I can tell you that the VF driver for the VM
http://sourceforge.net/projects/e1000/files/igbvf%20stable/

Quoted 6 lines

compiles cleaning against WRL3 and appears to work fine (limited testing with benchmarks).

kernel module wise I ran some basic KVM tests with kvm-kmod-2.6.32.27; I have not tried VF passthrough. This is on my radar as well, so I would be interested in what you learn.

I downloaded the latest kvm-kmod release. It looks interesting, but it has some issues:

1) I use separate source and build output directories. kvm-kmod assumes that there is a "source" link in my build output directory pointing back to the source directory. This doesn't exist, so the build breaks.

2) If I create such a link, it builds a few files, and then gives the following:

/home/cfriesen/Download/kvm-kmod-3.0b/x86/x86.c: In function ‘vcpu_enter_guest’:
/home/cfriesen/Download/kvm-kmod-3.0b/x86/x86.c:5539:26: error: ‘prof_on’ undeclared (first use in this function) /home/cfriesen/Download/kvm-kmod-3.0b/x86/x86.c:5539:26: note: each undeclared identifier is reported only once for each function it appears in

This appears to be due to the fact that KVM_PROFILING is defined unconditionally in my kernel, but "prof_on" is only defined if CONFIG_PROFILING is enabled. Commenting out that chunk of code in x86.c lets everything build.

I haven't tried actually loading it yet, that will be next.

Thanks for the pointer.
Chris

Signature · 5 lines
Chris Friesen
Software Developer
GENBAND
[email protected]
www.genband.com
#5 lore raw
Quoted 35 lines · Chris Friesen
Quoted 8 lines · David Ahern

In my case I am setting the MAC address for the VFs in the host at boot; they are getting reset on passthrough so the VM driver creates a random one. A backport of commit a6b5ea3 fixes one aspect. Looking at the code for the igb driver I think commit 8fa7e0f fixes the other reset problem.

On the plus side I can tell you that the VF driver for the VM
http://sourceforge.net/projects/e1000/files/igbvf%20stable/

Quoted 6 lines

compiles cleaning against WRL3 and appears to work fine (limited testing with benchmarks).

kernel module wise I ran some basic KVM tests with kvm-kmod-2.6.32.27; I have not tried VF passthrough. This is on my radar as well, so I would be interested in what you learn.

I downloaded the latest kvm-kmod release. It looks interesting, but it has some issues:

1) I use separate source and build output directories. kvm-kmod assumes that there is a "source" link in my build output directory pointing back to the source directory. This doesn't exist, so the build breaks.

2) If I create such a link, it builds a few files, and then gives the following:

/home/cfriesen/Download/kvm-kmod-3.0b/x86/x86.c: In function ‘vcpu_enter_guest’:
/home/cfriesen/Download/kvm-kmod-3.0b/x86/x86.c:5539:26: error:
‘prof_on’ undeclared (first use in this function) /home/cfriesen/Download/kvm-kmod-3.0b/x86/x86.c:5539:26: note: each undeclared identifier is reported only once for each function it appears in

Quoted 4 lines

This appears to be due to the fact that KVM_PROFILING is defined unconditionally in my kernel, but "prof_on" is only defined if CONFIG_PROFILING is enabled. Commenting out that chunk of code in x86.c lets everything build.

I was a bit nervous trying kmod's from that much ahead of 2.6.27; that's why I picked the 2.6.32 train.

Also, the notes for adding the device assignment to Fedora 11 might come in handy:
http://fedoraproject.org/wiki/Features/KVM_PCI_Device_Assignment

I have not walked through the bug references yet, but Fedora 11 is not too far ahead of WRL3. When I backported perf to WRL3 I used Fedora 10 as a reference. For VT-d, Fedora 11 would be interesting to look at -- is the support complete in F-11 and if so take a look at its kernel source.

David

Quoted 7 lines

I haven't tried actually loading it yet, that will be next.

Thanks for the pointer.
Chris

#6 lore raw
In reply to Chris Friesen
Quoted 17 lines · Chris Friesen
Quoted 8 lines · David Ahern

In my case I am setting the MAC address for the VFs in the host at boot; they are getting reset on passthrough so the VM driver creates a random one. A backport of commit a6b5ea3 fixes one aspect. Looking at the code for the igb driver I think commit 8fa7e0f fixes the other reset problem.

On the plus side I can tell you that the VF driver for the VM
http://sourceforge.net/projects/e1000/files/igbvf%20stable/

Quoted 6 lines

compiles cleaning against WRL3 and appears to work fine (limited testing with benchmarks).

kernel module wise I ran some basic KVM tests with kvm-kmod-2.6.32.27; I have not tried VF passthrough. This is on my radar as well, so I would be interested in what you learn.

Quoted 6 lines

I downloaded the latest kvm-kmod release. It looks interesting, but it has some issues:

1) I use separate source and build output directories. kvm-kmod assumes that there is a "source" link in my build output directory pointing back to the source directory. This doesn't exist, so the build breaks.

That's at least how recent vanilla kernels work. I think to remember that my old OpenSUSE 11.1 (2.6.27) used to work fine with kvm-kmod as well, and that was an out-of-tree setup too (but maybe a post-processed one).

Anyway, if you have some patch to fix this, it will always be welcome.

Quoted 12 lines

2) If I create such a link, it builds a few files, and then gives the following:

/home/cfriesen/Download/kvm-kmod-3.0b/x86/x86.c: In function ‘vcpu_enter_guest’:
/home/cfriesen/Download/kvm-kmod-3.0b/x86/x86.c:5539:26: error:
‘prof_on’ undeclared (first use in this function) /home/cfriesen/Download/kvm-kmod-3.0b/x86/x86.c:5539:26: note: each undeclared identifier is reported only once for each function it appears in

Quoted 4 lines

This appears to be due to the fact that KVM_PROFILING is defined unconditionally in my kernel, but "prof_on" is only defined if CONFIG_PROFILING is enabled. Commenting out that chunk of code in x86.c lets everything build.

If CONFIG_PROFILING is off, prof_on is defined to 0. But we may miss some include on older kernels. Does adding linux/profile.h to x86.c solve the issue as well?

Jan

#7 lore raw
Quoted 13 lines · Jan Kiszka
Quoted 11 lines · Chris Friesen

2) If I create such a link, it builds a few files, and then gives the following:

/home/cfriesen/Download/kvm-kmod-3.0b/x86/x86.c: In function ‘vcpu_enter_guest’:
/home/cfriesen/Download/kvm-kmod-3.0b/x86/x86.c:5539:26: error:
‘prof_on’ undeclared (first use in this function) /home/cfriesen/Download/kvm-kmod-3.0b/x86/x86.c:5539:26: note: each undeclared identifier is reported only once for each function it appears in

Quoted 8 lines
Quoted 4 lines

This appears to be due to the fact that KVM_PROFILING is defined unconditionally in my kernel, but "prof_on" is only defined if CONFIG_PROFILING is enabled. Commenting out that chunk of code in x86.c lets everything build.

If CONFIG_PROFILING is off, prof_on is defined to 0. But we may miss some include on older kernels. Does adding linux/profile.h to x86.c solve the issue as well?

It would appear that I got the explanation wrong. CONFIG_PROFILING is enabled on my kernel. The problem is that our OS vendor has included a patch to convert prof_on to immediate values, which requires a change in x86.c.

Never mind, nothing to see here.

Chris

Signature · 5 lines
Chris Friesen
Software Developer
GENBAND
[email protected]
www.genband.com
#8 lore raw
Quoted 25 lines · Chris Friesen
Quoted 14 lines · Jan Kiszka
Quoted 12 lines · Chris Friesen

2) If I create such a link, it builds a few files, and then gives the following:

/home/cfriesen/Download/kvm-kmod-3.0b/x86/x86.c: In function ‘vcpu_enter_guest’:
/home/cfriesen/Download/kvm-kmod-3.0b/x86/x86.c:5539:26: error:
‘prof_on’ undeclared (first use in this function) /home/cfriesen/Download/kvm-kmod-3.0b/x86/x86.c:5539:26: note: each undeclared identifier is reported only once for each function it appears in

Quoted 8 lines
Quoted 4 lines

This appears to be due to the fact that KVM_PROFILING is defined unconditionally in my kernel, but "prof_on" is only defined if CONFIG_PROFILING is enabled. Commenting out that chunk of code in x86.c lets everything build.

If CONFIG_PROFILING is off, prof_on is defined to 0. But we may miss some include on older kernels. Does adding linux/profile.h to x86.c solve the issue as well?

Quoted 4 lines

It would appear that I got the explanation wrong. CONFIG_PROFILING is enabled on my kernel. The problem is that our OS vendor has included a patch to convert prof_on to immediate values, which requires a change in x86.c.

Ah, now I remember: LTTng fallouts. I think there was just some EXPORT_SYMBOL missing in the kernel, but my memory is weak.

Jan

#9 lore raw

[BUG] error compiling qemu-kvm-0.15.0 without vnc

In reply to Jan Kiszka

Hi,

I've run into another problem. I configured qemu-kvm-0.15.0 as:

./configure --target-list="i386-softmmu,x86_64-softmmu" --disable-sdl --disable-vnc --disable-curses

Building it, I get:

  CC    i386-softmmu/pcspk.o
  CC    i386-softmmu/i8254.o
  CC    i386-softmmu/i8254-kvm.o
  CC    i386-softmmu/device-assignment.o
  LINK  i386-softmmu/qemu

monitor.o: In function `add_graphics_client':
/home/cfriesen/Download/qemu-kvm-0.15.0/monitor.c:1226: undefined reference to `vnc_display_add_client'
collect2: ld returned 1 exit status
make[1]: *** [qemu] Error 1
make: *** [subdir-i386-softmmu] Error 2

The following patch allowed it to compile, but I don't know enough about kvm to know if this is the proper thing to do.

Chris

Download/qemu-kvm-0.15.0/console.h +4 -1
Index: cfriesen/Download/qemu-kvm-0.15.0/console.h
===================================================================
@@ -372,15 +372,18 @@ void cocoa_display_init(DisplayState *ds
 void vnc_display_init(DisplayState *ds);
 void vnc_display_close(DisplayState *ds);
 int vnc_display_open(DisplayState *ds, const char *display);
-void vnc_display_add_client(DisplayState *ds, int csock, int skipauth);
 int vnc_display_disable_login(DisplayState *ds);
 char *vnc_display_local_addr(DisplayState *ds);
 #ifdef CONFIG_VNC
+void vnc_display_add_client(DisplayState *ds, int csock, int skipauth);
 int vnc_display_password(DisplayState *ds, const char *password);
 int vnc_display_pw_expire(DisplayState *ds, time_t expires);
 void do_info_vnc_print(Monitor *mon, const QObject *data);
 void do_info_vnc(Monitor *mon, QObject **ret_data);
 #else
+static inline void vnc_display_add_client(DisplayState *ds, int csock, int skipauth)
+{
+}
 static inline int vnc_display_password(DisplayState *ds, const char *password)
 {
     qerror_report(QERR_FEATURE_DISABLED, "vnc");
Signature · 5 lines
Chris Friesen
Software Developer
GENBAND
[email protected]
www.genband.com
#10 lore raw

Re: [BUG] error compiling qemu-kvm-0.15.0 without vnc

Quoted 44 lines · Chris Friesen

Hi,

I've run into another problem. I configured qemu-kvm-0.15.0 as:

./configure --target-list="i386-softmmu,x86_64-softmmu" --disable-sdl --disable-vnc --disable-curses

Building it, I get:

  CC    i386-softmmu/pcspk.o
  CC    i386-softmmu/i8254.o
  CC    i386-softmmu/i8254-kvm.o
  CC    i386-softmmu/device-assignment.o
  LINK  i386-softmmu/qemu

monitor.o: In function `add_graphics_client':
/home/cfriesen/Download/qemu-kvm-0.15.0/monitor.c:1226: undefined reference to `vnc_display_add_client'
collect2: ld returned 1 exit status
make[1]: *** [qemu] Error 1
make: *** [subdir-i386-softmmu] Error 2

The following patch allowed it to compile, but I don't know enough about kvm to know if this is the proper thing to do.

Chris

Index: cfriesen/Download/qemu-kvm-0.15.0/console.h
===================================================================

Download/qemu-kvm-0.15.0/console.h +1 -1
@@ -372,15 +372,18 @@ void cocoa_display_init(DisplayState *ds
 void vnc_display_init(DisplayState *ds);
 void vnc_display_close(DisplayState *ds);
 int vnc_display_open(DisplayState *ds, const char *display);
-void vnc_display_add_client(DisplayState *ds, int csock, int skipauth);
 int vnc_display_disable_login(DisplayState *ds);
 char *vnc_display_local_addr(DisplayState *ds);
 #ifdef CONFIG_VNC
+void vnc_display_add_client(DisplayState *ds, int csock, int skipauth);
 int vnc_display_password(DisplayState *ds, const char *password);

Download/qemu-kvm-0.15.0/console.h void cocoa_display_init(DisplayState *ds · line 389

Quoted 10 lines
Download/qemu-kvm-0.15.0/console.h +3 -0
 int vnc_display_pw_expire(DisplayState *ds, time_t expires);
 void do_info_vnc_print(Monitor *mon, const QObject *data);
 void do_info_vnc(Monitor *mon, QObject **ret_data);
 #else
+static inline void vnc_display_add_client(DisplayState *ds, int csock, int skipauth)
+{
+}
 static inline int vnc_display_password(DisplayState *ds, const char *password)
 {
     qerror_report(QERR_FEATURE_DISABLED, "vnc");

c62f6d1d76aea587556c85b6b7b5c44167006264 and 860341f60582959698d2e1d839a5b7a004a2d76f need to be applied to stable-0.15 of upstream QEMU.

Jan

#11 lore raw

Re: [BUG] error compiling qemu-kvm-0.15.0 without vnc

Am 31.08.2011 09:40, schrieb Jan Kiszka:

Quoted 51 lines · Chris Friesen
Quoted 50 lines

Hi,

I've run into another problem. I configured qemu-kvm-0.15.0 as:

./configure --target-list="i386-softmmu,x86_64-softmmu" --disable-sdl --disable-vnc --disable-curses

Building it, I get:

  CC    i386-softmmu/pcspk.o
  CC    i386-softmmu/i8254.o
  CC    i386-softmmu/i8254-kvm.o
  CC    i386-softmmu/device-assignment.o
  LINK  i386-softmmu/qemu

monitor.o: In function `add_graphics_client':
/home/cfriesen/Download/qemu-kvm-0.15.0/monitor.c:1226: undefined reference to `vnc_display_add_client'
collect2: ld returned 1 exit status
make[1]: *** [qemu] Error 1
make: *** [subdir-i386-softmmu] Error 2

The following patch allowed it to compile, but I don't know enough about kvm to know if this is the proper thing to do.

Chris

Index: cfriesen/Download/qemu-kvm-0.15.0/console.h
===================================================================

Download/qemu-kvm-0.15.0/console.h +4 -1
@@ -372,15 +372,18 @@ void cocoa_display_init(DisplayState *ds
 void vnc_display_init(DisplayState *ds);
 void vnc_display_close(DisplayState *ds);
 int vnc_display_open(DisplayState *ds, const char *display);
-void vnc_display_add_client(DisplayState *ds, int csock, int skipauth);
 int vnc_display_disable_login(DisplayState *ds);
 char *vnc_display_local_addr(DisplayState *ds);
 #ifdef CONFIG_VNC
+void vnc_display_add_client(DisplayState *ds, int csock, int skipauth);
 int vnc_display_password(DisplayState *ds, const char *password);
 int vnc_display_pw_expire(DisplayState *ds, time_t expires);
 void do_info_vnc_print(Monitor *mon, const QObject *data);
 void do_info_vnc(Monitor *mon, QObject **ret_data);
 #else
+static inline void vnc_display_add_client(DisplayState *ds, int csock, int skipauth)
+{
+}
 static inline int vnc_display_password(DisplayState *ds, const char *password)
 {
Quoted 8 lines

Download/qemu-kvm-0.15.0/console.h void cocoa_display_init(DisplayState *ds · line 389

Quoted 4 lines
Download/qemu-kvm-0.15.0/console.h +0 -0
     qerror_report(QERR_FEATURE_DISABLED, "vnc");

c62f6d1d76aea587556c85b6b7b5c44167006264 and 860341f60582959698d2e1d839a5b7a004a2d76f need to be applied to stable-0.15 of upstream QEMU.

So I guess Justin should be CCed?

Kevin

#12 lore raw

Re: [Qemu-devel] [BUG] error compiling qemu-kvm-0.15.0 without vnc

Quoted 10 lines · Kevin Wolf

Am 31.08.2011 09:40, schrieb Jan Kiszka:

Quoted 5 lines · Chris Friesen

c62f6d1d76aea587556c85b6b7b5c44167006264 and 860341f60582959698d2e1d839a5b7a004a2d76f need to be applied to stable-0.15 of upstream QEMU.

So I guess Justin should be CCed?

Kevin

Thanks for the CC, and pulled.

Justin