pv-grub doesn't run on rhel5

Trying to boot pv-grub on RHEL-5 makes it fail with

[ ... ]
gnttab_table mapped at 0x10001000.
Initialising scheduler
(XEN) mm.c:694:d6 Bad L3 flags 6
ERROR: mmu_update failed
Do_exit called!

Guess more recent xen versions are less picky here and just silenly fixup this?

This is with 32bit pae.

Another issue: As 64bit pv-grub can't boot 32bit kernels, the 64bit xen build should create both 32bit and 64bit builds of pv-grub, so you can boot 32-on-64 guests with it?

Quoted 5 lines · Samuel Thibault

Quoted 2 lines

Guess more recent xen versions are less picky here and just silenly fixup this?

No, the check is still there, and I wonder how flags 6 could make it to L3 flags. Could you send me your mini-os.gz image?

Attached.

Quoted 3 lines · Gerd Hoffmann, le Thu 14 Aug 2008 12:38:15 +0200, a écrit :

Another issue: As 64bit pv-grub can't boot 32bit kernels, the 64bit xen build should create both 32bit and 64bit builds of pv-grub, so you can boot 32-on-64 guests with it?

I'm afraid Keir will consider it is too late in the 3.3 release to make such a change in the build process.

Quoted 6 lines · Gerd Hoffmann, le Thu 14 Aug 2008 14:20:59 +0200, a écrit :

Quoted 4 lines · Samuel Thibault

Quoted 2 lines

Guess more recent xen versions are less picky here and just silenly fixup this?

No, the check is still there, and I wonder how flags 6 could make it to L3 flags. Could you send me your mini-os.gz image?

Attached.

I have no problem running it under xen 3.3 indeed. Are you able to run the 3.3 mini-os itself for instance?

Quoted 6 lines · Samuel Thibault

Quoted 5 lines · Gerd Hoffmann, le Thu 14 Aug 2008 14:20:59 +0200, a écrit :

Quoted 4 lines · Samuel Thibault

Quoted 2 lines

Guess more recent xen versions are less picky here and just silenly fixup this?

No, the check is still there, and I wonder how flags 6 could make it to L3 flags. Could you send me your mini-os.gz image?

Quoted 4 lines

Quoted 1 line

Attached.

I have no problem running it under xen 3.3 indeed. Are you able to run the 3.3 mini-os itself for instance?

That runs fine. It doesn't touch address space above 1gb though, thus it has no need to touch the pae pgd.

The problem is a bug in the hypervisor's mod_l3_entry which calls adjust_guest_l3e before calling get_page_from_l3e, see c/s 17061. Is it possible for you to backport the patch?

Quoted 3 lines · Samuel Thibault

The problem is a bug in the hypervisor's mod_l3_entry which calls adjust_guest_l3e before calling get_page_from_l3e, see c/s 17061. Is it possible for you to backport the patch?

So due to the wrong ordering xen first sets the bits, then notices they shouldn't be set, then complains the guest passed invalid bits?

Uhm, well, no way to fix minios to run on older xen versions then I guess (which I'd prefer). Except maybe limiting the 32bit version to 1GB address space. Which should be fine at least for pvgrub ...

Another pvgrub issue: Is booting guests with the framebuffer enabled supposed to work?

I suspect that needs some work on the qemu side.

Quoted 3 lines · Gerd Hoffmann, le Thu 14 Aug 2008 17:04:07 +0200, a écrit :

Uhm, well, no way to fix minios to run on older xen versions then I guess (which I'd prefer). Except maybe limiting the 32bit version to 1GB address space. Which should be fine at least for pvgrub ...

That should be feasible indeed.

The problem is a bug in the hypervisor's mod_l3_entry which calls adjust_guest_l3e before calling get_page_from_l3e, see c/s 17061. Is it possible for you to backport the patch?

Quoted 2 lines

Another pvgrub issue: Is booting guests with the framebuffer enabled supposed to work?

Yes, but you have to not enable the framebuffer in the grub configuration, because the framebuffer is currently not able to restart.

Quoted 1 line

I suspect that needs some work on the qemu side.

Exactly.

Quoted 4 lines · Samuel Thibault

The problem is a bug in the hypervisor's mod_l3_entry which calls adjust_guest_l3e before calling get_page_from_l3e, see c/s 17061. Is it possible for you to backport the patch?

Fedora 8 just updated their Xen package because the pvops kernels also trigger this bug.

Quoted 2 lines

Another pvgrub issue: Is booting guests with the framebuffer enabled supposed to work?

Yes, but you have to not enable the framebuffer in the grub configuration, because the framebuffer is currently not able to restart.

Quoted 4 lines · Samuel Thibault, le Thu 14 Aug 2008 16:14:00 +0100, a écrit :

Quoted 2 lines

Another pvgrub issue: Is booting guests with the framebuffer enabled supposed to work?

Yes, but you have to not enable the framebuffer in the grub configuration, because the framebuffer is currently not able to restart.

Oh actually it was a lot easier than I thought, see patch below (which is actually partly bug fixes).

diff -r 38783464a671 tools/ioemu/hw/xenfb.c

tools/ioemu/hw/xenfb.c +0 -0

@@ -445,6 +445,13 @@ static void xenfb_unbind(struct xenfb_de
 		xc_evtchn_unbind(dev->xenfb->evt_xch, dev->port);
 		dev->port = -1;
 	}

tools/ioemu/hw/xenfb.c +0 -4

@@ -452,10 +459,6 @@ static void xenfb_detach_dom(struct xenf
 {
 	xenfb_unbind(&xenfb->fb);
 	xenfb_unbind(&xenfb->kbd);
-	if (xenfb->pixels) {
-		munmap(xenfb->pixels, xenfb->fb_len);
-		xenfb->pixels = NULL;
-	}
 }
 
 /* Remove the backend area in xenbus since the framebuffer really is

tools/ioemu/hw/xenfb.c +3 -0

 	case XenbusStateInitWait:
 	case XenbusStateInitialised:
 	case XenbusStateConnected:

@@ -1274,6 +1287,9 @@ static void xenfb_update(void *opaque)
     struct xenfb *xenfb = opaque;
     int period;
 
+    if (!xenfb->fb.page)
+        return;
+
     if (xenfb_queue_full(xenfb))
         return;
 

Quoted 4 lines

tools/ioemu/hw/xenfb.c +4 -0

+	if (!strcmp(dev->devicetype, "vfb")) {
+	    if (dev->xenfb->pixels) {
+		    munmap(dev->xenfb->pixels, dev->xenfb->fb_len);
+		    dev->xenfb->pixels = NULL;

tools/ioemu/hw/xenfb.c static void xenfb_unbind(struct xenfb_de · line 455

Quoted 5 lines

tools/ioemu/hw/xenfb.c +2 -0

+	    }
+	}
 }

The check for vfb is ugly. The only alternative I can see is fb == dev->xenfb->fb. Matter of taste.

tools/ioemu/hw/xenfb.c static int xenfb_on_state_change(struct · line 661

Quoted 7 lines

tools/ioemu/hw/xenfb.c +3 -0

@@ -653,6 +656,16 @@ static int xenfb_on_state_change(struct 
 		   not much point in us continuing. */
 		return -1;
 	case XenbusStateInitialising:
+		if (dev->state != XenbusStateClosed)
+		    /* Do not let a domain make us skip the closing state */
+		    return 0;

Why does this work?

tools/ioemu/hw/xenfb.c static int xenfb_on_state_change(struct · line 668

Quoted 7 lines

tools/ioemu/hw/xenfb.c +7 -0

+		xenfb_switch_state(dev, XenbusStateInitWait);
+		xs_unwatch(dev->xenfb->xsh, dev->otherend, "");
+		if (!strcmp(dev->devicetype, "vkbd")) {
+		    fprintf(stderr, "FB: Waiting for KBD backend creation\n");
+		    xenfb_wait_for_frontend(&dev->xenfb->kbd, xenfb_frontend_initialized_kbd);
+		}
+		break;

Man, I hate this state machine...

Gerd's new one is much clearer (I asked for that).

Quoted 4 lines · Markus Armbruster, le Fri 15 Aug 2008 08:39:06 -0400, a écrit :

Quoted 4 lines

tools/ioemu/hw/xenfb.c +4 -0

+	if (!strcmp(dev->devicetype, "vfb")) {
+	    if (dev->xenfb->pixels) {
+		    munmap(dev->xenfb->pixels, dev->xenfb->fb_len);
+		    dev->xenfb->pixels = NULL;

Quoted 8 lines

tools/ioemu/hw/xenfb.c static void xenfb_unbind(struct xenfb_de · line 455

Quoted 5 lines

tools/ioemu/hw/xenfb.c +2 -0

+	    }
+	}
 }

The check for vfb is ugly. The only alternative I can see is fb == dev->xenfb->fb. Matter of taste.

Hum, that was what I had written previously actually. I don't see why it is "ugly", in that pixels is a device resource. It should probably be in xenfb->fb actually.

Quoted 9 lines

tools/ioemu/hw/xenfb.c static int xenfb_on_state_change(struct · line 661

Quoted 7 lines

tools/ioemu/hw/xenfb.c +3 -0

@@ -653,6 +656,16 @@ static int xenfb_on_state_change(struct 
 		   not much point in us continuing. */
 		return -1;
 	case XenbusStateInitialising:
+		if (dev->state != XenbusStateClosed)
+		    /* Do not let a domain make us skip the closing state */
+		    return 0;

Why does this work?

It's meant to not work. If the frontend is not playing well (going through the closing state), then ignore what it's doing until it plays well. An alternative could be to just shut down. In any case, we don't want to let it drive us to remapping something else without unmapping the previous fb.

We need to make up our mind how to handle invalid state transitions.

Is silently ignoring guest misbehavior smart? I'd rather log it.