Skip to contentSkip to conversation

2.6.24 says "serial8250: too much work for irq4" a lot.

DISCUSSION Rob Landley 6 participants 17 messages Feb 5, 2008 – Mar 11, 2008

lore
Share & tools
lore
#0 lore raw

When running a 2.6.24 kernel built for x86-64 under qemu via serial console, doing CPU-intensive things that also produce a lot of output (such as compiling software) tends to produce the error message in the title.

Anybody have a clue why? It doesn't seem to cause an actual problem, but it's kind of annoying.

(If it's a qemu issue, I can go bother them. It's possible that qemu isn't delivering interrupts as often as it expects, since that's limited by the granularity of the host timer; I know the clock in qemu can run a bit slow because it only gets clock interrupts when the host system isn't too busy to schedule the emulator. But this doesn't usually cause a problem. I _think_ the message is just a "this should never happen" type warning, which is happening to me. But I break stuff. :)

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#1 lore raw
Quoted 5 lines · Rob Landley

When running a 2.6.24 kernel built for x86-64 under qemu via serial console, doing CPU-intensive things that also produce a lot of output (such as compiling software) tends to produce the error message in the title.

Anybody have a clue why? It doesn't seem to cause an actual problem, but it's kind of annoying.

(If it's a qemu issue, I can go bother them. It's possible that qemu isn't delivering interrupts as often as it expects, since that's limited by the granularity of the host timer; I know the clock in qemu can run a bit slow because it only gets clock interrupts when the host system isn't too busy to schedule the emulator. But this doesn't usually cause a problem. I _think_ the message is just a "this should never happen" type warning, which is happening to me. But I break stuff. :)

This is because Qemu spews data to the serial port without any rate limiting; this causes the in-kernel serial port driver to think the port is stuck. The serial port emulation needs to make it possible to drain the virtual FIFO every now and then, as opposed to filling it again immediately.

-hpa

#2 lore raw
Quoted 17 lines · On Tuesday 05 February 2008 15:07:27 H. Peter Anvin wrote:
Quoted 8 lines · Rob Landley

When running a 2.6.24 kernel built for x86-64 under qemu via serial console, doing CPU-intensive things that also produce a lot of output (such as compiling software) tends to produce the error message in the title.

Anybody have a clue why? It doesn't seem to cause an actual problem, but it's kind of annoying.

Quoted 7 lines

(If it's a qemu issue, I can go bother them. It's possible that qemu isn't delivering interrupts as often as it expects, since that's limited by the granularity of the host timer; I know the clock in qemu can run a bit slow because it only gets clock interrupts when the host system isn't too busy to schedule the emulator. But this doesn't usually cause a problem. I _think_ the message is just a "this should never happen" type warning, which is happening to me. But I break stuff. :)

Quoted 7 lines

This is because Qemu spews data to the serial port without any rate limiting; this causes the in-kernel serial port driver to think the port is stuck. The serial port emulation needs to make it possible to drain the virtual FIFO every now and then, as opposed to filling it again immediately.

-hpa

Thanks. cc'd the right list on this one. :)

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#3 lore raw
In reply to H. Peter Anvin
Quoted 9 lines · H. Peter Anvin
Quoted 8 lines

(If it's a qemu issue, I can go bother them. It's possible that qemu isn't delivering interrupts as often as it expects, since that's limited by the granularity of the host timer; I know the clock in qemu can run a bit slow because it only gets clock interrupts when the host system isn't too busy to schedule the emulator. But this doesn't usually cause a problem. I _think_ the message is just a "this should never happen" type warning, which is happening to me. But I break stuff. :)

Quoted 5 lines

This is because Qemu spews data to the serial port without any rate limiting; this causes the in-kernel serial port driver to think the port is stuck. The serial port emulation needs to make it possible to drain the virtual FIFO every now and then, as opposed to filling it again immediately.

actually, the way i solved it for qemu+KVM+paravirt was to just turn off this rather silly check in the serial driver if inside a paravirt guest. When we are emulated then the serial 'hardware' is totally reliable and we should just trust it. That way i never dropped a single bit of kernel log output again.

Ingo

#4 lore raw
Quoted 7 lines · Ingo Molnar
Quoted 2 lines · H. Peter Anvin
Quoted 1 line

(If it's a qemu issue, I can go bother them. It's possible that qemu isn't delivering interrupts as often as it expects, since that's limited by the granularity of the host timer; I know the clock in qemu can run a bit slow because it only gets clock interrupts when the host system isn't too busy to schedule the emulator. But this doesn't usually cause a problem. I _think_ the message is just a "this should never happen" type warning, which is happening to me. But I break stuff. :)

This is because Qemu spews data to the serial port without any rate limiting; this causes the in-kernel serial port driver to think the port is stuck. The serial port emulation needs to make it possible to drain the virtual FIFO every now and then, as opposed to filling it again immediately.

actually, the way i solved it for qemu+KVM+paravirt was to just turn off this rather silly check in the serial driver if inside a paravirt guest. When we are emulated then the serial 'hardware' is totally reliable and we should just trust it. That way i never dropped a single bit of kernel log output again.

Yes, but keying that on paravirt is silly in the extreme. After all, there is no need for this to be paravirtualized.

-hpa

#5 lore raw
Quoted 10 lines · On Thursday 07 February 2008 11:37:12 H. Peter Anvin wrote:
Quoted 5 lines

actually, the way i solved it for qemu+KVM+paravirt was to just turn off this rather silly check in the serial driver if inside a paravirt guest. When we are emulated then the serial 'hardware' is totally reliable and we should just trust it. That way i never dropped a single bit of kernel log output again.

Yes, but keying that on paravirt is silly in the extreme. After all, there is no need for this to be paravirtualized.

-hpa

Specifically, qemu isn't paravirtualized, it's fully virtualized. The same kernel can run on real hardware just fine. (Sort of the point of the project...)

I can yank the warning for the kernels I build (or set PASS_LIMIT to 9999999), but I'd rather not carry any more patches than I can avoid...

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#6 lore raw
Quoted 5 lines · Rob Landley

Specifically, qemu isn't paravirtualized, it's fully virtualized. The same kernel can run on real hardware just fine. (Sort of the point of the project...)

I can yank the warning for the kernels I build (or set PASS_LIMIT to 9999999), but I'd rather not carry any more patches than I can avoid...

The right thing to do is to add virtual FIFO exhaustion into the Qemu device model. It really isn't a valid emulation of a UART that it has an infinite FIFO behind it that can flood the interrupt handler for an arbitrary number of fetches.

I was going to give a technical description of how to do it here, but then I realized I might as well just write it up as a patch. Stay tuned.

-hpa

#7 lore raw
In reply to Rob Landley
Quoted 5 lines · Rob Landley

Specifically, qemu isn't paravirtualized, it's fully virtualized. The same kernel can run on real hardware just fine. (Sort of the point of the project...)

I can yank the warning for the kernels I build (or set PASS_LIMIT to 9999999), but I'd rather not carry any more patches than I can avoid...

Patch attached, completely untested beyond compilation.

In particular:

- we should probably clear the burst counter when the interrupt line goes from inactive to active.
- there probably should be a timer which clears the burst counter.

However, I think I've covered most of the bases...

-hpa

hw/serial.c +28 -3
index b1bd0ff..c902792 100644
@@ -73,6 +73,15 @@
 #define UART_LSR_OE	0x02	/* Overrun error indicator */
 #define UART_LSR_DR	0x01	/* Receiver data ready */
 
+/*
+ * It's common for an IRQ handler to keep reading the RBR until
+ * the LSR indicates that the FIFO is empty, expecting that the
+ * CPU is vastly faster than the serial line.  This can cause
+ * overruns or error indications if the FIFO never empties, so
+ * give the target OS a breather every so often.
+ */
+#define MAX_BURST	512
+
 struct SerialState {
     uint16_t divider;
     uint8_t rbr; /* receive register */
@@ -91,8 +100,14 @@ struct SerialState {
     int last_break_enable;
     target_phys_addr_t base;
     int it_shift;
+    int burst_len;
 };
 
+static void serial_clear_burst(SerialState *s)
+{
+    s->burst_len = 0;
+}
+
 static void serial_update_irq(SerialState *s)
 {
     if ((s->lsr & UART_LSR_DR) && (s->ier & UART_IER_RDI)) {
@@ -114,6 +129,8 @@ static void serial_update_parameters(SerialState *s)
     int speed, parity, data_bits, stop_bits;
     QEMUSerialSetParams ssp;
 
+    serial_clear_burst(s);
+
     if (s->lcr & 0x08) {
         if (s->lcr & 0x10)
             parity = 'E';
@@ -221,9 +238,12 @@ static uint32_t serial_ioport_read(void *opaque, uint32_t addr)
             ret = s->divider & 0xff;
         } else {
             ret = s->rbr;
-            s->lsr &= ~(UART_LSR_DR | UART_LSR_BI);
-            serial_update_irq(s);
-            qemu_chr_accept_input(s->chr);
+	    if (s->burst_len < MAX_BURST) {
+		s->burst_len++;
+		s->lsr &= ~(UART_LSR_DR | UART_LSR_BI);
+		serial_update_irq(s);
+		qemu_chr_accept_input(s->chr);
+	    }
         }
         break;
     case 1:
@@ -235,6 +255,7 @@ static uint32_t serial_ioport_read(void *opaque, uint32_t addr)
         break;
     case 2:
         ret = s->iir;
+	serial_clear_burst(s);
         /* reset THR pending bit */
         if ((ret & 0x7) == UART_IIR_THRI)
             s->thr_ipending = 0;
@@ -248,6 +269,10 @@ static uint32_t serial_ioport_read(void *opaque, uint32_t addr)
         break;
     case 5:
         ret = s->lsr;
+	if (s->burst_len >= MAX_BURST)
+	    ret &= ~(UART_LSR_DR|UART_LSR_BI);
+	if (!(ret & UART_LSR_DR))
+	    serial_clear_burst(s);
         break;
     case 6:
         if (s->mcr & UART_MCR_LOOP) {
#8 lore raw
Quoted 10 lines · On Thursday 07 February 2008 15:19:39 H. Peter Anvin wrote:
Quoted 6 lines · Rob Landley

Specifically, qemu isn't paravirtualized, it's fully virtualized. The same kernel can run on real hardware just fine. (Sort of the point of the project...)

I can yank the warning for the kernels I build (or set PASS_LIMIT to 9999999), but I'd rather not carry any more patches than I can avoid...

Patch attached, completely untested beyond compilation.

Quoted 8 lines

In particular:

- we should probably clear the burst counter when the interrupt line goes from inactive to active.
- there probably should be a timer which clears the burst counter.

However, I think I've covered most of the bases...

Well, qemu still seems to work with the patch applied, and I haven't reproduced the error message yet...

Rob

Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#9 lore raw
In reply to H. Peter Anvin

Here's a patch Peter Anvin wrote so the serial I/O doesn't flood the kernel.

Here's the thread on linux-kernel aboout it:
http://lkml.org/lkml/2008/2/5/401

Rob

Quoted 12 lines · On Thursday 07 February 2008 15:19:39 you wrote:
Quoted 6 lines · Rob Landley

Specifically, qemu isn't paravirtualized, it's fully virtualized. The same kernel can run on real hardware just fine. (Sort of the point of the project...)

I can yank the warning for the kernels I build (or set PASS_LIMIT to 9999999), but I'd rather not carry any more patches than I can avoid...

Patch attached, completely untested beyond compilation.

In particular:

Quoted 8 lines

- we should probably clear the burst counter when the interrupt line goes from inactive to active.
- there probably should be a timer which clears the burst counter.

However, I think I've covered most of the bases...

-hpa

hw/serial.c +28 -3
index b1bd0ff..c902792 100644
@@ -73,6 +73,15 @@
 #define UART_LSR_OE	0x02	/* Overrun error indicator */
 #define UART_LSR_DR	0x01	/* Receiver data ready */
 
+/*
+ * It's common for an IRQ handler to keep reading the RBR until
+ * the LSR indicates that the FIFO is empty, expecting that the
+ * CPU is vastly faster than the serial line.  This can cause
+ * overruns or error indications if the FIFO never empties, so
+ * give the target OS a breather every so often.
+ */
+#define MAX_BURST	512
+
 struct SerialState {
     uint16_t divider;
     uint8_t rbr; /* receive register */
@@ -91,8 +100,14 @@ struct SerialState {
     int last_break_enable;
     target_phys_addr_t base;
     int it_shift;
+    int burst_len;
 };
 
+static void serial_clear_burst(SerialState *s)
+{
+    s->burst_len = 0;
+}
+
 static void serial_update_irq(SerialState *s)
 {
     if ((s->lsr & UART_LSR_DR) && (s->ier & UART_IER_RDI)) {
@@ -114,6 +129,8 @@ static void serial_update_parameters(SerialState *s)
     int speed, parity, data_bits, stop_bits;
     QEMUSerialSetParams ssp;
 
+    serial_clear_burst(s);
+
     if (s->lcr & 0x08) {
         if (s->lcr & 0x10)
             parity = 'E';
@@ -221,9 +238,12 @@ static uint32_t serial_ioport_read(void *opaque, uint32_t addr)
             ret = s->divider & 0xff;
         } else {
             ret = s->rbr;
-            s->lsr &= ~(UART_LSR_DR | UART_LSR_BI);
-            serial_update_irq(s);
-            qemu_chr_accept_input(s->chr);
+	    if (s->burst_len < MAX_BURST) {
+		s->burst_len++;
+		s->lsr &= ~(UART_LSR_DR | UART_LSR_BI);
+		serial_update_irq(s);
+		qemu_chr_accept_input(s->chr);
+	    }
         }
         break;
     case 1:
@@ -235,6 +255,7 @@ static uint32_t serial_ioport_read(void *opaque, uint32_t addr)
         break;
     case 2:
         ret = s->iir;
+	serial_clear_burst(s);
         /* reset THR pending bit */
         if ((ret & 0x7) == UART_IIR_THRI)
             s->thr_ipending = 0;
@@ -248,6 +269,10 @@ static uint32_t serial_ioport_read(void *opaque, uint32_t addr)
         break;
     case 5:
         ret = s->lsr;
+	if (s->burst_len >= MAX_BURST)
+	    ret &= ~(UART_LSR_DR|UART_LSR_BI);
+	if (!(ret & UART_LSR_DR))
+	    serial_clear_burst(s);
         break;
     case 6:
         if (s->mcr & UART_MCR_LOOP) {
Signature · 2 lines
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
#11 lore raw
Quoted 6 lines · Blue Swirl
Quoted 1 line · Rob Landley

Here's a patch Peter Anvin wrote so the serial I/O doesn't flood the kernel.

The patch looks OK, but the throttling should benefit all devices, as discussed here:
http://lists.gnu.org/archive/html/qemu-devel/2007-12/msg00283.html

I strongly disagree with the sentiments in that post.

This is not a matter of rate throttling, but simulated FIFO exhaustion -- they are NOT the same thing. Simulated FIFO exhaustion is functionally equivalent to making sure there are interrupt windows opened in an otherwise-too-long critical section; it doesn't constrain any particular flow rate, as it still permits another interrupt to immediately come in.

If you look at the patch, there are no timing dependencies; the only parameter is the depth of the virtual queue. The exhaustion is completely controlled by target OS access patterns.

-hpa

#12 lore raw
Quoted 17 lines · H. Peter Anvin
Quoted 6 lines · Blue Swirl
Quoted 1 line · Rob Landley

Here's a patch Peter Anvin wrote so the serial I/O doesn't flood the kernel.

The patch looks OK, but the throttling should benefit all devices, as discussed here:
http://lists.gnu.org/archive/html/qemu-devel/2007-12/msg00283.html

I strongly disagree with the sentiments in that post.

This is not a matter of rate throttling, but simulated FIFO exhaustion -- they are NOT the same thing. Simulated FIFO exhaustion is functionally equivalent to making sure there are interrupt windows opened in an otherwise-too-long critical section; it doesn't constrain any particular flow rate, as it still permits another interrupt to immediately come in.

Quoted 3 lines

If you look at the patch, there are no timing dependencies; the only parameter is the depth of the virtual queue. The exhaustion is completely controlled by target OS access patterns.

Thanks, this clarified the difference. But I'll rephrase my original comment:

The patch looks OK, but the simulated FIFO exhaustion should benefit all devices, as
discussed here:
http://lists.gnu.org/archive/html/qemu-devel/2007-12/msg00283.html

#13 lore raw
Quoted 11 lines · Blue Swirl
Quoted 4 lines

If you look at the patch, there are no timing dependencies; the only parameter is the depth of the virtual queue. The exhaustion is completely controlled by target OS access patterns.

Thanks, this clarified the difference. But I'll rephrase my original comment:

The patch looks OK, but the simulated FIFO exhaustion should benefit all devices, as
discussed here:
http://lists.gnu.org/archive/html/qemu-devel/2007-12/msg00283.html

The difference is you *can't* do that in a general layer.

-hpa

#14 lore raw
Quoted 6 lines · H. Peter Anvin
Quoted 5 lines · Blue Swirl
Quoted 4 lines

If you look at the patch, there are no timing dependencies; the only parameter is the depth of the virtual queue. The exhaustion is completely controlled by target OS access patterns.

Quoted 8 lines
Quoted 6 lines

Thanks, this clarified the difference. But I'll rephrase my original comment:

The patch looks OK, but the simulated FIFO exhaustion should benefit all devices, as
discussed here:
http://lists.gnu.org/archive/html/qemu-devel/2007-12/msg00283.html

The difference is you *can't* do that in a general layer.

What makes you think that is impossible? Just move the serial_clear_burst to vl.c under name chr_clear_burst, move burst_len to CharDriverState and introduce a new function in vl.c that contains the burst length check. This is functionally identical to your patch. For 100% compatibility, the init functions could be changed so that only PC serial is affected, but I think all character devices would benefit from this.

Also win2k install hack in ide.c seems to be related to this problem, so even more generic solution would be desirable.

#15 lore raw
Quoted 9 lines · Blue Swirl
Quoted 8 lines · H. Peter Anvin
Quoted 7 lines · Blue Swirl
Quoted 3 lines

If you look at the patch, there are no timing dependencies; the only parameter is the depth of the virtual queue. The exhaustion is completely controlled by target OS access patterns.

Thanks, this clarified the difference. But I'll rephrase my original comment:

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

The patch looks OK, but the simulated FIFO exhaustion should benefit all devices, as
discussed here:
http://lists.gnu.org/archive/html/qemu-devel/2007-12/msg00283.html

The difference is you *can't* do that in a general layer.

What makes you think that is impossible?

IIUC the proposed patch makes the serial driver return an empty FIFO exactly once, them immediately continue receiving data. Throughput should be approximately the same, you've just got a bit of extra overhead to process the additional interrupts. This is very different to the previous patch which did time-based throughput limiting.

You can't do this in generic code because there's no way to guess when the guest os has seen the FIFO empty condition. The best you can do is pause for some arbitrary length of time, which is both unreliable (the guest OS may not have got to far enough yet, especially if the host machine is heavily loaded), and has a significant negative impact on throughput.

Quoted 2 lines

Also win2k install hack in ide.c seems to be related to this problem, so even more generic solution would be desirable.

IIUC the win2k hack is an actual timing problem. The win2k IDE drivers are buggy, and fall over if the drive responds too soon.

Paul

#16 lore raw
Quoted 31 lines · Paul Brook
Quoted 9 lines · Blue Swirl
Quoted 8 lines · H. Peter Anvin
Quoted 7 lines · Blue Swirl
Quoted 3 lines

If you look at the patch, there are no timing dependencies; the only parameter is the depth of the virtual queue. The exhaustion is completely controlled by target OS access patterns.

Thanks, this clarified the difference. But I'll rephrase my original comment:

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

The patch looks OK, but the simulated FIFO exhaustion should benefit all devices, as
discussed here:
http://lists.gnu.org/archive/html/qemu-devel/2007-12/msg00283.html

The difference is you *can't* do that in a general layer.

What makes you think that is impossible?

IIUC the proposed patch makes the serial driver return an empty FIFO exactly once, them immediately continue receiving data. Throughput should be approximately the same, you've just got a bit of extra overhead to process the additional interrupts. This is very different to the previous patch which did time-based throughput limiting.

You can't do this in generic code because there's no way to guess when the guest os has seen the FIFO empty condition. The best you can do is pause for some arbitrary length of time, which is both unreliable (the guest OS may not have got to far enough yet, especially if the host machine is heavily loaded), and has a significant negative impact on throughput.

Quoted 6 lines
Quoted 2 lines

Also win2k install hack in ide.c seems to be related to this problem, so even more generic solution would be desirable.

IIUC the win2k hack is an actual timing problem. The win2k IDE drivers are buggy, and fall over if the drive responds too soon.

Is everybody convinced about this patch now? I would really like to see it in the CVS, as it greatly improves the usability of the serial consoles on targets running GNU/Linux (and probably OSes).

Signature · 4 lines
  .''`.  Aurelien Jarno	            | GPG: 1024D/F1BCDB73
 : :' :  Debian developer           | Electrical Engineer
 `. `'   [email protected]         | [email protected]
   `-    people.debian.org/~aurel32 | www.aurel32.net