Skip to contentSkip to conversation

pci: interrupt status/interrupt disable support

PATCH Superseded Michael S. Tsirkin 3 participants 9 messages Nov 25, 2009 – Nov 26, 2009

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover pci: interrupt status/interrupt disable support

This patchset adds support for mandatory interupt status and interrupt disable bits to all PCI devices. This is required for PCI compliancy.

These patches are on top of my pci tree, including Isaku Yamahata's fixes.
If this is a problem, let me know and
I will rebase.

This works fine for me, but since this touches all PCI devices, please review carefully.

Michael S. Tsirkin (4):
pci: rearrange code for interrupts
pci: track IRQ status
pci: interrupt status bit implementation
pci: interrupt disable bit support

 hw/pci.c |   83 +++++++++++++++++++++++++++++++++++++++++++++++++++++---------
 hw/pci.h |    8 ++++++
 2 files changed, 79 insertions(+), 12 deletions(-)
#1 lore raw
Quoted 11 lines · Michael S. Tsirkin

This patchset adds support for mandatory interupt status and interrupt disable bits to all PCI devices. This is required for PCI compliancy.

These patches are on top of my pci tree, including Isaku Yamahata's fixes.
If this is a problem, let me know and
I will rebase.

This works fine for me, but since this touches all PCI devices, please review carefully.

Just a curiosity, what OS do you have in your mind?

You introduced new members, irq_status and irq_disabled
and maintain them according configuration space write. Another approach is to use irq_state[PCI_NUM_PINS] and interrupt disabled bit in command register.
At least I think irq_disable can be removed by moving !change check from pci_set_irq() into pci_change_irq_level().

As for irq_status, only user of irq_status is pci_update_irq_status() so if (irq_statue) can be open coded. On the other hand, PCIBus already has irq_count member for same purpose.
So probably open coding or introducing irq_count instead of irq_status would be reasonable.

Quoted 11 lines

Michael S. Tsirkin (4):
pci: rearrange code for interrupts
pci: track IRQ status
pci: interrupt status bit implementation
pci: interrupt disable bit support

 hw/pci.c |   83 +++++++++++++++++++++++++++++++++++++++++++++++++++++---------
 hw/pci.h |    8 ++++++
 2 files changed, 79 insertions(+), 12 deletions(-)
Signature · 1 line
yamahata
#2 lore raw
Quoted 10 lines · Isaku Yamahata
Quoted 9 lines · Michael S. Tsirkin

This patchset adds support for mandatory interupt status and interrupt disable bits to all PCI devices. This is required for PCI compliancy.

These patches are on top of my pci tree, including Isaku Yamahata's fixes.
If this is a problem, let me know and
I will rebase.

Quoted 4 lines
Quoted 2 lines

This works fine for me, but since this touches all PCI devices, please review carefully.

Just a curiosity, what OS do you have in your mind?

windows and linux :)

Quoted 4 lines

You introduced new members, irq_status and irq_disabled
and maintain them according configuration space write. Another approach is to use irq_state[PCI_NUM_PINS] and interrupt disabled bit in command register.

I will explain. irq_status is an optimization: it is a sum of all irq_state values. Since interrupts is a fast-path operation, we do not want to add another loop there.

Quoted 2 lines

At least I think irq_disable can be removed by moving !change check from pci_set_irq() into pci_change_irq_level().

I don't see how is pci_set_irq relevant here: the reason I added irq_disabled is because we need to re-trigger interrupts when interrupts are enabled, either by load or config cycle. So it is there simply to detect change.

Another approach would be to make irq_disabled a local variable in pci_default_write_config and in pci_device_load, and pass old value to pci_update_irq_disabled. I tried and it seems more code, and routines become less self-contained. As it is, I think it's cleaner because we have an idem-potent routine that is always safe it call, just like pci_update_mappings.

Quoted 5 lines

As for irq_status, only user of irq_status is pci_update_irq_status() so if (irq_statue) can be open coded. On the other hand, PCIBus already has irq_count member for same purpose.
So probably open coding or introducing irq_count instead of irq_status would be reasonable.

No, this would slow us down because these are per-pin. We need a sum of interrupts so that config space
can be updated by a single command.
Interrupts are a fastpath, extra loops there should be avoided.

Quoted 15 lines
Quoted 12 lines

Michael S. Tsirkin (4):
pci: rearrange code for interrupts
pci: track IRQ status
pci: interrupt status bit implementation
pci: interrupt disable bit support

 hw/pci.c |   83 +++++++++++++++++++++++++++++++++++++++++++++++++++++---------
 hw/pci.h |    8 ++++++
 2 files changed, 79 insertions(+), 12 deletions(-)
Signature · 1 line
yamahata
#3 lore raw
Quoted 4 lines

No, this would slow us down because these are per-pin. We need a sum of interrupts so that config space
can be updated by a single command.
Interrupts are a fastpath, extra loops there should be avoided.

It's really not that much of a fast path. Unless you're doing something particularly obscure then even under heavy load you're unlikely to exceed a few kHz. Compared to the average PIC implementation, and the overhead of the actual CPU interrupt, I find it hard to believe that looping over precisely 4 entries has any real performance hit.

Paul

#4 lore raw
Quoted 8 lines · Paul Brook
Quoted 4 lines

No, this would slow us down because these are per-pin. We need a sum of interrupts so that config space
can be updated by a single command.
Interrupts are a fastpath, extra loops there should be avoided.

It's really not that much of a fast path. Unless you're doing something particularly obscure then even under heavy load you're unlikely to exceed a few kHz.

I think with kvm, heavy disk stressing benchmark can get higher.

Quoted 5 lines

Compared to the average PIC implementation, and the overhead of the actual CPU interrupt, I find it hard to believe that looping over precisely 4 entries has any real performance hit.

Paul

I don't think it is major, but I definitely have seen, in the past, that extra branches and memory accesses have small but measureable effect when taken in interrupt handler routines in drivers, and same should apply here.

OTOH keeping the sum around is trivial.

It might not be easily measureable now, but IMO that's just because the whole interrupt delivery is so complex. E.g. we currently have there a loop re-computing interrupt routing on each access, this is just silly and I intend to fix it. I would rather not introduce more code that will have to be cleaned up later.

Signature · 1 line
MST
#5 lore raw
Quoted 5 lines
Quoted 3 lines

It's really not that much of a fast path. Unless you're doing something particularly obscure then even under heavy load you're unlikely to exceed a few kHz.

I think with kvm, heavy disk stressing benchmark can get higher.

I'd still expect this to be the least of your problems.

If nothing else you've at least one host signal delivery and/or thread context switch in there. Not to mention the overhead to forwarding the interrupt to the guest CPU.

Quoted 10 lines
Quoted 3 lines

Compared to the average PIC implementation, and the overhead of the actual CPU interrupt, I find it hard to believe that looping over precisely 4 entries has any real performance hit.

I don't think it is major, but I definitely have seen, in the past, that extra branches and memory accesses have small but measureable effect when taken in interrupt handler routines in drivers, and same should apply here.

OTOH keeping the sum around is trivial.

Not entirely. You now have two different bits of information that you have to keep consistent.

Unless you can show that this is performance critical code I strongly recommend keeping it as simple as possible.

Paul

#6 lore raw
Quoted 10 lines · Paul Brook
Quoted 5 lines
Quoted 3 lines

It's really not that much of a fast path. Unless you're doing something particularly obscure then even under heavy load you're unlikely to exceed a few kHz.

I think with kvm, heavy disk stressing benchmark can get higher.

I'd still expect this to be the least of your problems.

If nothing else you've at least one host signal delivery and/or thread context switch in there.

iotread which does the signalling might be running in parallel with the guest CPU.

Quoted 2 lines

Not to mention the overhead to forwarding the interrupt to the guest CPU.

This is often mitigated as KVM knows to inject the interrupt on the next vmexit.

Quoted 9 lines
Quoted 9 lines
Quoted 3 lines

Compared to the average PIC implementation, and the overhead of the actual CPU interrupt, I find it hard to believe that looping over precisely 4 entries has any real performance hit.

I don't think it is major, but I definitely have seen, in the past, that extra branches and memory accesses have small but measureable effect when taken in interrupt handler routines in drivers, and same should apply here.

Quoted 4 lines
Quoted 1 line

OTOH keeping the sum around is trivial.

Not entirely. You now have two different bits of information that you have to keep consistent.

This is inherent in pci spec definition: interrupt status bit in config space duplicates interrupt state.

Quoted 4 lines

Unless you can show that this is performance critical code I strongly recommend keeping it as simple as possible.

Paul

I don't see there is anything left show: interrupt delivery is *obviously* performance critical: people are running *latency benchmarks* measuring how fast a packet can get from an external interface into guest, in microseconds. We definitely want to remove obvious waste there.

Signature · 1 line
MST
#7 lore raw
In reply to Isaku Yamahata
Quoted 1 line · Isaku Yamahata

At least I think irq_disable can be removed

The following patch on top of mine removes irq_disabled field in PCIDevice. I am of two minds whether this makes the code better. What is your opinion?

hw/pci.c +16 -14
index 3daae46..75f97df 100644
@@ -128,15 +128,18 @@ static void pci_change_irq_level(PCIDevice *pci_dev, int irq_num, int change)
     bus->set_irq(bus->irq_opaque, irq_num, bus->irq_count[irq_num] != 0);
 }
 
-/* Update irq disabled field after config space change,
- * assert/deassert interrupts if necessary. */
-static void pci_update_irq_disabled(PCIDevice *d)
+static inline int pci_irq_disabled(PCIDevice *d)
 {
-    int i;
-    int disabled = !!(pci_get_word(d->config + PCI_COMMAND) &
-                      PCI_COMMAND_INTX_DISABLE);
-    if (disabled == d->irq_disabled)
+    return !!(pci_get_word(d->config + PCI_COMMAND) & PCI_COMMAND_INTX_DISABLE);
+}
+
+/* Called after interrupt disabled field update in config space,
+ * assert/deassert interrupts if necessary.
+ * Gets original interrupt disable bit value (before update). */
+static void pci_update_irq_disabled(PCIDevice *d, int was_irq_disabled)
+{
+    int i, disabled = pci_irq_disabled(d);
+    if (disabled == was_irq_disabled)
         return;
-    d->irq_disabled = disabled;
     for (i = 0; i < ARRAY_SIZE(d->irq_state); ++i) {
         int state = d->irq_state[i];
@@ -150,7 +153,6 @@ static void pci_device_reset(PCIDevice *dev)
 
     memset(dev->irq_state, 0, sizeof dev->irq_state);
     dev->irq_status = 0;
-    dev->irq_disabled = 0;
     pci_update_irq_status(dev);
     dev->config[PCI_COMMAND] &= ~(PCI_COMMAND_IO | PCI_COMMAND_MEMORY |
                                   PCI_COMMAND_MASTER);
@@ -369,14 +371,14 @@ void pci_device_save(PCIDevice *s, QEMUFile *f)
 
 int pci_device_load(PCIDevice *s, QEMUFile *f)
 {
-    int ret, i;
+    int ret, i, was_irq_disabled = pci_irq_disabled(d);
     ret = vmstate_load_state(f, pci_get_vmstate(s), s, s->version_id);
     for (i = 0; i < ARRAY_SIZE(s->irq_state); ++i) {
         s->irq_status += s->irq_state[i];
     }
     /* Restore the interrupt status bit. */
     pci_update_irq_status(s);
-    pci_update_irq_disabled(s);
+    pci_update_irq_disabled(s, was_irq_disabled);
     return ret;
 }
 
@@ -924,7 +926,7 @@ uint32_t pci_default_read_config(PCIDevice *d,
 
 void pci_default_write_config(PCIDevice *d, uint32_t addr, uint32_t val, int l)
 {
-    int i;
+    int i, was_irq_disabled = pci_irq_disabled(d);
     uint32_t config_size = pci_config_size(d);
 
     for (i = 0; i < l && addr + i < config_size; val >>= 8, ++i) {
@@ -938,7 +940,7 @@ void pci_default_write_config(PCIDevice *d, uint32_t addr, uint32_t val, int l)
         pci_update_mappings(d);
 
     if (range_covers_byte(addr, l, PCI_COMMAND))
-        pci_update_irq_disabled(d);
+        pci_update_irq_disabled(d, was_irq_disabled);
 }
 
 /***********************************************************/
@@ -957,7 +959,7 @@ static void pci_set_irq(void *opaque, int irq_num, int level)
     pci_dev->irq_state[irq_num] = level;
     pci_dev->irq_status += change;
     pci_update_irq_status(pci_dev);
-    if (pci_dev->irq_disabled)
+    if (pci_irq_disabled(pci_dev))
         return;
     pci_change_irq_level(pci_dev, irq_num, change);
 }
hw/pci.h +0 -3
index 994b8bc..25ad114 100644
@@ -227,9 +227,6 @@ struct PCIDevice {
     /* Sum of all irq levels. Used to implement irq status register. */
     int irq_status;
 
-    /* Whether interrupts are disabled by command bit. */
-    int irq_disabled;
-
     /* Capability bits */
     uint32_t cap_present;
 
#8 lore raw
Quoted 6 lines · Michael S. Tsirkin
Quoted 1 line · Isaku Yamahata

At least I think irq_disable can be removed

The following patch on top of mine removes irq_disabled field in PCIDevice. I am of two minds whether this makes the code better. What is your opinion?

Thinking about this some more, I really like the original approach better, mostly because it helps keep the fastpath streamlined.

On thing to note is that irq_disabled field was in the same or adjacent cache line with irq_state, and it is accessed on fast path. Reading config space instead will access an extra cache line which would be cold otherwise (config space is normally not accessed on fast path), and also needs an extra memory access (it is allocated on heap separately).

As I said separately, the whole of interrupt delivery is far from optimal, but we have to start somewhere.

Quoted 97 lines
hw/pci.c +16 -14
index 3daae46..75f97df 100644
@@ -128,15 +128,18 @@ static void pci_change_irq_level(PCIDevice *pci_dev, int irq_num, int change)
     bus->set_irq(bus->irq_opaque, irq_num, bus->irq_count[irq_num] != 0);
 }
 
-/* Update irq disabled field after config space change,
- * assert/deassert interrupts if necessary. */
-static void pci_update_irq_disabled(PCIDevice *d)
+static inline int pci_irq_disabled(PCIDevice *d)
 {
-    int i;
-    int disabled = !!(pci_get_word(d->config + PCI_COMMAND) &
-                      PCI_COMMAND_INTX_DISABLE);
-    if (disabled == d->irq_disabled)
+    return !!(pci_get_word(d->config + PCI_COMMAND) & PCI_COMMAND_INTX_DISABLE);
+}
+
+/* Called after interrupt disabled field update in config space,
+ * assert/deassert interrupts if necessary.
+ * Gets original interrupt disable bit value (before update). */
+static void pci_update_irq_disabled(PCIDevice *d, int was_irq_disabled)
+{
+    int i, disabled = pci_irq_disabled(d);
+    if (disabled == was_irq_disabled)
         return;
-    d->irq_disabled = disabled;
     for (i = 0; i < ARRAY_SIZE(d->irq_state); ++i) {
         int state = d->irq_state[i];
@@ -150,7 +153,6 @@ static void pci_device_reset(PCIDevice *dev)
 
     memset(dev->irq_state, 0, sizeof dev->irq_state);
     dev->irq_status = 0;
-    dev->irq_disabled = 0;
     pci_update_irq_status(dev);
     dev->config[PCI_COMMAND] &= ~(PCI_COMMAND_IO | PCI_COMMAND_MEMORY |
                                   PCI_COMMAND_MASTER);
@@ -369,14 +371,14 @@ void pci_device_save(PCIDevice *s, QEMUFile *f)
 
 int pci_device_load(PCIDevice *s, QEMUFile *f)
 {
-    int ret, i;
+    int ret, i, was_irq_disabled = pci_irq_disabled(d);
     ret = vmstate_load_state(f, pci_get_vmstate(s), s, s->version_id);
     for (i = 0; i < ARRAY_SIZE(s->irq_state); ++i) {
         s->irq_status += s->irq_state[i];
     }
     /* Restore the interrupt status bit. */
     pci_update_irq_status(s);
-    pci_update_irq_disabled(s);
+    pci_update_irq_disabled(s, was_irq_disabled);
     return ret;
 }
 
@@ -924,7 +926,7 @@ uint32_t pci_default_read_config(PCIDevice *d,
 
 void pci_default_write_config(PCIDevice *d, uint32_t addr, uint32_t val, int l)
 {
-    int i;
+    int i, was_irq_disabled = pci_irq_disabled(d);
     uint32_t config_size = pci_config_size(d);
 
     for (i = 0; i < l && addr + i < config_size; val >>= 8, ++i) {
@@ -938,7 +940,7 @@ void pci_default_write_config(PCIDevice *d, uint32_t addr, uint32_t val, int l)
         pci_update_mappings(d);
 
     if (range_covers_byte(addr, l, PCI_COMMAND))
-        pci_update_irq_disabled(d);
+        pci_update_irq_disabled(d, was_irq_disabled);
 }
 
 /***********************************************************/
@@ -957,7 +959,7 @@ static void pci_set_irq(void *opaque, int irq_num, int level)
     pci_dev->irq_state[irq_num] = level;
     pci_dev->irq_status += change;
     pci_update_irq_status(pci_dev);
-    if (pci_dev->irq_disabled)
+    if (pci_irq_disabled(pci_dev))
         return;
     pci_change_irq_level(pci_dev, irq_num, change);
 }
hw/pci.h +0 -3
index 994b8bc..25ad114 100644
@@ -227,9 +227,6 @@ struct PCIDevice {
     /* Sum of all irq levels. Used to implement irq status register. */
     int irq_status;
 
-    /* Whether interrupts are disabled by command bit. */
-    int irq_disabled;
-
     /* Capability bits */
     uint32_t cap_present;