pci: interrupt status/interrupt disable support
cover pci: interrupt status/interrupt disable support
This patchset adds support for mandatory interupt status and interrupt disable bits to all PCI devices. This is required for PCI compliancy.
These patches are on top of my pci tree, including Isaku Yamahata's fixes.
If this is a problem, let me know and
I will rebase.
This works fine for me, but since this touches all PCI devices, please review carefully.
Michael S. Tsirkin (4):
pci: rearrange code for interrupts
pci: track IRQ status
pci: interrupt status bit implementation
pci: interrupt disable bit support
hw/pci.c | 83 +++++++++++++++++++++++++++++++++++++++++++++++++++++--------- hw/pci.h | 8 ++++++ 2 files changed, 79 insertions(+), 12 deletions(-)
Quoted 11 lines · Michael S. Tsirkin
This patchset adds support for mandatory interupt status and interrupt disable bits to all PCI devices. This is required for PCI compliancy.
These patches are on top of my pci tree, including Isaku Yamahata's fixes.
If this is a problem, let me know and
I will rebase.This works fine for me, but since this touches all PCI devices, please review carefully.
Just a curiosity, what OS do you have in your mind?
You introduced new members, irq_status and irq_disabled
and maintain them according configuration space write. Another approach is to use irq_state[PCI_NUM_PINS] and interrupt disabled bit in command register.
At least I think irq_disable can be removed by moving !change check from pci_set_irq() into pci_change_irq_level().
As for irq_status, only user of irq_status is pci_update_irq_status() so if (irq_statue) can be open coded. On the other hand, PCIBus already has irq_count member for same purpose.
So probably open coding or introducing irq_count instead of irq_status would be reasonable.
Quoted 11 lines
Michael S. Tsirkin (4):
pci: rearrange code for interrupts
pci: track IRQ status
pci: interrupt status bit implementation
pci: interrupt disable bit supporthw/pci.c | 83 +++++++++++++++++++++++++++++++++++++++++++++++++++++--------- hw/pci.h | 8 ++++++ 2 files changed, 79 insertions(+), 12 deletions(-)
Signature · 1 line
yamahata
Quoted 10 lines · Isaku Yamahata
Quoted 9 lines · Michael S. Tsirkin
This patchset adds support for mandatory interupt status and interrupt disable bits to all PCI devices. This is required for PCI compliancy.
These patches are on top of my pci tree, including Isaku Yamahata's fixes.
If this is a problem, let me know and
I will rebase.
Quoted 4 lines
Quoted 2 lines
This works fine for me, but since this touches all PCI devices, please review carefully.
Just a curiosity, what OS do you have in your mind?
windows and linux :)
Quoted 4 lines
You introduced new members,
irq_statusandirq_disabled
and maintain them according configuration space write. Another approach is to use irq_state[PCI_NUM_PINS] and interrupt disabled bit in command register.
I will explain. irq_status is an optimization: it is a sum of all irq_state values. Since interrupts is a fast-path operation, we do not want to add another loop there.
Quoted 2 lines
At least I think
irq_disablecan be removed by moving !change check frompci_set_irq()intopci_change_irq_level().
I don't see how is pci_set_irq relevant here: the reason I added irq_disabled is because we need to re-trigger interrupts when interrupts are enabled, either by load or config cycle. So it is there simply to detect change.
Another approach would be to make irq_disabled a local variable in pci_default_write_config and in pci_device_load, and pass old value to pci_update_irq_disabled. I tried and it seems more code, and routines become less self-contained. As it is, I think it's cleaner because we have an idem-potent routine that is always safe it call, just like pci_update_mappings.
Quoted 5 lines
As for
irq_status, only user ofirq_statusispci_update_irq_status()so if (irq_statue) can be open coded. On the other hand, PCIBus already hasirq_countmember for same purpose.
So probably open coding or introducingirq_countinstead ofirq_statuswould be reasonable.
No, this would slow us down because these are per-pin. We need a sum of interrupts so that config space
can be updated by a single command.
Interrupts are a fastpath, extra loops there should be avoided.
Quoted 15 lines
Quoted 12 lines
Michael S. Tsirkin (4):
pci: rearrange code for interrupts
pci: track IRQ status
pci: interrupt status bit implementation
pci: interrupt disable bit supporthw/pci.c | 83 +++++++++++++++++++++++++++++++++++++++++++++++++++++--------- hw/pci.h | 8 ++++++ 2 files changed, 79 insertions(+), 12 deletions(-)Signature · 1 line
yamahata
Quoted 4 lines
No, this would slow us down because these are per-pin. We need a sum of interrupts so that config space
can be updated by a single command.
Interrupts are a fastpath, extra loops there should be avoided.
It's really not that much of a fast path. Unless you're doing something particularly obscure then even under heavy load you're unlikely to exceed a few kHz. Compared to the average PIC implementation, and the overhead of the actual CPU interrupt, I find it hard to believe that looping over precisely 4 entries has any real performance hit.
Paul
Quoted 8 lines · Paul Brook
Quoted 4 lines
No, this would slow us down because these are per-pin. We need a sum of interrupts so that config space
can be updated by a single command.
Interrupts are a fastpath, extra loops there should be avoided.It's really not that much of a fast path. Unless you're doing something particularly obscure then even under heavy load you're unlikely to exceed a few kHz.
I think with kvm, heavy disk stressing benchmark can get higher.
Quoted 5 lines
Compared to the average PIC implementation, and the overhead of the actual CPU interrupt, I find it hard to believe that looping over precisely 4 entries has any real performance hit.
Paul
I don't think it is major, but I definitely have seen, in the past, that extra branches and memory accesses have small but measureable effect when taken in interrupt handler routines in drivers, and same should apply here.
OTOH keeping the sum around is trivial.
It might not be easily measureable now, but IMO that's just because the whole interrupt delivery is so complex. E.g. we currently have there a loop re-computing interrupt routing on each access, this is just silly and I intend to fix it. I would rather not introduce more code that will have to be cleaned up later.
Signature · 1 line
MST
Quoted 5 lines
Quoted 3 lines
It's really not that much of a fast path. Unless you're doing something particularly obscure then even under heavy load you're unlikely to exceed a few kHz.
I think with kvm, heavy disk stressing benchmark can get higher.
I'd still expect this to be the least of your problems.
If nothing else you've at least one host signal delivery and/or thread context switch in there. Not to mention the overhead to forwarding the interrupt to the guest CPU.
Quoted 10 lines
Quoted 3 lines
Compared to the average PIC implementation, and the overhead of the actual CPU interrupt, I find it hard to believe that looping over precisely 4 entries has any real performance hit.
I don't think it is major, but I definitely have seen, in the past, that extra branches and memory accesses have small but measureable effect when taken in interrupt handler routines in drivers, and same should apply here.
OTOH keeping the sum around is trivial.
Not entirely. You now have two different bits of information that you have to keep consistent.
Unless you can show that this is performance critical code I strongly recommend keeping it as simple as possible.
Paul
Quoted 10 lines · Paul Brook
Quoted 5 lines
Quoted 3 lines
It's really not that much of a fast path. Unless you're doing something particularly obscure then even under heavy load you're unlikely to exceed a few kHz.
I think with kvm, heavy disk stressing benchmark can get higher.
I'd still expect this to be the least of your problems.
If nothing else you've at least one host signal delivery and/or thread context switch in there.
iotread which does the signalling might be running in parallel with the guest CPU.
Quoted 2 lines
Not to mention the overhead to forwarding the interrupt to the guest CPU.
This is often mitigated as KVM knows to inject the interrupt on the next vmexit.
Quoted 9 lines
Quoted 9 lines
Quoted 3 lines
Compared to the average PIC implementation, and the overhead of the actual CPU interrupt, I find it hard to believe that looping over precisely 4 entries has any real performance hit.
I don't think it is major, but I definitely have seen, in the past, that extra branches and memory accesses have small but measureable effect when taken in interrupt handler routines in drivers, and same should apply here.
Quoted 4 lines
Quoted 1 line
OTOH keeping the sum around is trivial.
Not entirely. You now have two different bits of information that you have to keep consistent.
This is inherent in pci spec definition: interrupt status bit in config space duplicates interrupt state.
Quoted 4 lines
Unless you can show that this is performance critical code I strongly recommend keeping it as simple as possible.
Paul
I don't see there is anything left show: interrupt delivery is *obviously* performance critical: people are running *latency benchmarks* measuring how fast a packet can get from an external interface into guest, in microseconds. We definitely want to remove obvious waste there.
Signature · 1 line
MST
Quoted 1 line · Isaku Yamahata
At least I think
irq_disablecan be removed
The following patch on top of mine removes irq_disabled field in PCIDevice. I am of two minds whether this makes the code better. What is your opinion?
hw/pci.c +16 -14
@@ -128,15 +128,18 @@ static void pci_change_irq_level(PCIDevice *pci_dev, int irq_num, int change) bus->set_irq (bus->irq_opaque, irq_num, bus->irq_count[irq_num] !=0 ); } -/* Update irq disabled field after config space change, -* assert/deassert interrupts if necessary. */ -static void pci_update_irq_disabled (PCIDevice *d) +static inline int pci_irq_disabled (PCIDevice *d) { -int i; -int disabled = !!(pci_get_word (d->config +PCI_COMMAND ) & -PCI_COMMAND_INTX_DISABLE ); -if (disabled == d->irq_disabled) +return !!(pci_get_word (d->config +PCI_COMMAND ) &PCI_COMMAND_INTX_DISABLE ); +} + +/* Called after interrupt disabled field update in config space, +* assert/deassert interrupts if necessary. +* Gets original interrupt disable bit value (before update). */ +static void pci_update_irq_disabled (PCIDevice *d,int was_irq_disabled) +{ +int i, disabled =pci_irq_disabled (d); +if (disabled == was_irq_disabled)return ; - d->irq_disabled = disabled;for (i =0 ; i <ARRAY_SIZE (d->irq_state); ++i) {int state = d->irq_state[i];
@@ -150,7 +153,6 @@ static void pci_device_reset(PCIDevice *dev)memset (dev->irq_state,0 ,sizeof dev->irq_state); dev->irq_status =0 ; - dev->irq_disabled =0 ;pci_update_irq_status (dev); dev->config[PCI_COMMAND ] &= ~(PCI_COMMAND_IO |PCI_COMMAND_MEMORY |PCI_COMMAND_MASTER );
@@ -369,14 +371,14 @@ void pci_device_save(PCIDevice *s, QEMUFile *f)int pci_device_load (PCIDevice *s,QEMUFile *f) { -int ret, i; +int ret, i, was_irq_disabled =pci_irq_disabled (d); ret =vmstate_load_state (f,pci_get_vmstate (s), s, s->version_id);for (i =0 ; i <ARRAY_SIZE (s->irq_state); ++i) { s->irq_status += s->irq_state[i]; }/* Restore the interrupt status bit. */ pci_update_irq_status (s); -pci_update_irq_disabled (s); +pci_update_irq_disabled (s, was_irq_disabled);return ret; }
@@ -924,7 +926,7 @@ uint32_t pci_default_read_config(PCIDevice *d,void pci_default_write_config (PCIDevice *d,uint32_t addr,uint32_t val,int l) { -int i; +int i, was_irq_disabled =pci_irq_disabled (d);uint32_t config_size =pci_config_size (d);for (i =0 ; i < l && addr + i < config_size; val >>=8 , ++i) {
@@ -938,7 +940,7 @@ void pci_default_write_config(PCIDevice *d, uint32_t addr, uint32_t val, int l)pci_update_mappings (d);if (range_covers_byte (addr, l,PCI_COMMAND )) -pci_update_irq_disabled (d); +pci_update_irq_disabled (d, was_irq_disabled); }/***********************************************************/
@@ -957,7 +959,7 @@ static void pci_set_irq(void *opaque, int irq_num, int level) pci_dev->irq_state[irq_num] = level; pci_dev->irq_status += change;pci_update_irq_status (pci_dev); -if (pci_dev->irq_disabled) +if (pci_irq_disabled (pci_dev))return ;pci_change_irq_level (pci_dev, irq_num, change); }
hw/pci.h +0 -3
@@ -227,9 +227,6 @@ struct PCIDevice {/* Sum of all irq levels. Used to implement irq status register. */ int irq_status; -/* Whether interrupts are disabled by command bit. */ -int irq_disabled; -/* Capability bits */ uint32_t cap_present;
Quoted 6 lines · Michael S. Tsirkin
Quoted 1 line · Isaku Yamahata
At least I think
irq_disablecan be removedThe following patch on top of mine removes
irq_disabledfield in PCIDevice. I am of two minds whether this makes the code better. What is your opinion?
Thinking about this some more, I really like the original approach better, mostly because it helps keep the fastpath streamlined.
On thing to note is that irq_disabled field was in the same or adjacent cache line with irq_state, and it is accessed on fast path. Reading config space instead will access an extra cache line which would be cold otherwise (config space is normally not accessed on fast path), and also needs an extra memory access (it is allocated on heap separately).
As I said separately, the whole of interrupt delivery is far from optimal, but we have to start somewhere.
Quoted 97 lines
hw/pci.c +16
-14@@ -128,15 +128,18 @@ static void pci_change_irq_level(PCIDevice *pci_dev, int irq_num, int change) bus->set_irq (bus->irq_opaque, irq_num, bus->irq_count[irq_num] !=0 ); } -/* Update irq disabled field after config space change, -* assert/deassert interrupts if necessary. */ -static void pci_update_irq_disabled (PCIDevice *d) +static inline int pci_irq_disabled (PCIDevice *d) { -int i; -int disabled = !!(pci_get_word (d->config +PCI_COMMAND ) & -PCI_COMMAND_INTX_DISABLE ); -if (disabled == d->irq_disabled) +return !!(pci_get_word (d->config +PCI_COMMAND ) &PCI_COMMAND_INTX_DISABLE ); +} + +/* Called after interrupt disabled field update in config space, +* assert/deassert interrupts if necessary. +* Gets original interrupt disable bit value (before update). */ +static void pci_update_irq_disabled (PCIDevice *d,int was_irq_disabled) +{ +int i, disabled =pci_irq_disabled (d); +if (disabled == was_irq_disabled)return ; - d->irq_disabled = disabled;for (i =0 ; i <ARRAY_SIZE (d->irq_state); ++i) {int state = d->irq_state[i];@@ -150,7 +153,6 @@ static void pci_device_reset(PCIDevice *dev)memset (dev->irq_state,0 ,sizeof dev->irq_state); dev->irq_status =0 ; - dev->irq_disabled =0 ;pci_update_irq_status (dev); dev->config[PCI_COMMAND ] &= ~(PCI_COMMAND_IO |PCI_COMMAND_MEMORY |PCI_COMMAND_MASTER );@@ -369,14 +371,14 @@ void pci_device_save(PCIDevice *s, QEMUFile *f)int pci_device_load (PCIDevice *s,QEMUFile *f) { -int ret, i; +int ret, i, was_irq_disabled =pci_irq_disabled (d); ret =vmstate_load_state (f,pci_get_vmstate (s), s, s->version_id);for (i =0 ; i <ARRAY_SIZE (s->irq_state); ++i) { s->irq_status += s->irq_state[i]; }/* Restore the interrupt status bit. */ pci_update_irq_status (s); -pci_update_irq_disabled (s); +pci_update_irq_disabled (s, was_irq_disabled);return ret; }@@ -924,7 +926,7 @@ uint32_t pci_default_read_config(PCIDevice *d,void pci_default_write_config (PCIDevice *d,uint32_t addr,uint32_t val,int l) { -int i; +int i, was_irq_disabled =pci_irq_disabled (d);uint32_t config_size =pci_config_size (d);for (i =0 ; i < l && addr + i < config_size; val >>=8 , ++i) {@@ -938,7 +940,7 @@ void pci_default_write_config(PCIDevice *d, uint32_t addr, uint32_t val, int l)pci_update_mappings (d);if (range_covers_byte (addr, l,PCI_COMMAND )) -pci_update_irq_disabled (d); +pci_update_irq_disabled (d, was_irq_disabled); }/***********************************************************/ @@ -957,7 +959,7 @@ static void pci_set_irq(void *opaque, int irq_num, int level) pci_dev->irq_state[irq_num] = level; pci_dev->irq_status += change;pci_update_irq_status (pci_dev); -if (pci_dev->irq_disabled) +if (pci_irq_disabled (pci_dev))return ;pci_change_irq_level (pci_dev, irq_num, change); }hw/pci.h +0
-3@@ -227,9 +227,6 @@ struct PCIDevice {/* Sum of all irq levels. Used to implement irq status register. */ int irq_status; -/* Whether interrupts are disabled by command bit. */ -int irq_disabled; -/* Capability bits */ uint32_t cap_present;