qemu: msi irq allocation api

define api for allocating/setting up msi-x irqs, and for updating them
with msi-x vector information, supply implementation in ioapic. Please comment on this API: I intend to port my msi-x patch to work on top of it.

Signed-off-by Michael S. Tsirkin <[email protected]>

define api for allocating/setting up msi-x irqs, and for updating them
with msi-x vector information, supply implementation in ioapic. Please comment on this API: I intend to port my msi-x patch to work on top of it.

Signed-off-by Michael S. Tsirkin <[email protected]>

Sparc64 also uses packets ("mondos", not implemented yet) for interrupt vector data, there the packet size is 8 * 64 bits. I think we should aim for a more generic API that covers this case also.

Sparc64 also uses packets ("mondos", not implemented yet) for interrupt vector data, there the packet size is 8 * 64 bits. I think we should aim for a more generic API that covers this case also.

Quoted 6 lines · Blue Swirl

Sparc64 also uses packets ("mondos", not implemented yet) for interrupt vector data, there the packet size is 8 * 64 bits. I think we should aim for a more generic API that covers this case also.

Is the packet structure visible to software?

Quoted 6 lines · Blue Swirl

Sparc64 also uses packets ("mondos", not implemented yet) for interrupt vector data, there the packet size is 8 * 64 bits. I think we should aim for a more generic API that covers this case also.

Is the packet structure visible to software?

Quoted 6 lines · Michael S. Tsirkin

define api for allocating/setting up msi-x irqs, and for updating them
with msi-x vector information, supply implementation in ioapic. Please comment on this API: I intend to port my msi-x patch to work on top of it.

Signed-off-by Michael S. Tsirkin <[email protected]>

Sparc64 also uses packets ("mondos", not implemented yet) for interrupt vector data, there the packet size is 8 * 64 bits. I think we should aim for a more generic API that covers this case also.

For example, irq.c could support opaque packet payload of unspecified/predefined size. MSI packet structure should be defined in ioapic.c.

The pci_msi_ops structure could be 'const', or do you expect it to change during execution?

Quoted 6 lines · Michael S. Tsirkin

define api for allocating/setting up msi-x irqs, and for updating them
with msi-x vector information, supply implementation in ioapic. Please comment on this API: I intend to port my msi-x patch to work on top of it.

Signed-off-by Michael S. Tsirkin <[email protected]>

Sparc64 also uses packets ("mondos", not implemented yet) for interrupt vector data, there the packet size is 8 * 64 bits. I think we should aim for a more generic API that covers this case also.

For example, irq.c could support opaque packet payload of unspecified/predefined size. MSI packet structure should be defined in ioapic.c.

The pci_msi_ops structure could be 'const', or do you expect it to change during execution?

Quoted 8 lines · Blue Swirl

Quoted 6 lines · Michael S. Tsirkin

define api for allocating/setting up msi-x irqs, and for updating them
with msi-x vector information, supply implementation in ioapic. Please comment on this API: I intend to port my msi-x patch to work on top of it.

Signed-off-by Michael S. Tsirkin <[email protected]>

Quoted 3 lines

Sparc64 also uses packets ("mondos", not implemented yet) for interrupt vector data, there the packet size is 8 * 64 bits. I think we should aim for a more generic API that covers this case also.

Are you sure this is a good idea? MSI is tied to PCI, and PCI only has
MSI, not "mondos". What code would benefit from this abstraction?

Quoted 3 lines

For example, irq.c could support opaque packet payload of unspecified/predefined size. MSI packet structure should be defined in ioapic.c.

Note that MSI does not have packets and MSI interrupts do not pass any payload.

Quoted 9 lines · Michael S. Tsirkin

Quoted 8 lines · Blue Swirl

Quoted 6 lines · Michael S. Tsirkin

define api for allocating/setting up msi-x irqs, and for updating them
with msi-x vector information, supply implementation in ioapic. Please comment on this API: I intend to port my msi-x patch to work on top of it.

Signed-off-by Michael S. Tsirkin <[email protected]>

Quoted 7 lines

Quoted 3 lines

Sparc64 also uses packets ("mondos", not implemented yet) for interrupt vector data, there the packet size is 8 * 64 bits. I think we should aim for a more generic API that covers this case also.

Are you sure this is a good idea? MSI is tied to PCI, and PCI only has
MSI, not "mondos". What code would benefit from this abstraction?

Sparc64 emulation, of course. I think also the API would be neater.

Quoted 6 lines

Quoted 3 lines

For example, irq.c could support opaque packet payload of unspecified/predefined size. MSI packet structure should be defined in ioapic.c.

Note that MSI does not have packets and MSI interrupts do not pass any payload.

I don't know too much about MSI, what's the 'data' field in msi_state then?

Quoted 9 lines · Michael S. Tsirkin

Quoted 8 lines · Blue Swirl

Quoted 6 lines · Michael S. Tsirkin

define api for allocating/setting up msi-x irqs, and for updating them
with msi-x vector information, supply implementation in ioapic. Please comment on this API: I intend to port my msi-x patch to work on top of it.

Signed-off-by Michael S. Tsirkin <[email protected]>

Quoted 7 lines

Quoted 3 lines

Sparc64 also uses packets ("mondos", not implemented yet) for interrupt vector data, there the packet size is 8 * 64 bits. I think we should aim for a more generic API that covers this case also.

Are you sure this is a good idea? MSI is tied to PCI, and PCI only has
MSI, not "mondos". What code would benefit from this abstraction?

Sparc64 emulation, of course. I think also the API would be neater.

Quoted 6 lines

Quoted 3 lines

For example, irq.c could support opaque packet payload of unspecified/predefined size. MSI packet structure should be defined in ioapic.c.

Note that MSI does not have packets and MSI interrupts do not pass any payload.

I don't know too much about MSI, what's the 'data' field in msi_state then?

Quoted 14 lines · Michael S. Tsirkin

Quoted 11 lines · Blue Swirl

Quoted 6 lines · Michael S. Tsirkin

define api for allocating/setting up msi-x irqs, and for updating them
with msi-x vector information, supply implementation in ioapic. Please comment on this API: I intend to port my msi-x patch to work on top of it.

Signed-off-by Michael S. Tsirkin <[email protected]>

Sparc64 also uses packets ("mondos", not implemented yet) for interrupt vector data, there the packet size is 8 * 64 bits. I think we should aim for a more generic API that covers this case also.

Quoted 2 lines

Are you sure this is a good idea? MSI is tied to PCI, and PCI only has
MSI, not "mondos". What code would benefit from this abstraction?

Sparc64 emulation, of course. I think also the API would be neater.

Quoted 15 lines · Blue Swirl

Quoted 14 lines · Michael S. Tsirkin

Quoted 11 lines · Blue Swirl

Quoted 6 lines · Michael S. Tsirkin

define api for allocating/setting up msi-x irqs, and for updating them
with msi-x vector information, supply implementation in ioapic. Please comment on this API: I intend to port my msi-x patch to work on top of it.

Signed-off-by Michael S. Tsirkin <[email protected]>

Sparc64 also uses packets ("mondos", not implemented yet) for interrupt vector data, there the packet size is 8 * 64 bits. I think we should aim for a more generic API that covers this case also.

Quoted 4 lines

Quoted 2 lines

Are you sure this is a good idea? MSI is tied to PCI, and PCI only has
MSI, not "mondos". What code would benefit from this abstraction?

Sparc64 emulation, of course. I think also the API would be neater.

Since "mondos" are not interrupts, why use irqs for them?

Quoted 15 lines · Blue Swirl

Quoted 14 lines · Michael S. Tsirkin

Quoted 11 lines · Blue Swirl

Quoted 6 lines · Michael S. Tsirkin

define api for allocating/setting up msi-x irqs, and for updating them
with msi-x vector information, supply implementation in ioapic. Please comment on this API: I intend to port my msi-x patch to work on top of it.

Signed-off-by Michael S. Tsirkin <[email protected]>

Sparc64 also uses packets ("mondos", not implemented yet) for interrupt vector data, there the packet size is 8 * 64 bits. I think we should aim for a more generic API that covers this case also.

Quoted 4 lines

Quoted 2 lines

Are you sure this is a good idea? MSI is tied to PCI, and PCI only has
MSI, not "mondos". What code would benefit from this abstraction?

Sparc64 emulation, of course. I think also the API would be neater.

Since "mondos" are not interrupts, why use irqs for them?

Quoted 19 lines · Michael S. Tsirkin

Quoted 18 lines · Blue Swirl

Quoted 16 lines · Michael S. Tsirkin

Quoted 11 lines · Blue Swirl

Quoted 6 lines · Michael S. Tsirkin

define api for allocating/setting up msi-x irqs, and for updating them
with msi-x vector information, supply implementation in ioapic. Please comment on this API: I intend to port my msi-x patch to work on top of it.

Signed-off-by Michael S. Tsirkin <[email protected]>

Sparc64 also uses packets ("mondos", not implemented yet) for interrupt vector data, there the packet size is 8 * 64 bits. I think we should aim for a more generic API that covers this case also.

Are you sure this is a good idea? MSI is tied to PCI, and PCI only has
MSI, not "mondos". What code would benefit from this abstraction?

Quoted 4 lines

Quoted 1 line

Sparc64 emulation, of course. I think also the API would be neater.

Since "mondos" are not interrupts, why use irqs for them?

I just said above that they are used for interrupt vector data. What makes you think they are not interrupts?

Quoted 19 lines · Michael S. Tsirkin

Quoted 18 lines · Blue Swirl

Quoted 16 lines · Michael S. Tsirkin

Quoted 11 lines · Blue Swirl

Quoted 6 lines · Michael S. Tsirkin

define api for allocating/setting up msi-x irqs, and for updating them
with msi-x vector information, supply implementation in ioapic. Please comment on this API: I intend to port my msi-x patch to work on top of it.

Signed-off-by Michael S. Tsirkin <[email protected]>

Sparc64 also uses packets ("mondos", not implemented yet) for interrupt vector data, there the packet size is 8 * 64 bits. I think we should aim for a more generic API that covers this case also.

Are you sure this is a good idea? MSI is tied to PCI, and PCI only has
MSI, not "mondos". What code would benefit from this abstraction?

Quoted 4 lines

Quoted 1 line

Sparc64 emulation, of course. I think also the API would be neater.

Since "mondos" are not interrupts, why use irqs for them?

I just said above that they are used for interrupt vector data. What makes you think they are not interrupts?

Quoted 20 lines · Blue Swirl

Quoted 19 lines · Michael S. Tsirkin

Quoted 18 lines · Blue Swirl

Quoted 16 lines · Michael S. Tsirkin

Quoted 11 lines · Blue Swirl

Quoted 6 lines · Michael S. Tsirkin

define api for allocating/setting up msi-x irqs, and for updating them
with msi-x vector information, supply implementation in ioapic. Please comment on this API: I intend to port my msi-x patch to work on top of it.

Signed-off-by Michael S. Tsirkin <[email protected]>

Sparc64 also uses packets ("mondos", not implemented yet) for interrupt vector data, there the packet size is 8 * 64 bits. I think we should aim for a more generic API that covers this case also.

Are you sure this is a good idea? MSI is tied to PCI, and PCI only has
MSI, not "mondos". What code would benefit from this abstraction?

Quoted 7 lines

Quoted 4 lines

Quoted 1 line

Sparc64 emulation, of course. I think also the API would be neater.

Since "mondos" are not interrupts, why use irqs for them?

I just said above that they are used for interrupt vector data. What makes you think they are not interrupts?

I'm sorry, I don't really know anything about sparc. All I am saying is that in PCI, interrupts never pass data, so qemu_set_irq as it is now, is a good API to send them.

For the sparc feature you describe, you probably want to add a message data parameter to qemu_set_irq, but it's not really useful for MSI.

Quoted 20 lines · Blue Swirl

Quoted 19 lines · Michael S. Tsirkin

Quoted 18 lines · Blue Swirl

Quoted 16 lines · Michael S. Tsirkin

Quoted 11 lines · Blue Swirl

Quoted 6 lines · Michael S. Tsirkin

define api for allocating/setting up msi-x irqs, and for updating them
with msi-x vector information, supply implementation in ioapic. Please comment on this API: I intend to port my msi-x patch to work on top of it.

Signed-off-by Michael S. Tsirkin <[email protected]>

Sparc64 also uses packets ("mondos", not implemented yet) for interrupt vector data, there the packet size is 8 * 64 bits. I think we should aim for a more generic API that covers this case also.

Are you sure this is a good idea? MSI is tied to PCI, and PCI only has
MSI, not "mondos". What code would benefit from this abstraction?

Quoted 7 lines

Quoted 4 lines

Quoted 1 line

Sparc64 emulation, of course. I think also the API would be neater.

Since "mondos" are not interrupts, why use irqs for them?

I just said above that they are used for interrupt vector data. What makes you think they are not interrupts?

I'm sorry, I don't really know anything about sparc.
All I am saying is that in PCI, interrupts never pass data, so qemu_set_irq as it is now, is a good API to send them.

For the sparc feature you describe, you probably want to add a message data parameter to qemu_set_irq, but it's not really useful for MSI.

Quoted 34 lines · Michael S. Tsirkin

Quoted 20 lines · Blue Swirl

Quoted 19 lines · Michael S. Tsirkin

Quoted 18 lines · Blue Swirl

Quoted 16 lines · Michael S. Tsirkin

Quoted 11 lines · Blue Swirl

Quoted 6 lines · Michael S. Tsirkin

define api for allocating/setting up msi-x irqs, and for updating them
with msi-x vector information, supply implementation in ioapic. Please comment on this API: I intend to port my msi-x patch to work on top of it.

Signed-off-by Michael S. Tsirkin <[email protected]>

Sparc64 also uses packets ("mondos", not implemented yet) for interrupt vector data, there the packet size is 8 * 64 bits. I think we should aim for a more generic API that covers this case also.

Are you sure this is a good idea? MSI is tied to PCI, and PCI only has
MSI, not "mondos". What code would benefit from this abstraction?

Quoted 7 lines

Quoted 4 lines

Quoted 1 line

Sparc64 emulation, of course. I think also the API would be neater.

Since "mondos" are not interrupts, why use irqs for them?

I just said above that they are used for interrupt vector data. What makes you think they are not interrupts?

I'm sorry, I don't really know anything about sparc.
All I am saying is that in PCI, interrupts never pass data, so qemu_set_irq as it is now, is a good API to send them.

Quoted 3 lines

For the sparc feature you describe, you probably want to add a message data parameter to qemu_set_irq, but it's not really useful for MSI.

But maybe this API would be useful for both MSI and mondo packets:
/* Get vector data stored in the irq */
void *qemu_irq_get_data(qemu_irq irq);

/* Set vector data stored in the irq */
void qemu_irq_set_data(qemu_irq irq, void *data); You'd use a struct msi_state as the opaque (not using the array indexed by irq number), I'd use some kind of
struct mondo {
uint64_t packet[8];
};

Instead of ioapic_update_msi(), you'd call qemu_irq_set_data().

Quoted 34 lines · Michael S. Tsirkin

Quoted 20 lines · Blue Swirl

Quoted 19 lines · Michael S. Tsirkin

Quoted 18 lines · Blue Swirl

Quoted 16 lines · Michael S. Tsirkin

Quoted 11 lines · Blue Swirl

Quoted 6 lines · Michael S. Tsirkin

define api for allocating/setting up msi-x irqs, and for updating them
with msi-x vector information, supply implementation in ioapic. Please comment on this API: I intend to port my msi-x patch to work on top of it.

Signed-off-by Michael S. Tsirkin <[email protected]>

Sparc64 also uses packets ("mondos", not implemented yet) for interrupt vector data, there the packet size is 8 * 64 bits. I think we should aim for a more generic API that covers this case also.

Are you sure this is a good idea? MSI is tied to PCI, and PCI only has
MSI, not "mondos". What code would benefit from this abstraction?

Quoted 7 lines

Quoted 4 lines

Quoted 1 line

Sparc64 emulation, of course. I think also the API would be neater.

Since "mondos" are not interrupts, why use irqs for them?

I just said above that they are used for interrupt vector data. What makes you think they are not interrupts?

I'm sorry, I don't really know anything about sparc.
All I am saying is that in PCI, interrupts never pass data, so qemu_set_irq as it is now, is a good API to send them.

Quoted 3 lines

For the sparc feature you describe, you probably want to add a message data parameter to qemu_set_irq, but it's not really useful for MSI.

But maybe this API would be useful for both MSI and mondo packets:
/* Get vector data stored in the irq */
void *qemu_irq_get_data(qemu_irq irq);

/* Set vector data stored in the irq */
void qemu_irq_set_data(qemu_irq irq, void *data); You'd use a struct msi_state as the opaque (not using the array indexed by irq number), I'd use some kind of
struct mondo {
uint64_t packet[8];
};

Instead of ioapic_update_msi(), you'd call qemu_irq_set_data().

Quoted 20 lines · Blue Swirl

Quoted 19 lines · Michael S. Tsirkin

Quoted 18 lines · Blue Swirl

Quoted 16 lines · Michael S. Tsirkin

Quoted 11 lines · Blue Swirl

Quoted 6 lines · Michael S. Tsirkin

define api for allocating/setting up msi-x irqs, and for updating them
with msi-x vector information, supply implementation in ioapic. Please comment on this API: I intend to port my msi-x patch to work on top of it.

Signed-off-by Michael S. Tsirkin <[email protected]>

Sparc64 also uses packets ("mondos", not implemented yet) for interrupt vector data, there the packet size is 8 * 64 bits. I think we should aim for a more generic API that covers this case also.

Are you sure this is a good idea? MSI is tied to PCI, and PCI only has
MSI, not "mondos". What code would benefit from this abstraction?

Quoted 7 lines

Quoted 4 lines

Quoted 1 line

Sparc64 emulation, of course. I think also the API would be neater.

Since "mondos" are not interrupts, why use irqs for them?

I just said above that they are used for interrupt vector data. What makes you think they are not interrupts?

I'm sorry, I don't really know anything about sparc.
All I am saying is that in PCI, interrupts never pass data, so qemu_set_irq as it is now, is a good API to send them.

For the sparc feature you describe, you probably want to add a message data parameter to qemu_set_irq, but it's not really useful for MSI.

Quoted 20 lines · Blue Swirl

Quoted 19 lines · Michael S. Tsirkin

Quoted 18 lines · Blue Swirl

Quoted 16 lines · Michael S. Tsirkin

Quoted 11 lines · Blue Swirl

Quoted 6 lines · Michael S. Tsirkin

define api for allocating/setting up msi-x irqs, and for updating them
with msi-x vector information, supply implementation in ioapic. Please comment on this API: I intend to port my msi-x patch to work on top of it.

Signed-off-by Michael S. Tsirkin <[email protected]>

Sparc64 also uses packets ("mondos", not implemented yet) for interrupt vector data, there the packet size is 8 * 64 bits. I think we should aim for a more generic API that covers this case also.

Are you sure this is a good idea? MSI is tied to PCI, and PCI only has
MSI, not "mondos". What code would benefit from this abstraction?

Quoted 7 lines

Quoted 4 lines

Quoted 1 line

Sparc64 emulation, of course. I think also the API would be neater.

Since "mondos" are not interrupts, why use irqs for them?

I just said above that they are used for interrupt vector data. What makes you think they are not interrupts?

I'm sorry, I don't really know anything about sparc. All I am saying is that in PCI, interrupts never pass data, so qemu_set_irq as it is now, is a good API to send them.

For the sparc feature you describe, you probably want to add a message data parameter to qemu_set_irq, but it's not really useful for MSI.

Quoted 29 lines · Michael S. Tsirkin

Quoted 20 lines · Blue Swirl

Quoted 19 lines · Michael S. Tsirkin

Quoted 18 lines · Blue Swirl

Quoted 16 lines · Michael S. Tsirkin

Quoted 11 lines · Blue Swirl

Quoted 6 lines · Michael S. Tsirkin

define api for allocating/setting up msi-x irqs, and for updating them
with msi-x vector information, supply implementation in ioapic. Please comment on this API: I intend to port my msi-x patch to work on top of it.

Signed-off-by Michael S. Tsirkin <[email protected]>

Sparc64 also uses packets ("mondos", not implemented yet) for interrupt vector data, there the packet size is 8 * 64 bits. I think we should aim for a more generic API that covers this case also.

Are you sure this is a good idea? MSI is tied to PCI, and PCI only has
MSI, not "mondos". What code would benefit from this abstraction?

Quoted 7 lines

Quoted 4 lines

Quoted 1 line

Sparc64 emulation, of course. I think also the API would be neater.

Since "mondos" are not interrupts, why use irqs for them?

I just said above that they are used for interrupt vector data. What makes you think they are not interrupts?

Quoted 7 lines

I'm sorry, I don't really know anything about sparc. All I am saying is that in PCI, interrupts never pass data, so qemu_set_irq as it is now, is a good API to send them.

For the sparc feature you describe, you probably want to add a message data parameter to qemu_set_irq, but it's not really useful for MSI.

Just to clarify, the main difference is that with MSI/MSI-X
both data and address fields are mostly static, modifying them involves ioapic and device updates which might be an expensive operation (e.g. with kvm, needs an extra system call).

So I don't think it makes sense to pass MSI-X data field with each call to qemu_set_irq.

Quoted 44 lines · Michael S. Tsirkin

Quoted 29 lines · Michael S. Tsirkin

Quoted 20 lines · Blue Swirl

Quoted 19 lines · Michael S. Tsirkin

Quoted 18 lines · Blue Swirl

Quoted 16 lines · Michael S. Tsirkin

Quoted 11 lines · Blue Swirl

Quoted 6 lines · Michael S. Tsirkin

define api for allocating/setting up msi-x irqs, and for updating them
with msi-x vector information, supply implementation in ioapic. Please comment on this API: I intend to port my msi-x patch to work on top of it.

Signed-off-by Michael S. Tsirkin <[email protected]>

Sparc64 also uses packets ("mondos", not implemented yet) for interrupt vector data, there the packet size is 8 * 64 bits. I think we should aim for a more generic API that covers this case also.

Are you sure this is a good idea? MSI is tied to PCI, and PCI only has
MSI, not "mondos". What code would benefit from this abstraction?

Quoted 7 lines

Quoted 4 lines

Quoted 1 line

Sparc64 emulation, of course. I think also the API would be neater.

Since "mondos" are not interrupts, why use irqs for them?

I just said above that they are used for interrupt vector data. What makes you think they are not interrupts?

Quoted 7 lines

I'm sorry, I don't really know anything about sparc. All I am saying is that in PCI, interrupts never pass data, so qemu_set_irq as it is now, is a good API to send them.

For the sparc feature you describe, you probably want to add a message data parameter to qemu_set_irq, but it's not really useful for MSI.

Just to clarify, the main difference is that with MSI/MSI-X
both data and address fields are mostly static, modifying them involves ioapic and device updates which might be an expensive operation (e.g. with kvm, needs an extra system call).

Quoted 2 lines

So I don't think it makes sense to pass MSI-X data field with each call to qemu_set_irq.

No, but I think the Sparc situation is the same, the packet data is static for the interrupt source in question.

Quoted 44 lines · Michael S. Tsirkin

Quoted 29 lines · Michael S. Tsirkin

Quoted 20 lines · Blue Swirl

Quoted 19 lines · Michael S. Tsirkin

Quoted 18 lines · Blue Swirl

Quoted 16 lines · Michael S. Tsirkin

Quoted 11 lines · Blue Swirl

Quoted 6 lines · Michael S. Tsirkin

define api for allocating/setting up msi-x irqs, and for updating them
with msi-x vector information, supply implementation in ioapic. Please comment on this API: I intend to port my msi-x patch to work on top of it.

Signed-off-by Michael S. Tsirkin <[email protected]>

Sparc64 also uses packets ("mondos", not implemented yet) for interrupt vector data, there the packet size is 8 * 64 bits. I think we should aim for a more generic API that covers this case also.

Are you sure this is a good idea? MSI is tied to PCI, and PCI only has
MSI, not "mondos". What code would benefit from this abstraction?

Quoted 7 lines

Quoted 4 lines

Quoted 1 line

Sparc64 emulation, of course. I think also the API would be neater.

Since "mondos" are not interrupts, why use irqs for them?

I just said above that they are used for interrupt vector data. What makes you think they are not interrupts?

Quoted 7 lines

I'm sorry, I don't really know anything about sparc. All I am saying is that in PCI, interrupts never pass data, so qemu_set_irq as it is now, is a good API to send them.

For the sparc feature you describe, you probably want to add a message data parameter to qemu_set_irq, but it's not really useful for MSI.

Just to clarify, the main difference is that with MSI/MSI-X
both data and address fields are mostly static, modifying them involves ioapic and device updates which might be an expensive operation (e.g. with kvm, needs an extra system call).

Quoted 2 lines

So I don't think it makes sense to pass MSI-X data field with each call to qemu_set_irq.

No, but I think the Sparc situation is the same, the packet data is static for the interrupt source in question.

Quoted 45 lines · Blue Swirl

Quoted 44 lines · Michael S. Tsirkin

Quoted 29 lines · Michael S. Tsirkin

Quoted 20 lines · Blue Swirl

Quoted 19 lines · Michael S. Tsirkin

Quoted 18 lines · Blue Swirl

Quoted 16 lines · Michael S. Tsirkin

Quoted 11 lines · Blue Swirl

Quoted 6 lines · Michael S. Tsirkin

define api for allocating/setting up msi-x irqs, and for updating them
with msi-x vector information, supply implementation in ioapic. Please comment on this API: I intend to port my msi-x patch to work on top of it.

Signed-off-by Michael S. Tsirkin <[email protected]>

Sparc64 also uses packets ("mondos", not implemented yet) for interrupt vector data, there the packet size is 8 * 64 bits. I think we should aim for a more generic API that covers this case also.

Are you sure this is a good idea? MSI is tied to PCI, and PCI only has
MSI, not "mondos". What code would benefit from this abstraction?

Quoted 7 lines

Quoted 4 lines

Quoted 1 line

Sparc64 emulation, of course. I think also the API would be neater.

Since "mondos" are not interrupts, why use irqs for them?

I just said above that they are used for interrupt vector data. What makes you think they are not interrupts?

Quoted 7 lines

I'm sorry, I don't really know anything about sparc. All I am saying is that in PCI, interrupts never pass data, so qemu_set_irq as it is now, is a good API to send them.

For the sparc feature you describe, you probably want to add a message data parameter to qemu_set_irq, but it's not really useful for MSI.

Just to clarify, the main difference is that with MSI/MSI-X
both data and address fields are mostly static, modifying them involves ioapic and device updates which might be an expensive operation (e.g. with kvm, needs an extra system call).

Quoted 5 lines

Quoted 2 lines

So I don't think it makes sense to pass MSI-X data field with each call to qemu_set_irq.

No, but I think the Sparc situation is the same, the packet data is static for the interrupt source in question.

So, ok, we could add data update callback and then MSI and sparc
would do their thing there. I'm not convinced I like all this play with untyped buffers, do you think it's helpful?

If yes, maybe I'll try to code it up and see how does it look.

Quoted 45 lines · Blue Swirl

Quoted 44 lines · Michael S. Tsirkin

Quoted 29 lines · Michael S. Tsirkin

Quoted 20 lines · Blue Swirl

Quoted 19 lines · Michael S. Tsirkin

Quoted 18 lines · Blue Swirl

Quoted 16 lines · Michael S. Tsirkin

Quoted 11 lines · Blue Swirl

Quoted 6 lines · Michael S. Tsirkin

define api for allocating/setting up msi-x irqs, and for updating them
with msi-x vector information, supply implementation in ioapic. Please comment on this API: I intend to port my msi-x patch to work on top of it.

Signed-off-by Michael S. Tsirkin <[email protected]>

Sparc64 also uses packets ("mondos", not implemented yet) for interrupt vector data, there the packet size is 8 * 64 bits. I think we should aim for a more generic API that covers this case also.

Are you sure this is a good idea? MSI is tied to PCI, and PCI only has
MSI, not "mondos". What code would benefit from this abstraction?

Quoted 7 lines

Quoted 4 lines

Quoted 1 line

Sparc64 emulation, of course. I think also the API would be neater.

Since "mondos" are not interrupts, why use irqs for them?

I just said above that they are used for interrupt vector data. What makes you think they are not interrupts?

Quoted 7 lines

I'm sorry, I don't really know anything about sparc. All I am saying is that in PCI, interrupts never pass data, so qemu_set_irq as it is now, is a good API to send them.

For the sparc feature you describe, you probably want to add a message data parameter to qemu_set_irq, but it's not really useful for MSI.

Just to clarify, the main difference is that with MSI/MSI-X
both data and address fields are mostly static, modifying them involves ioapic and device updates which might be an expensive operation (e.g. with kvm, needs an extra system call).

Quoted 5 lines

Quoted 2 lines

So I don't think it makes sense to pass MSI-X data field with each call to qemu_set_irq.

No, but I think the Sparc situation is the same, the packet data is static for the interrupt source in question.

So, ok, we could add data update callback and then MSI and sparc
would do their thing there. I'm not convinced I like all this play with untyped buffers, do you think it's helpful?

If yes, maybe I'll try to code it up and see how does it look.

Quoted 57 lines · Michael S. Tsirkin

Quoted 45 lines · Blue Swirl

Quoted 44 lines · Michael S. Tsirkin

Quoted 29 lines · Michael S. Tsirkin

Quoted 20 lines · Blue Swirl

Quoted 19 lines · Michael S. Tsirkin

Quoted 18 lines · Blue Swirl

Quoted 16 lines · Michael S. Tsirkin

Quoted 11 lines · Blue Swirl

Quoted 6 lines · Michael S. Tsirkin

define api for allocating/setting up msi-x irqs, and for updating them
with msi-x vector information, supply implementation in ioapic. Please comment on this API: I intend to port my msi-x patch to work on top of it.

Signed-off-by Michael S. Tsirkin <[email protected]>

Sparc64 also uses packets ("mondos", not implemented yet) for interrupt vector data, there the packet size is 8 * 64 bits. I think we should aim for a more generic API that covers this case also.

Are you sure this is a good idea? MSI is tied to PCI, and PCI only has
MSI, not "mondos". What code would benefit from this abstraction?

Quoted 7 lines

Quoted 4 lines

Quoted 1 line

Sparc64 emulation, of course. I think also the API would be neater.

Since "mondos" are not interrupts, why use irqs for them?

I just said above that they are used for interrupt vector data. What makes you think they are not interrupts?

Quoted 7 lines

I'm sorry, I don't really know anything about sparc. All I am saying is that in PCI, interrupts never pass data, so qemu_set_irq as it is now, is a good API to send them.

For the sparc feature you describe, you probably want to add a message data parameter to qemu_set_irq, but it's not really useful for MSI.

Just to clarify, the main difference is that with MSI/MSI-X
both data and address fields are mostly static, modifying them involves ioapic and device updates which might be an expensive operation (e.g. with kvm, needs an extra system call).

Quoted 5 lines

Quoted 2 lines

So I don't think it makes sense to pass MSI-X data field with each call to qemu_set_irq.

No, but I think the Sparc situation is the same, the packet data is static for the interrupt source in question.

So, ok, we could add data update callback and then MSI and sparc
would do their thing there. I'm not convinced I like all this play with untyped buffers, do you think it's helpful?

Quoted 1 line

If yes, maybe I'll try to code it up and see how does it look.

Well, get/set_data could be limited to irq.c, ioapic.c could export something like get/set_msi_data. MSI callers should only use get/set_msi_data. Ditto for get/set_mondo_data.

Quoted 57 lines · Michael S. Tsirkin

Quoted 45 lines · Blue Swirl

Quoted 44 lines · Michael S. Tsirkin

Quoted 29 lines · Michael S. Tsirkin

Quoted 20 lines · Blue Swirl

Quoted 19 lines · Michael S. Tsirkin

Quoted 18 lines · Blue Swirl

Quoted 16 lines · Michael S. Tsirkin

Quoted 11 lines · Blue Swirl

Quoted 6 lines · Michael S. Tsirkin

define api for allocating/setting up msi-x irqs, and for updating them
with msi-x vector information, supply implementation in ioapic. Please comment on this API: I intend to port my msi-x patch to work on top of it.

Signed-off-by Michael S. Tsirkin <[email protected]>

Sparc64 also uses packets ("mondos", not implemented yet) for interrupt vector data, there the packet size is 8 * 64 bits. I think we should aim for a more generic API that covers this case also.

Are you sure this is a good idea? MSI is tied to PCI, and PCI only has
MSI, not "mondos". What code would benefit from this abstraction?

Quoted 7 lines

Quoted 4 lines

Quoted 1 line

Sparc64 emulation, of course. I think also the API would be neater.

Since "mondos" are not interrupts, why use irqs for them?

I just said above that they are used for interrupt vector data. What makes you think they are not interrupts?

Quoted 7 lines

I'm sorry, I don't really know anything about sparc. All I am saying is that in PCI, interrupts never pass data, so qemu_set_irq as it is now, is a good API to send them.

For the sparc feature you describe, you probably want to add a message data parameter to qemu_set_irq, but it's not really useful for MSI.

Just to clarify, the main difference is that with MSI/MSI-X
both data and address fields are mostly static, modifying them involves ioapic and device updates which might be an expensive operation (e.g. with kvm, needs an extra system call).

Quoted 5 lines

Quoted 2 lines

So I don't think it makes sense to pass MSI-X data field with each call to qemu_set_irq.

No, but I think the Sparc situation is the same, the packet data is static for the interrupt source in question.

So, ok, we could add data update callback and then MSI and sparc
would do their thing there. I'm not convinced I like all this play with untyped buffers, do you think it's helpful?

Quoted 1 line

If yes, maybe I'll try to code it up and see how does it look.

Well, get/set_data could be limited to irq.c, ioapic.c could export something like get/set_msi_data. MSI callers should only use get/set_msi_data. Ditto for get/set_mondo_data.

Quoted 29 lines · Michael S. Tsirkin

Quoted 20 lines · Blue Swirl

Quoted 19 lines · Michael S. Tsirkin

Quoted 18 lines · Blue Swirl

Quoted 16 lines · Michael S. Tsirkin

Quoted 11 lines · Blue Swirl

Quoted 6 lines · Michael S. Tsirkin

define api for allocating/setting up msi-x irqs, and for updating them
with msi-x vector information, supply implementation in ioapic. Please comment on this API: I intend to port my msi-x patch to work on top of it.

Signed-off-by Michael S. Tsirkin <[email protected]>

Sparc64 also uses packets ("mondos", not implemented yet) for interrupt vector data, there the packet size is 8 * 64 bits. I think we should aim for a more generic API that covers this case also.

Are you sure this is a good idea? MSI is tied to PCI, and PCI only has
MSI, not "mondos". What code would benefit from this abstraction?

Quoted 7 lines

Quoted 4 lines

Quoted 1 line

Sparc64 emulation, of course. I think also the API would be neater.

Since "mondos" are not interrupts, why use irqs for them?

I just said above that they are used for interrupt vector data. What makes you think they are not interrupts?

Quoted 7 lines

I'm sorry, I don't really know anything about sparc. All I am saying is that in PCI, interrupts never pass data, so qemu_set_irq as it is now, is a good API to send them.

For the sparc feature you describe, you probably want to add a message data parameter to qemu_set_irq, but it's not really useful for MSI.

Just to clarify, the main difference is that with MSI/MSI-X
both data and address fields are mostly static, modifying them involves ioapic and device updates which might be an expensive operation (e.g. with kvm, needs an extra system call).

So I don't think it makes sense to pass MSI-X data field with each call to qemu_set_irq.

Quoted 8 lines · Blue Swirl

Quoted 6 lines · Michael S. Tsirkin

define api for allocating/setting up msi-x irqs, and for updating them
with msi-x vector information, supply implementation in ioapic. Please comment on this API: I intend to port my msi-x patch to work on top of it.

Signed-off-by Michael S. Tsirkin <[email protected]>

Quoted 3 lines

Sparc64 also uses packets ("mondos", not implemented yet) for interrupt vector data, there the packet size is 8 * 64 bits. I think we should aim for a more generic API that covers this case also.

Are you sure this is a good idea? MSI is tied to PCI, and PCI only has
MSI, not "mondos". What code would benefit from this abstraction?

Quoted 3 lines

For example, irq.c could support opaque packet payload of unspecified/predefined size. MSI packet structure should be defined in ioapic.c.

Note that MSI does not have packets and MSI interrupts do not pass any payload.

define api for allocating/setting up msi-x irqs, and for updating them with msi-x vector information, supply implementation in ioapic. Please comment on this API: I intend to port my msi-x patch to work on top of it.

define api for allocating/setting up msi-x irqs, and for updating them with msi-x vector information, supply implementation in ioapic. Please comment on this API: I intend to port my msi-x patch to work on top of it.

Quoted 5 lines · Michael S. Tsirkin

define api for allocating/setting up msi-x irqs, and for updating them with msi-x vector information, supply implementation in ioapic. Please comment on this API: I intend to port my msi-x patch to work on top of it.

I though the point of MSI is that they are just a regular memory writes, and don't require any special bus support.

Quoted 5 lines · Michael S. Tsirkin

define api for allocating/setting up msi-x irqs, and for updating them with msi-x vector information, supply implementation in ioapic. Please comment on this API: I intend to port my msi-x patch to work on top of it.

I though the point of MSI is that they are just a regular memory writes, and don't require any special bus support.

The PCI bus doesn't need any special support (I think) but something on the other end needs to interpret those writes.

In any case we need some internal API for this, and qemu_irq looks like a good choice.

The PCI bus doesn't need any special support (I think) but something on the other end needs to interpret those writes.

In any case we need some internal API for this, and qemu_irq looks like a good choice.

Quoted 2 lines

The PCI bus doesn't need any special support (I think) but something on the other end needs to interpret those writes.

Sure. But there's definitely nothing PCI specific about it. I assumed this would all be contained within the APIC.

Quoted 2 lines

In any case we need some internal API for this, and qemu_irq looks like a good choice.

What do you expect to be using this API?

Paul

Quoted 2 lines

The PCI bus doesn't need any special support (I think) but something on the other end needs to interpret those writes.

Sure. But there's definitely nothing PCI specific about it. I assumed this would all be contained within the APIC.

Quoted 2 lines

In any case we need some internal API for this, and qemu_irq looks like a good choice.

What do you expect to be using this API?

Paul

Quoted 3 lines

The PCI bus doesn't need any special support (I think) but something on the other end needs to interpret those writes.

Sure. But there's definitely nothing PCI specific about it. I assumed this would all be contained within the APIC.

Quoted 3 lines

In any case we need some internal API for this, and qemu_irq looks like a good choice.

What do you expect to be using this API?

Quoted 3 lines

The PCI bus doesn't need any special support (I think) but something on the other end needs to interpret those writes.

Sure. But there's definitely nothing PCI specific about it. I assumed this would all be contained within the APIC.

Quoted 3 lines

In any case we need some internal API for this, and qemu_irq looks like a good choice.

What do you expect to be using this API?

Quoted 5 lines

Quoted 2 lines

The PCI bus doesn't need any special support (I think) but something on the other end needs to interpret those writes.

Sure. But there's definitely nothing PCI specific about it. I assumed this would all be contained within the APIC.

MSIs are defined by PCI and their configuration is done using the PCI configuration space.

Quoted 4 lines

Quoted 2 lines

In any case we need some internal API for this, and qemu_irq looks like a good choice.

What do you expect to be using this API?

virtio, emulated devices capable of supporting MSI (e1000?), device assignment (not yet in qemu.git).

Quoted 5 lines

Quoted 2 lines

The PCI bus doesn't need any special support (I think) but something on the other end needs to interpret those writes.

Sure. But there's definitely nothing PCI specific about it. I assumed this would all be contained within the APIC.

MSIs are defined by PCI and their configuration is done using the PCI configuration space.

Quoted 4 lines

Quoted 2 lines

In any case we need some internal API for this, and qemu_irq looks like a good choice.

What do you expect to be using this API?

virtio, emulated devices capable of supporting MSI (e1000?), device assignment (not yet in qemu.git).

Quoted 8 lines

Quoted 5 lines

Quoted 3 lines

In any case we need some internal API for this, and qemu_irq looks like a good choice.

What do you expect to be using this API?

virtio, emulated devices capable of supporting MSI (e1000?), device assignment (not yet in qemu.git).

It probably makes sense to have common infrastructure in pci.c to expose/implement device side MSI functionality. However I see no need for a direct API between the device and the APIC. We already have an API for memory accesses and MMIO regions. I'm pretty sure a system could implement MSI by pointing the device at system ram, and having the CPU periodically poll that.

Quoted 8 lines

Quoted 5 lines

Quoted 3 lines

In any case we need some internal API for this, and qemu_irq looks like a good choice.

What do you expect to be using this API?

virtio, emulated devices capable of supporting MSI (e1000?), device assignment (not yet in qemu.git).

It probably makes sense to have common infrastructure in pci.c to expose/implement device side MSI functionality. However I see no need for a direct API between the device and the APIC. We already have an API for memory accesses and MMIO regions. I'm pretty sure a system could implement MSI by pointing the device at system ram, and having the CPU periodically poll that.

Quoted 9 lines · Paul Brook

Quoted 8 lines

Quoted 5 lines

Quoted 3 lines

In any case we need some internal API for this, and qemu_irq looks like a good choice.

What do you expect to be using this API?

virtio, emulated devices capable of supporting MSI (e1000?), device assignment (not yet in qemu.git).

Quoted 7 lines

It probably makes sense to have common infrastructure in pci.c to expose/implement device side MSI functionality. However I see no need for a direct API between the device and the APIC. We already have an API for memory accesses and MMIO regions. I'm pretty sure a system could implement MSI by pointing the device at system ram, and having the CPU periodically poll that.

Instead of writing directly, let's abstract it behind a qemu_set_irq(). This is easier for device authors. The default implementation of the irq callback could write to apic memory, while for kvm we can directly trigger the interrupt via the kvm APIs.

Quoted 9 lines · Paul Brook

Quoted 8 lines

Quoted 5 lines

Quoted 3 lines

In any case we need some internal API for this, and qemu_irq looks like a good choice.

What do you expect to be using this API?

virtio, emulated devices capable of supporting MSI (e1000?), device assignment (not yet in qemu.git).

Quoted 7 lines

It probably makes sense to have common infrastructure in pci.c to expose/implement device side MSI functionality. However I see no need for a direct API between the device and the APIC. We already have an API for memory accesses and MMIO regions. I'm pretty sure a system could implement MSI by pointing the device at system ram, and having the CPU periodically poll that.

Instead of writing directly, let's abstract it behind a qemu_set_irq(). This is easier for device authors. The default implementation of the irq callback could write to apic memory, while for kvm we can directly trigger the interrupt via the kvm APIs.

Quoted 8 lines · Paul Brook

Quoted 7 lines

Quoted 4 lines

Quoted 2 lines

In any case we need some internal API for this, and qemu_irq looks like a good choice.

What do you expect to be using this API?

virtio, emulated devices capable of supporting MSI (e1000?), device assignment (not yet in qemu.git).

Quoted 6 lines

It probably makes sense to have common infrastructure in pci.c to expose/implement device side MSI functionality. However I see no need for a direct API between the device and the APIC. We already have an API for memory accesses and MMIO regions. I'm pretty sure a system could implement MSI by pointing the device at system ram, and having the CPU periodically poll that.

Instead of writing directly, let's abstract it behind a qemu_set_irq(). This is easier for device authors. The default implementation of the irq callback could write to apic memory, while for kvm we can directly trigger the interrupt via the kvm APIs.

Quoted 8 lines · Paul Brook

Quoted 7 lines

Quoted 4 lines

Quoted 2 lines

In any case we need some internal API for this, and qemu_irq looks like a good choice.

What do you expect to be using this API?

virtio, emulated devices capable of supporting MSI (e1000?), device assignment (not yet in qemu.git).

Quoted 6 lines

It probably makes sense to have common infrastructure in pci.c to expose/implement device side MSI functionality. However I see no need for a direct API between the device and the APIC. We already have an API for memory accesses and MMIO regions. I'm pretty sure a system could implement MSI by pointing the device at system ram, and having the CPU periodically poll that.

Instead of writing directly, let's abstract it behind a qemu_set_irq(). This is easier for device authors. The default implementation of the irq callback could write to apic memory, while for kvm we can directly trigger the interrupt via the kvm APIs.

Quoted 5 lines

Instead of writing directly, let's abstract it behind a qemu_set_irq(). This is easier for device authors. The default implementation of the irq callback could write to apic memory, while for kvm we can directly trigger the interrupt via the kvm APIs.

I'm still not convinced.

A tight coupling between PCI devices and the APIC is just going to cause us problems later one. I'm going to come back to the fact that these are memory writes so once we get IOMMU support they will presumably be subject to remapping by that, just like any other memory access.

Even ignoring that, qemu_irq isn't really the right interface. A MSI is a one- off event, not a level state. OTOH stl_phys is exactly the right interface.

The KVM interface should be contained within the APIC implementation.

Quoted 5 lines

Instead of writing directly, let's abstract it behind a qemu_set_irq(). This is easier for device authors. The default implementation of the irq callback could write to apic memory, while for kvm we can directly trigger the interrupt via the kvm APIs.

I'm still not convinced.

A tight coupling between PCI devices and the APIC is just going to cause us problems later one. I'm going to come back to the fact that these are memory writes so once we get IOMMU support they will presumably be subject to remapping by that, just like any other memory access.

Even ignoring that, qemu_irq isn't really the right interface. A MSI is a one- off event, not a level state. OTOH stl_phys is exactly the right interface.

The KVM interface should be contained within the APIC implementation.

Quoted 6 lines · Paul Brook

Quoted 5 lines

Instead of writing directly, let's abstract it behind a qemu_set_irq(). This is easier for device authors. The default implementation of the irq callback could write to apic memory, while for kvm we can directly trigger the interrupt via the kvm APIs.

Quoted 8 lines

I'm still not convinced.

A tight coupling between PCI devices and the APIC is just going to cause us problems later one. I'm going to come back to the fact that these are memory writes so once we get IOMMU support they will presumably be subject to remapping by that, just like any other memory access.

I'm not suggesting the qemu_irq will extend all the way to the apic. Think of it as connecting the device core with its interrupt unit.

Quoted 3 lines

Even ignoring that, qemu_irq isn't really the right interface. A MSI is a one- off event, not a level state. OTOH stl_phys is exactly the right interface.

The qemu_irq callback should do an stl_phys(). The device is happy since it's using the same API it uses for non-MSI. The APIC is happy since it isn't connected directly to the device. stl_phys() is happy since it sees more traffic and can serve more ads. kvm is happy since it can hijack the callback to throw the interrupt directly into the kernel.

Quoted 2 lines

The KVM interface should be contained within the APIC implementation.

Tricky, but doable.

Quoted 4 lines

A tight coupling between PCI devices and the APIC is just going to cause us problems later one. I'm going to come back to the fact that these are memory writes so once we get IOMMU support they will presumably be subject to remapping by that, just like any other memory access.

I'm not suggesting the qemu_irq will extend all the way to the apic. Think of it as connecting the device core with its interrupt unit.

Quoted 3 lines

Even ignoring that, qemu_irq isn't really the right interface. A MSI is a one- off event, not a level state. OTOH stl_phys is exactly the right interface.

The qemu_irq callback should do an stl_phys(). The device is happy since it's using the same API it uses for non-MSI.

Quoted 4 lines

A tight coupling between PCI devices and the APIC is just going to cause us problems later one. I'm going to come back to the fact that these are memory writes so once we get IOMMU support they will presumably be subject to remapping by that, just like any other memory access.

I'm not suggesting the qemu_irq will extend all the way to the apic. Think of it as connecting the device core with its interrupt unit.

Quoted 3 lines

Even ignoring that, qemu_irq isn't really the right interface. A MSI is a one- off event, not a level state. OTOH stl_phys is exactly the right interface.

The qemu_irq callback should do an stl_phys(). The device is happy since it's using the same API it uses for non-MSI.

Quoted 12 lines

Quoted 4 lines

A tight coupling between PCI devices and the APIC is just going to cause us problems later one. I'm going to come back to the fact that these are memory writes so once we get IOMMU support they will presumably be subject to remapping by that, just like any other memory access.

I'm not suggesting the qemu_irq will extend all the way to the apic. Think of it as connecting the device core with its interrupt unit.

Quoted 3 lines

Even ignoring that, qemu_irq isn't really the right interface. A MSI is a one- off event, not a level state. OTOH stl_phys is exactly the right interface.

Quoted 2 lines

The qemu_irq callback should do an stl_phys(). The device is happy since it's using the same API it uses for non-MSI.

MSI provides multiple edge triggered interrupts, whereas traditional mode provides a single level triggered interrupt. My guess is most devices will want to treat these differently anyway.

Either way, this is an implementation detail between pci.c and individual devices. It has nothing to do with the APIC.

Paul

Quoted 12 lines

Quoted 4 lines

A tight coupling between PCI devices and the APIC is just going to cause us problems later one. I'm going to come back to the fact that these are memory writes so once we get IOMMU support they will presumably be subject to remapping by that, just like any other memory access.

I'm not suggesting the qemu_irq will extend all the way to the apic. Think of it as connecting the device core with its interrupt unit.

Quoted 3 lines

Even ignoring that, qemu_irq isn't really the right interface. A MSI is a one- off event, not a level state. OTOH stl_phys is exactly the right interface.

Quoted 2 lines

The qemu_irq callback should do an stl_phys(). The device is happy since it's using the same API it uses for non-MSI.

MSI provides multiple edge triggered interrupts, whereas traditional mode provides a single level triggered interrupt. My guess is most devices will want to treat these differently anyway.

Either way, this is an implementation detail between pci.c and individual devices. It has nothing to do with the APIC.

Paul

Quoted 3 lines

MSI provides multiple edge triggered interrupts, whereas traditional mode provides a single level triggered interrupt. My guess is most devices will want to treat these differently anyway.

So, is qemu_send_msi better than qemu_set_irq.

Quoted 3 lines

MSI provides multiple edge triggered interrupts, whereas traditional mode provides a single level triggered interrupt. My guess is most devices will want to treat these differently anyway.

So, is qemu_send_msi better than qemu_set_irq.

Quoted 5 lines

Quoted 3 lines

MSI provides multiple edge triggered interrupts, whereas traditional mode provides a single level triggered interrupt. My guess is most devices will want to treat these differently anyway.

So, is qemu_send_msi better than qemu_set_irq.

Neither. pci_send_msi, which is a trivial wrapper around stl_phys.

Quoted 5 lines

Quoted 3 lines

MSI provides multiple edge triggered interrupts, whereas traditional mode provides a single level triggered interrupt. My guess is most devices will want to treat these differently anyway.

So, is qemu_send_msi better than qemu_set_irq.

Neither. pci_send_msi, which is a trivial wrapper around stl_phys.

Quoted 7 lines · Paul Brook

Quoted 5 lines

Quoted 3 lines

MSI provides multiple edge triggered interrupts, whereas traditional mode provides a single level triggered interrupt. My guess is most devices will want to treat these differently anyway.

So, is qemu_send_msi better than qemu_set_irq.

Neither. pci_send_msi, which is a trivial wrapper around stl_phys.

To clarify, you seem to be trying to fuse two largely separate features together.

MSI is a standard PCI device capability[1] that involves the device performing a 32-bit memory write when something interesting occurs. These writes may or may not be directed at a APIC.

The x86 APIC has a memory mapped interface that allows generation of CPU interrupts in response response to memory writes. These may or may not come from an MSI capable PCI device.

Paul

[1] Note a *device* capability, not a bus capability.

Quoted 7 lines · Paul Brook

Quoted 5 lines

Quoted 3 lines

MSI provides multiple edge triggered interrupts, whereas traditional mode provides a single level triggered interrupt. My guess is most devices will want to treat these differently anyway.

So, is qemu_send_msi better than qemu_set_irq.

Neither. pci_send_msi, which is a trivial wrapper around stl_phys.

To clarify, you seem to be trying to fuse two largely separate features together.

MSI is a standard PCI device capability[1] that involves the device performing a 32-bit memory write when something interesting occurs. These writes may or may not be directed at a APIC.

The x86 APIC has a memory mapped interface that allows generation of CPU interrupts in response response to memory writes. These may or may not come from an MSI capable PCI device.

Paul

[1] Note a *device* capability, not a bus capability.

Quoted 16 lines · Paul Brook

Quoted 7 lines · Paul Brook

Quoted 5 lines

Quoted 3 lines

MSI provides multiple edge triggered interrupts, whereas traditional mode provides a single level triggered interrupt. My guess is most devices will want to treat these differently anyway.

So, is qemu_send_msi better than qemu_set_irq.

Neither. pci_send_msi, which is a trivial wrapper around stl_phys.

To clarify, you seem to be trying to fuse two largely separate features together.

MSI is a standard PCI device capability[1] that involves the device performing a 32-bit memory write when something interesting occurs. These writes may or may not be directed at a APIC.

Quoted 7 lines

The x86 APIC has a memory mapped interface that allows generation of CPU interrupts in response response to memory writes. These may or may not come from an MSI capable PCI device.

Paul

[1] Note a *device* capability, not a bus capability.

Paul, so I went over specs, and what you say about APIC here does not seem to be what Intel actually implemented. Specifically, Intel implemented *MSI support in APIC*. This lets PCI devices, but not the CPU, signal interrupts by memory writes.

For example, after reset, when CPU writes to address 0xfee00000 this is an access to a reserved register in APIC, but when PCI device does write to 0xfee00000, this triggers an interrupt to destination 0.

See section 9.12 in Intel® 64 and IA-32 Architectures Software Developer’s Manual Volume 3A: System Programming Guide, Part 1
http://www.intel.com/Assets/PDF/manual/253668.pdf

So it seems that what we need to do in pci is:

if (!msi_ops || msi_ops->send_msi(address, data))

	stl_phy(address, data);

where send_msi is wired to apic_send_msi and where apic_send_msi returns an error for an address outside of the MSI range 0xfee00000 - 0xfeefffff

Makes sense?

Quoted 16 lines · Paul Brook

Quoted 7 lines · Paul Brook

Quoted 5 lines

Quoted 3 lines

MSI provides multiple edge triggered interrupts, whereas traditional mode provides a single level triggered interrupt. My guess is most devices will want to treat these differently anyway.

So, is qemu_send_msi better than qemu_set_irq.

Neither. pci_send_msi, which is a trivial wrapper around stl_phys.

To clarify, you seem to be trying to fuse two largely separate features together.

MSI is a standard PCI device capability[1] that involves the device performing a 32-bit memory write when something interesting occurs. These writes may or may not be directed at a APIC.

Quoted 7 lines

The x86 APIC has a memory mapped interface that allows generation of CPU interrupts in response response to memory writes. These may or may not come from an MSI capable PCI device.

Paul

[1] Note a *device* capability, not a bus capability.

Paul, so I went over specs, and what you say about APIC here does not seem to be what Intel actually implemented. Specifically, Intel implemented *MSI support in APIC*. This lets PCI devices, but not the CPU, signal interrupts by memory writes.

For example, after reset, when CPU writes to address 0xfee00000 this is an access to a reserved register in APIC, but when PCI device does write to 0xfee00000, this triggers an interrupt to destination 0.

See section 9.12 in Intel® 64 and IA-32 Architectures Software Developer’s Manual Volume 3A: System Programming Guide, Part 1
http://www.intel.com/Assets/PDF/manual/253668.pdf

So it seems that what we need to do in pci is:

if (!msi_ops || msi_ops->send_msi(address, data))

	stl_phy(address, data);

where send_msi is wired to apic_send_msi and where apic_send_msi returns an error for an address outside of the MSI range 0xfee00000 - 0xfeefffff

Makes sense?

Quoted 5 lines

Quoted 3 lines

MSI provides multiple edge triggered interrupts, whereas traditional mode provides a single level triggered interrupt. My guess is most devices will want to treat these differently anyway.

So, is qemu_send_msi better than qemu_set_irq.

Neither. pci_send_msi,

which is a trivial wrapper around stl_phys.

Quoted 5 lines

Quoted 3 lines

MSI provides multiple edge triggered interrupts, whereas traditional mode provides a single level triggered interrupt. My guess is most devices will want to treat these differently anyway.

So, is qemu_send_msi better than qemu_set_irq.

Neither. pci_send_msi,

which is a trivial wrapper around stl_phys.

Quoted 1 line

which is a trivial wrapper around stl_phys.

OK, but I'm adding another level of indirection in the middle, to allow us to tie in a kvm backend.

Quoted 5 lines

Quoted 2 lines

which is a trivial wrapper around stl_phys.

OK, but I'm adding another level of indirection in the middle, to allow us to tie in a kvm backend.

kvm has no business messing with the PCI device code.

Quoted 6 lines · Paul Brook

Quoted 4 lines

Quoted 1 line

which is a trivial wrapper around stl_phys.

OK, but I'm adding another level of indirection in the middle, to allow us to tie in a kvm backend.

kvm has no business messing with the PCI device code.

kvm has a fast path for irq injection. If qemu wants to support it we need some abstraction here.

Quoted 9 lines · Avi Kivity

Quoted 7 lines · Paul Brook

Quoted 5 lines

Quoted 2 lines

which is a trivial wrapper around stl_phys.

OK, but I'm adding another level of indirection in the middle, to allow us to tie in a kvm backend.

kvm has no business messing with the PCI device code.

Quoted 3 lines

kvm has a fast path for irq injection. If qemu wants to support it we need some abstraction here.

Fast path from where to where? Having the PCI layer bypass/re-implement the APIC and inject the interrupt directly into the cpu core sounds a particularly bad idea.

Quoted 7 lines · Avi Kivity

Quoted 5 lines · Paul Brook

Quoted 5 lines

Quoted 2 lines

which is a trivial wrapper around stl_phys.

OK, but I'm adding another level of indirection in the middle, to allow us to tie in a kvm backend.

Quoted 5 lines

Quoted 2 lines

kvm has no business messing with the PCI device code.

kvm has a fast path for irq injection. If qemu wants to support it we need some abstraction here.

Fast path from where to where? Having the PCI layer bypass/re-implement the APIC and inject the interrupt directly into the cpu core sounds a particularly bad idea.

Quoted 8 lines

Quoted 4 lines

Quoted 1 line

kvm has no business messing with the PCI device code.

kvm has a fast path for irq injection. If qemu wants to support it we need some abstraction here.

Fast path from where to where? Having the PCI layer bypass/re-implement the APIC and inject the interrupt directly into the cpu core sounds a particularly bad idea.

kvm implements the APIC in the host kernel (qemu upstream doesn't support this yet). The fast path is wired to the in-kernel APIC, not the cpu core directly.

The idea is to wire it to UIO for device assignment, to a virtio-device implemented in the kernel, and to qemu.

Quoted 8 lines

Quoted 4 lines

Quoted 1 line

kvm has no business messing with the PCI device code.

kvm has a fast path for irq injection. If qemu wants to support it we need some abstraction here.

Fast path from where to where? Having the PCI layer bypass/re-implement the APIC and inject the interrupt directly into the cpu core sounds a particularly bad idea.

kvm implements the APIC in the host kernel (qemu upstream doesn't support this yet). The fast path is wired to the in-kernel APIC, not the cpu core directly.

The idea is to wire it to UIO for device assignment, to a virtio-device implemented in the kernel, and to qemu.

Quoted 4 lines

kvm implements the APIC in the host kernel (qemu upstream doesn't support this yet). The fast path is wired to the in-kernel APIC, not the cpu core directly.

Quoted 3 lines

The idea is to wire it to UIO for device assignment, to a virtio-device implemented in the kernel, and to qemu.

I still don't see why you're trying to bypass straight from the pci layer to the apic. Why can't you just pass the apic MMIO writes to the kernel? You've presumably got to update the apic state anyway.

Quoted 7 lines

kvm implements the APIC in the host kernel (qemu upstream doesn't support this yet). The fast path is wired to the in-kernel APIC, not the cpu core directly.

The idea is to wire it to UIO for device assignment, to a virtio-device implemented in the kernel, and to qemu.

I still don't see why you're trying to bypass straight from the pci layer to the apic. Why can't you just pass the apic MMIO writes to the kernel? You've presumably got to update the apic state anyway.

Quoted 10 lines · Paul Brook

Quoted 6 lines

kvm implements the APIC in the host kernel (qemu upstream doesn't support this yet). The fast path is wired to the in-kernel APIC, not the cpu core directly.

The idea is to wire it to UIO for device assignment, to a virtio-device implemented in the kernel, and to qemu.

I still don't see why you're trying to bypass straight from the pci layer to the apic. Why can't you just pass the apic MMIO writes to the kernel? You've presumably got to update the apic state anyway.

The fast path is an eventfd so that we don't have to teach all the clients about the details of MSI. Userspace programs the MSI details into kvm and hands the client an eventfd. All the client has to do is bang on the eventfd for the interrupt to be queued. The eventfd provides event coalescing and is equally useful from the kernel and userspace, and can be used with targets other than kvm.

Quoted 10 lines · Paul Brook

Quoted 6 lines

kvm implements the APIC in the host kernel (qemu upstream doesn't support this yet). The fast path is wired to the in-kernel APIC, not the cpu core directly.

The idea is to wire it to UIO for device assignment, to a virtio-device implemented in the kernel, and to qemu.

I still don't see why you're trying to bypass straight from the pci layer to the apic. Why can't you just pass the apic MMIO writes to the kernel? You've presumably got to update the apic state anyway.

The fast path is an eventfd so that we don't have to teach all the clients about the details of MSI. Userspace programs the MSI details into kvm and hands the client an eventfd. All the client has to do is bang on the eventfd for the interrupt to be queued. The eventfd provides event coalescing and is equally useful from the kernel and userspace, and can be used with targets other than kvm.

Quoted 7 lines

The fast path is an eventfd so that we don't have to teach all the clients about the details of MSI. Userspace programs the MSI details into kvm and hands the client an eventfd. All the client has to do is bang on the eventfd for the interrupt to be queued. The eventfd provides event coalescing and is equally useful from the kernel and userspace, and can be used with targets other than kvm.

So presumably if a device triggers an APIC interrupt using a write that isn't one of the currently configured PCI devices, it all explodes horribly?

Quoted 7 lines

The fast path is an eventfd so that we don't have to teach all the clients about the details of MSI. Userspace programs the MSI details into kvm and hands the client an eventfd. All the client has to do is bang on the eventfd for the interrupt to be queued. The eventfd provides event coalescing and is equally useful from the kernel and userspace, and can be used with targets other than kvm.

So presumably if a device triggers an APIC interrupt using a write that isn't one of the currently configured PCI devices, it all explodes horribly?

Quoted 13 lines

Quoted 8 lines

Quoted 4 lines

Quoted 1 line

kvm has no business messing with the PCI device code.

kvm has a fast path for irq injection. If qemu wants to support it we need some abstraction here.

Fast path from where to where? Having the PCI layer bypass/re-implement the APIC and inject the interrupt directly into the cpu core sounds a particularly bad idea.

kvm implements the APIC in the host kernel (qemu upstream doesn't support this yet). The fast path is wired to the in-kernel APIC, not the cpu core directly.

Quoted 2 lines

The idea is to wire it to UIO for device assignment, to a virtio-device implemented in the kernel, and to qemu.

I still don't see why you're trying to bypass straight from the pci layer to the apic. Why can't you just pass the apic MMIO writes to the kernel? You've presumably got to update the apic state anyway.

Paul

Quoted 13 lines

Quoted 8 lines

Quoted 4 lines

Quoted 1 line

kvm has no business messing with the PCI device code.

kvm has a fast path for irq injection. If qemu wants to support it we need some abstraction here.

Fast path from where to where? Having the PCI layer bypass/re-implement the APIC and inject the interrupt directly into the cpu core sounds a particularly bad idea.

kvm implements the APIC in the host kernel (qemu upstream doesn't support this yet). The fast path is wired to the in-kernel APIC, not the cpu core directly.

Quoted 2 lines

The idea is to wire it to UIO for device assignment, to a virtio-device implemented in the kernel, and to qemu.

I still don't see why you're trying to bypass straight from the pci layer to the apic. Why can't you just pass the apic MMIO writes to the kernel? You've presumably got to update the apic state anyway.

Paul

Quoted 6 lines · Paul Brook

Quoted 4 lines

Quoted 1 line

which is a trivial wrapper around stl_phys.

OK, but I'm adding another level of indirection in the middle, to allow us to tie in a kvm backend.

kvm has no business messing with the PCI device code.

kvm has a fast path for irq injection. If qemu wants to support it we need some abstraction here.

Quoted 5 lines

Quoted 2 lines

which is a trivial wrapper around stl_phys.

OK, but I'm adding another level of indirection in the middle, to allow us to tie in a kvm backend.

kvm has no business messing with the PCI device code.

Quoted 4 lines

Quoted 1 line

which is a trivial wrapper around stl_phys.

OK, but I'm adding another level of indirection in the middle, to allow us to tie in a kvm backend.

kvm has no business messing with the PCI device code.

Quoted 4 lines

Quoted 1 line

which is a trivial wrapper around stl_phys.

OK, but I'm adding another level of indirection in the middle, to allow us to tie in a kvm backend.

kvm has no business messing with the PCI device code.

kvm needs data on MSI entries: that's the interface current kernel exposes for injecting these interrupts.

I think we also need to support in-kernel devices which would inject MSI interrupt directly from kernel. For these, kvm would need to know when mask bit changes and give us info on pending bit.

That's a fair amount of PCI specific code in kvm

kvm needs data on MSI entries: that's the interface current kernel exposes for injecting these interrupts.

I think we also need to support in-kernel devices which would inject MSI interrupt directly from kernel. For these, kvm would need to know when mask bit changes and give us info on pending bit.

That's a fair amount of PCI specific code in kvm

Quoted 1 line

which is a trivial wrapper around stl_phys.

OK, but I'm adding another level of indirection in the middle, to allow us to tie in a kvm backend.

Quoted 5 lines

Quoted 3 lines

MSI provides multiple edge triggered interrupts, whereas traditional mode provides a single level triggered interrupt. My guess is most devices will want to treat these differently anyway.

So, is qemu_send_msi better than qemu_set_irq.

Neither. pci_send_msi, which is a trivial wrapper around stl_phys.

Quoted 5 lines

Quoted 3 lines

MSI provides multiple edge triggered interrupts, whereas traditional mode provides a single level triggered interrupt. My guess is most devices will want to treat these differently anyway.

So, is qemu_send_msi better than qemu_set_irq.

Neither. pci_send_msi, which is a trivial wrapper around stl_phys.

Quoted 6 lines · Paul Brook

Quoted 5 lines

Instead of writing directly, let's abstract it behind a qemu_set_irq(). This is easier for device authors. The default implementation of the irq callback could write to apic memory, while for kvm we can directly trigger the interrupt via the kvm APIs.

Quoted 8 lines

I'm still not convinced.

A tight coupling between PCI devices and the APIC is just going to cause us problems later one. I'm going to come back to the fact that these are memory writes so once we get IOMMU support they will presumably be subject to remapping by that, just like any other memory access.

I'm not suggesting the qemu_irq will extend all the way to the apic. Think of it as connecting the device core with its interrupt unit.

Quoted 3 lines

Even ignoring that, qemu_irq isn't really the right interface. A MSI is a one- off event, not a level state. OTOH stl_phys is exactly the right interface.

The qemu_irq callback should do an stl_phys().

The device is happy
since it's using the same API it uses for non-MSI. The APIC is happy since it isn't connected directly to the device. stl_phys() is happy since it sees more traffic and can serve more ads. kvm is happy since it can hijack the callback to throw the interrupt directly into the kernel.

Quoted 2 lines

The KVM interface should be contained within the APIC implementation.

Tricky, but doable.

Signature · 1 line

error compiling committee.c: too many arguments to function

Quoted 6 lines · Paul Brook

Quoted 5 lines

Instead of writing directly, let's abstract it behind a qemu_set_irq(). This is easier for device authors. The default implementation of the irq callback could write to apic memory, while for kvm we can directly trigger the interrupt via the kvm APIs.

Quoted 8 lines

I'm still not convinced.

A tight coupling between PCI devices and the APIC is just going to cause us problems later one. I'm going to come back to the fact that these are memory writes so once we get IOMMU support they will presumably be subject to remapping by that, just like any other memory access.

I'm not suggesting the qemu_irq will extend all the way to the apic. Think of it as connecting the device core with its interrupt unit.

Quoted 3 lines

Even ignoring that, qemu_irq isn't really the right interface. A MSI is a one- off event, not a level state. OTOH stl_phys is exactly the right interface.

The qemu_irq callback should do an stl_phys(). The device is happy since it's using the same API it uses for non-MSI. The APIC is happy since it isn't connected directly to the device. stl_phys() is happy since it sees more traffic and can serve more ads. kvm is happy since it can hijack the callback to throw the interrupt directly into the kernel.

Quoted 2 lines

The KVM interface should be contained within the APIC implementation.

Tricky, but doable.

Quoted 6 lines · Paul Brook

Quoted 5 lines

Instead of writing directly, let's abstract it behind a qemu_set_irq(). This is easier for device authors. The default implementation of the irq callback could write to apic memory, while for kvm we can directly trigger the interrupt via the kvm APIs.

Quoted 8 lines

I'm still not convinced.

A tight coupling between PCI devices and the APIC is just going to cause us problems later one. I'm going to come back to the fact that these are memory writes so once we get IOMMU support they will presumably be subject to remapping by that, just like any other memory access.

I'm not suggesting the qemu_irq will extend all the way to the apic. Think of it as connecting the device core with its interrupt unit.

Quoted 3 lines

Even ignoring that, qemu_irq isn't really the right interface. A MSI is a one- off event, not a level state. OTOH stl_phys is exactly the right interface.

The qemu_irq callback should do an stl_phys().

The device is happy
since it's using the same API it uses for non-MSI. The APIC is happy since it isn't connected directly to the device. stl_phys() is happy since it sees more traffic and can serve more ads. kvm is happy since it can hijack the callback to throw the interrupt directly into the kernel.

Quoted 2 lines

The KVM interface should be contained within the APIC implementation.

Tricky, but doable.

Signature · 1 line

error compiling committee.c: too many arguments to function

Quoted 16 lines · Avi Kivity

Quoted 8 lines · Paul Brook

Quoted 7 lines

Quoted 4 lines

Quoted 2 lines

In any case we need some internal API for this, and qemu_irq looks like a good choice.

What do you expect to be using this API?

virtio, emulated devices capable of supporting MSI (e1000?), device assignment (not yet in qemu.git).

Quoted 6 lines

It probably makes sense to have common infrastructure in pci.c to expose/implement device side MSI functionality. However I see no need for a direct API between the device and the APIC. We already have an API for memory accesses and MMIO regions. I'm pretty sure a system could implement MSI by pointing the device at system ram, and having the CPU periodically poll that.

Quoted 4 lines

Instead of writing directly, let's abstract it behind a qemu_set_irq(). This is easier for device authors. The default implementation of the irq callback could write to apic memory, while for kvm we can directly trigger the interrupt via the kvm APIs.

I'm still not convinced.

A tight coupling between PCI devices and the APIC is just going to cause us problems later one. I'm going to come back to the fact that these are memory writes so once we get IOMMU support they will presumably be subject to remapping by that, just like any other memory access.

Even ignoring that, qemu_irq isn't really the right interface. A MSI is a one- off event, not a level state.

OTOH stl_phys is exactly the right interface.

The KVM interface should be contained within the APIC implementation.

Paul

Quoted 16 lines · Avi Kivity

Quoted 8 lines · Paul Brook

Quoted 7 lines

Quoted 4 lines

Quoted 2 lines

In any case we need some internal API for this, and qemu_irq looks like a good choice.

What do you expect to be using this API?

virtio, emulated devices capable of supporting MSI (e1000?), device assignment (not yet in qemu.git).

Quoted 6 lines

It probably makes sense to have common infrastructure in pci.c to expose/implement device side MSI functionality. However I see no need for a direct API between the device and the APIC. We already have an API for memory accesses and MMIO regions. I'm pretty sure a system could implement MSI by pointing the device at system ram, and having the CPU periodically poll that.

Quoted 4 lines

Instead of writing directly, let's abstract it behind a qemu_set_irq(). This is easier for device authors. The default implementation of the irq callback could write to apic memory, while for kvm we can directly trigger the interrupt via the kvm APIs.

I'm still not convinced.

A tight coupling between PCI devices and the APIC is just going to cause us problems later one. I'm going to come back to the fact that these are memory writes so once we get IOMMU support they will presumably be subject to remapping by that, just like any other memory access.

Even ignoring that, qemu_irq isn't really the right interface. A MSI is a one- off event, not a level state.

OTOH stl_phys is exactly the right interface.

The KVM interface should be contained within the APIC implementation.

Paul