Skip to contentSkip to conversation

plan for device assignment upstream

DISCUSSION Michael S. Tsirkin 4 participants 10 messages Jul 2, 2012 – Jul 5, 2012 KVM

lore
Share & tools
lore
#0 lore raw

I've been thinking hard about Jan's patches for device assignment. Basically while I thought it makes sense
to make all devices: assignment and not - behave the
same and use same APIs for injecting irqs, Anthony thinks there is huge value in making irq propagation hierarchical and device assignment should be special cased.

We seem to be at impasse for now and I think merging assignment ASAP has higher value than this specific issue. So I fold - let's do it as Anthony and Jan's original patches proposed.

Jan, can you please rebase and repost your original patchset (against master, not against pci) that added query for host irqs callbacks for device assignment? I'll re-review ignoring the idea of using the cache, with intent apply after I'll drop cache code from the pci branch in a couple of days (busy otherwise now).

I still intend to rework this later on, but that can wait.

Thanks,

Signature · 1 line
MST
#1 lore raw
Quoted 12 lines · Michael S. Tsirkin

I've been thinking hard about Jan's patches for device assignment. Basically while I thought it makes sense
to make all devices: assignment and not - behave the
same and use same APIs for injecting irqs, Anthony thinks there is huge value in making irq propagation hierarchical and device assignment should be special cased.

We seem to be at impasse for now and I think merging assignment ASAP has higher value than this specific issue. So I fold - let's do it as Anthony and Jan's original patches proposed.

Quoted 7 lines

Jan, can you please rebase and repost your original patchset (against master, not against pci) that added query for host irqs callbacks for device assignment? I'll re-review ignoring the idea of using the cache, with intent apply after I'll drop cache code from the pci branch in a couple of days (busy otherwise now).

I still intend to rework this later on, but that can wait.

Agree with both your ideas about the API and the decision to rework it in tree.

Signature · 1 line
error compiling committee.c: too many arguments to function
#2 lore raw
In reply to Michael S. Tsirkin
Quoted 6 lines · Michael S. Tsirkin

I've been thinking hard about Jan's patches for device assignment. Basically while I thought it makes sense
to make all devices: assignment and not - behave the
same and use same APIs for injecting irqs, Anthony thinks there is huge value in making irq propagation hierarchical and device assignment should be special cased.

On the long term, we will need direct injection, ie. caching, to allow making it lock-less. Stepping through all intermediate layers will cause troubles, at least performance-wise, when having to take and drop a lock at each stop.

Quoted 11 lines

We seem to be at impasse for now and I think merging assignment ASAP has higher value than this specific issue. So I fold - let's do it as Anthony and Jan's original patches proposed.

Jan, can you please rebase and repost your original patchset (against master, not against pci) that added query for host irqs callbacks for device assignment? I'll re-review ignoring the idea of using the cache, with intent apply after I'll drop cache code from the pci branch in a couple of days (busy otherwise now).

OK, will do ASAP.

Quoted 5 lines

I still intend to rework this later on, but that can wait.

Thanks,

Thanks as well,
Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux
#3 lore raw
Quoted 8 lines · Jan Kiszka
Quoted 6 lines · Michael S. Tsirkin

I've been thinking hard about Jan's patches for device assignment. Basically while I thought it makes sense
to make all devices: assignment and not - behave the
same and use same APIs for injecting irqs, Anthony thinks there is huge value in making irq propagation hierarchical and device assignment should be special cased.

Quoted 4 lines

On the long term, we will need direct injection, ie. caching, to allow making it lock-less. Stepping through all intermediate layers will cause troubles, at least performance-wise, when having to take and drop a lock at each stop.

So we precalculate everything beforehand. Instead of each qemu_irq triggering a callback, calculating the next hop and firing the next qemu_irq, configure each qemu_irq array with a function that describes how to take the next hop. Whenever the configuration changes, recalculate all routes.

For device assignment or vhost, we can have a qemu_irq_irqfd() which converts a qemu_irq to an eventfd. If the route calculations determine that it can be serviced via a real irqfd, they also configure it as an irqfd. Otherwise qemu configures a poll on this eventfd and calls the callback when needed.

Signature · 1 line
error compiling committee.c: too many arguments to function
#4 lore raw
Quoted 14 lines · Avi Kivity
Quoted 8 lines · Jan Kiszka
Quoted 6 lines · Michael S. Tsirkin

I've been thinking hard about Jan's patches for device assignment. Basically while I thought it makes sense
to make all devices: assignment and not - behave the
same and use same APIs for injecting irqs, Anthony thinks there is huge value in making irq propagation hierarchical and device assignment should be special cased.

Quoted 4 lines

On the long term, we will need direct injection, ie. caching, to allow making it lock-less. Stepping through all intermediate layers will cause troubles, at least performance-wise, when having to take and drop a lock at each stop.

Quoted 5 lines

So we precalculate everything beforehand. Instead of each qemu_irq triggering a callback, calculating the next hop and firing the next qemu_irq, configure each qemu_irq array with a function that describes how to take the next hop. Whenever the configuration changes, recalculate all routes.

Yes, we had this discussion last year when I proposed the IRQ matrix:
http://lists.nongnu.org/archive/html/qemu-devel/2011-09/msg00474.html

One problem with the matrix is that it only works for enable/disable level, not for more complex situations like boolean logic or multiplexed outputs.

Perhaps the devices should describe the currently valid logic with packet filter type mechanism? I think that could scale arbitrarily and it could be more friendly even as a kernel interface?

Quoted 13 lines

For device assignment or vhost, we can have a qemu_irq_irqfd() which converts a qemu_irq to an eventfd. If the route calculations determine that it can be serviced via a real irqfd, they also configure it as an irqfd. Otherwise qemu configures a poll on this eventfd and calls the callback when needed.

--
error compiling committee.c: too many arguments to function

#5 lore raw
Quoted 21 lines · Blue Swirl
Quoted 14 lines · Avi Kivity
Quoted 8 lines · Jan Kiszka
Quoted 6 lines · Michael S. Tsirkin

I've been thinking hard about Jan's patches for device assignment. Basically while I thought it makes sense
to make all devices: assignment and not - behave the
same and use same APIs for injecting irqs, Anthony thinks there is huge value in making irq propagation hierarchical and device assignment should be special cased.

Quoted 4 lines

On the long term, we will need direct injection, ie. caching, to allow making it lock-less. Stepping through all intermediate layers will cause troubles, at least performance-wise, when having to take and drop a lock at each stop.

Quoted 5 lines

So we precalculate everything beforehand. Instead of each qemu_irq triggering a callback, calculating the next hop and firing the next qemu_irq, configure each qemu_irq array with a function that describes how to take the next hop. Whenever the configuration changes, recalculate all routes.

Quoted 6 lines

Yes, we had this discussion last year when I proposed the IRQ matrix:
http://lists.nongnu.org/archive/html/qemu-devel/2011-09/msg00474.html

One problem with the matrix is that it only works for enable/disable level, not for more complex situations like boolean logic or multiplexed outputs.

I think we do need to support inverters etc.

Quoted 3 lines

Perhaps the devices should describe the currently valid logic with packet filter type mechanism? I think that could scale arbitrarily and it could be more friendly even as a kernel interface?

Interesting idea. So qemu creates multiple eventfds, gives half to devices and half to kvm (as irqfds), and configures bpf programs that calculate the irqfd outputs from the vfio inputs.

At least for x86 this is overkill. I would be okay with one-input-one-output cases handled with the current code and everything else routed through qemu.

Signature · 1 line
error compiling committee.c: too many arguments to function
#7 lore raw
In reply to Avi Kivity
Quoted 20 lines · Avi Kivity
Quoted 8 lines · Jan Kiszka
Quoted 6 lines · Michael S. Tsirkin

I've been thinking hard about Jan's patches for device assignment. Basically while I thought it makes sense
to make all devices: assignment and not - behave the
same and use same APIs for injecting irqs, Anthony thinks there is huge value in making irq propagation hierarchical and device assignment should be special cased.

Quoted 4 lines

On the long term, we will need direct injection, ie. caching, to allow making it lock-less. Stepping through all intermediate layers will cause troubles, at least performance-wise, when having to take and drop a lock at each stop.

So we precalculate everything beforehand. Instead of each qemu_irq triggering a callback, calculating the next hop and firing the next qemu_irq, configure each qemu_irq array with a function that describes how to take the next hop. Whenever the configuration changes, recalculate all routes.

Quoted 5 lines

For device assignment or vhost, we can have a qemu_irq_irqfd() which converts a qemu_irq to an eventfd. If the route calculations determine that it can be serviced via a real irqfd, they also configure it as an irqfd. Otherwise qemu configures a poll on this eventfd and calls the callback when needed.

This is more or less what I had in mind and what Anthony objects to.

Quoted 3 lines
Signature · 1 line
error compiling committee.c: too many arguments to function
#8 lore raw
Quoted 21 lines · Michael S. Tsirkin
Quoted 20 lines · Avi Kivity
Quoted 8 lines · Jan Kiszka
Quoted 6 lines · Michael S. Tsirkin

I've been thinking hard about Jan's patches for device assignment. Basically while I thought it makes sense
to make all devices: assignment and not - behave the
same and use same APIs for injecting irqs, Anthony thinks there is huge value in making irq propagation hierarchical and device assignment should be special cased.

Quoted 4 lines

On the long term, we will need direct injection, ie. caching, to allow making it lock-less. Stepping through all intermediate layers will cause troubles, at least performance-wise, when having to take and drop a lock at each stop.

So we precalculate everything beforehand. Instead of each qemu_irq triggering a callback, calculating the next hop and firing the next qemu_irq, configure each qemu_irq array with a function that describes how to take the next hop. Whenever the configuration changes, recalculate all routes.

Quoted 7 lines
Quoted 5 lines

For device assignment or vhost, we can have a qemu_irq_irqfd() which converts a qemu_irq to an eventfd. If the route calculations determine that it can be serviced via a real irqfd, they also configure it as an irqfd. Otherwise qemu configures a poll on this eventfd and calls the callback when needed.

This is more or less what I had in mind and what Anthony objects to.

Can you post an interface that supports this? Then we can see exactly what is objectionable.

Signature · 1 line
error compiling committee.c: too many arguments to function
#9 lore raw
Quoted 28 lines · Avi Kivity
Quoted 27 lines · Michael S. Tsirkin
Quoted 25 lines · Avi Kivity
Quoted 8 lines · Jan Kiszka
Quoted 6 lines · Michael S. Tsirkin

I've been thinking hard about Jan's patches for device assignment. Basically while I thought it makes sense
to make all devices: assignment and not - behave the
same and use same APIs for injecting irqs, Anthony thinks there is huge value in making irq propagation hierarchical and device assignment should be special cased.

Quoted 4 lines

On the long term, we will need direct injection, ie. caching, to allow making it lock-less. Stepping through all intermediate layers will cause troubles, at least performance-wise, when having to take and drop a lock at each stop.

So we precalculate everything beforehand. Instead of each qemu_irq triggering a callback, calculating the next hop and firing the next qemu_irq, configure each qemu_irq array with a function that describes how to take the next hop. Whenever the configuration changes, recalculate all routes.

For device assignment or vhost, we can have a qemu_irq_irqfd() which converts a qemu_irq to an eventfd. If the route calculations determine that it can be serviced via a real irqfd, they also configure it as an irqfd. Otherwise qemu configures a poll on this eventfd and calls the callback when needed.

Quoted 4 lines
Quoted 1 line

This is more or less what I had in mind and what Anthony objects to.

Can you post an interface that supports this? Then we can see exactly what is objectionable.

There wasn't a patch, just discussion. I'll try to sketch an API so people can respond more specifically.

Quoted 4 lines
Signature · 2 lines
-- 
error compiling committee.c: too many arguments to function
#6 lore raw
In reply to Avi Kivity
Quoted 31 lines · Avi Kivity
Quoted 21 lines · Blue Swirl
Quoted 14 lines · Avi Kivity
Quoted 8 lines · Jan Kiszka
Quoted 6 lines · Michael S. Tsirkin

I've been thinking hard about Jan's patches for device assignment. Basically while I thought it makes sense
to make all devices: assignment and not - behave the
same and use same APIs for injecting irqs, Anthony thinks there is huge value in making irq propagation hierarchical and device assignment should be special cased.

Quoted 4 lines

On the long term, we will need direct injection, ie. caching, to allow making it lock-less. Stepping through all intermediate layers will cause troubles, at least performance-wise, when having to take and drop a lock at each stop.

Quoted 5 lines

So we precalculate everything beforehand. Instead of each qemu_irq triggering a callback, calculating the next hop and firing the next qemu_irq, configure each qemu_irq array with a function that describes how to take the next hop. Whenever the configuration changes, recalculate all routes.

Quoted 6 lines

Yes, we had this discussion last year when I proposed the IRQ matrix:
http://lists.nongnu.org/archive/html/qemu-devel/2011-09/msg00474.html

One problem with the matrix is that it only works for enable/disable level, not for more complex situations like boolean logic or multiplexed outputs.

I think we do need to support inverters etc.

Quoted 7 lines
Quoted 3 lines

Perhaps the devices should describe the currently valid logic with packet filter type mechanism? I think that could scale arbitrarily and it could be more friendly even as a kernel interface?

Interesting idea. So qemu creates multiple eventfds, gives half to devices and half to kvm (as irqfds), and configures bpf programs that calculate the irqfd outputs from the vfio inputs.

I wasn't thinking of using fds, I guess that could work too but just that the interface could be similar to packet filters. So a device which implements an enable switch and ORs 8 inputs to a global output could be implemented with:
context = rule_init();
context = append_rule(context, R_OR, 8, &irq_array[]); context = append_rule(context, R_AND, 1, irq_enable); send_to_kernel_or_master_irq_controller(context);

Quoted 4 lines

At least for x86 this is overkill. I would be okay with one-input-one-output cases handled with the current code and everything else routed through qemu.

If this is efficient, some of the internal logic inside devices (for example PCI) could be implemented with the rules. Usually devices have one or just a few IRQ outputs but several possible internal sources for these.

Quoted 5 lines

--
error compiling committee.c: too many arguments to function