PCI cap for larger offsets/lengths
A message that is not in the archive (sent off-list or not yet received): 20180921095459.GA2842@work-vm
Quoted 3 lines · Dr. David Alan Gilbert
Hi,
We've got an experimental virtio device (using vhost-user) we're playing with
that would like to share multiple large mappings from the client back to qemu.
CCing Michael Tsirkin and Gerd Hoffman. Gerd could use this for virtio-gpu where some memory must be owned by the host.
Quoted 28 lines
'
virtio_pci_cap' only has 32bit offset and length fields and so I've got a different capability to express larger regions:/* Additional shared memory capability */
#defineVIRTIO_PCI_CAP_SHARED_MEMORY_CFG8struct
virtio_pci_shm_cap{
structvirtio_pci_capcap;
le32offset_hi; /* Most sig 32 bits of offset */
le32length_hi; /* Most sig 32 bits of length */
u8 id; /* To distinguish shm chunks */
};One oddity is that I'm allowing multiple instances of this capability on one device, distinguished by their 'id' field which I've made device type specific, e.g.:
#define
VIRTIO_MYDEV_PCI_SHMCAP_ID_CACHE0
#defineVIRTIO_MYDEV_PCI_SHMCAP_ID_JOURNAL1Having more than one capability of the same type is a little different from what the virtio-spec currently suggests; it currently talks about 'order of preference suggested by the device' - i.e. two capabilities where the user picks one; in this case I'm using all of the instances for different things.
Quoted 3 lines
If this makes sense to people, I'd be happy to write this up into a more formal change; if it doesn't I'd be interested to hear about alternative suggestions.
Another option is to define a unique PCI capability for each type:
struct virtio_pci_shm_cap {
struct virtio_pci_cap cap;
le32 offset_hi; /* Most significant 32 bits of offset */
le32 length_hi; /* Most significant 32 bits of length */
};
#define VIRTIO_PCI_CAP_MYDEV_SHM_CACHE 8
#define VIRTIO_PCI_CAP_MYDEV_SHM_JOURNAL 9This way the "order of preference" semantics can be kept. The drawback is consuming more PCI capability numbers.
Stefan
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 42 lines · * Gerd Hoffmann ([email protected]) wrote:
Quoted 7 lines · Stefan Hajnoczi
Quoted 3 lines · Dr. David Alan Gilbert
Hi,
We've got an experimental virtio device (using vhost-user) we're playing with
that would like to share multiple large mappings from the client back to qemu.CCing Michael Tsirkin and Gerd Hoffman. Gerd could use this for virtio-gpu where some memory must be owned by the host.
Yep. For virtio-gpu I want be able to map host gpu resources (which must be allocated by the host gpu driver) into the guest address space.
Quoted 14 lines
Quoted 14 lines
'
virtio_pci_cap' only has 32bit offset and length fields and so I've got a different capability to express larger regions:/* Additional shared memory capability */
#defineVIRTIO_PCI_CAP_SHARED_MEMORY_CFG8struct
virtio_pci_shm_cap{
structvirtio_pci_capcap;
le32offset_hi; /* Most sig 32 bits of offset */
le32length_hi; /* Most sig 32 bits of length */
u8 id; /* To distinguish shm chunks */
};Quoted 6 lines
Quoted 6 lines
One oddity is that I'm allowing multiple instances of this capability on one device, distinguished by their 'id' field which I've made device type specific, e.g.:
#define
VIRTIO_MYDEV_PCI_SHMCAP_ID_CACHE0
#defineVIRTIO_MYDEV_PCI_SHMCAP_ID_JOURNAL1For my experimental virtio-gpu code I use one pci bar to reserve address space. It is a separate pci bar. First, because it is a 64bit bar. Second, because it is declared as prefetchable (unlike the mmio bar which is not). I also simply use the whole bar, so no offset/length is needed.
gpu resources are sub-regions within that pci bar, and they are managed using device-specific commands.
Quoted 6 lines
So, I'm wondering whenever it makes sense to just do the same for your device. Just use one pci bar as shared memory umbrella, specify that one using the virtio vendor cap, then have sub-regions within that bar for the various regions you have. Manage them dynamically (using device-specific virtio commands) or just have a static configuration (in device-specific config space).
Ours are static subdivisions; so it felt easier to declare them; it's a shame to make that device specific.
Quoted 3 lines
That avoids the problem with multiple capabilities of the same kind, and it also avoids exhausting the cap IDs quicky if every device defines their own
VIRTIO_FOO_DEVICE_PCI_SHMCAP_ID_BAR_REGION.
Is having multiple capabilities of the same type actually a problem, or is it just historical in the defitinition of virtio?
Dave
Quoted 3 lines
cheers,
Gerd
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 23 lines · Gerd Hoffmann
Hi,
Quoted 9 lines
Quoted 6 lines
So, I'm wondering whenever it makes sense to just do the same for your device. Just use one pci bar as shared memory umbrella, specify that one using the virtio vendor cap, then have sub-regions within that bar for the various regions you have. Manage them dynamically (using device-specific virtio commands) or just have a static configuration (in device-specific config space).
Ours are static subdivisions; so it felt easier to declare them; it's a shame to make that device specific.
Shared memory handling is device specific anyway, so I fail to see why this is a problem. Or do you want place virtio queues there (which could be common ground for multiple device types) ?
Quoted 6 lines
Quoted 3 lines
That avoids the problem with multiple capabilities of the same kind, and it also avoids exhausting the cap IDs quicky if every device defines their own
VIRTIO_FOO_DEVICE_PCI_SHMCAP_ID_BAR_REGION.Is having multiple capabilities of the same type actually a problem, or is it just historical in the defitinition of virtio?
Quoted 8 lines
I think the reason is that you can in theory have the same region twice, once in an IO bar and once in an MMIO bar, and then the guest could prefer the IO bar if possible and use the MMIO bar otherwise (PCIe slot without IO address window for example). I think that was never actually done in practice, and for (prefetchable) memory bars it doesn't make sense at all. So that would unlikely be a problem in practice.
Running out of capability IDs could become a real problem though.
If we use a dedicated prefetchable MMIO BAR for shared memory resources, then that BAR is unavailable. For example, if the virtio-pci transport itself ever wants to use a prefetchable MMIO BAR too.
I would keep the indirection offered by PCI capabilities so that devices retain the ability to decide how exactly BARs are used without requiring spec or driver changes.
Stefan
Quoted 20 lines · Gerd Hoffmann
Hi,
Quoted 9 lines
Quoted 6 lines
So, I'm wondering whenever it makes sense to just do the same for your device. Just use one pci bar as shared memory umbrella, specify that one using the virtio vendor cap, then have sub-regions within that bar for the various regions you have. Manage them dynamically (using device-specific virtio commands) or just have a static configuration (in device-specific config space).
Ours are static subdivisions; so it felt easier to declare them; it's a shame to make that device specific.
Shared memory handling is device specific anyway, so I fail to see why this is a problem. Or do you want place virtio queues there (which could be common ground for multiple device types) ?
Quoted 4 lines
Quoted 3 lines
That avoids the problem with multiple capabilities of the same kind, and it also avoids exhausting the cap IDs quicky if every device defines their own
VIRTIO_FOO_DEVICE_PCI_SHMCAP_ID_BAR_REGION.
Quoted 8 lines
Quoted 2 lines
Is having multiple capabilities of the same type actually a problem, or is it just historical in the defitinition of virtio?
I think the reason is that you can in theory have the same region twice, once in an IO bar and once in an MMIO bar, and then the guest could prefer the IO bar if possible and use the MMIO bar otherwise (PCIe slot without IO address window for example). I think that was never actually done in practice,
There's an option to enable that for AMD CPUs where MMIO faults are slower than on intel CPUs.
Quoted 7 lines
and for (prefetchable) memory bars it doesn't make sense at all. So that would unlikely be a problem in practice.
Running out of capability IDs could become a real problem though.
cheers,
Gerd
We can always add more bits if we run out of these. there's no real limit on capability size.
Signature · 17 lines
MST This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
Quoted 36 lines · * Michael S. Tsirkin ([email protected]) wrote:
Quoted 20 lines · Gerd Hoffmann
Hi,
Quoted 9 lines
Quoted 6 lines
So, I'm wondering whenever it makes sense to just do the same for your device. Just use one pci bar as shared memory umbrella, specify that one using the virtio vendor cap, then have sub-regions within that bar for the various regions you have. Manage them dynamically (using device-specific virtio commands) or just have a static configuration (in device-specific config space).
Ours are static subdivisions; so it felt easier to declare them; it's a shame to make that device specific.
Shared memory handling is device specific anyway, so I fail to see why this is a problem. Or do you want place virtio queues there (which could be common ground for multiple device types) ?
Quoted 4 lines
Quoted 3 lines
That avoids the problem with multiple capabilities of the same kind, and it also avoids exhausting the cap IDs quicky if every device defines their own
VIRTIO_FOO_DEVICE_PCI_SHMCAP_ID_BAR_REGION.Quoted 8 lines
Quoted 2 lines
Is having multiple capabilities of the same type actually a problem, or is it just historical in the defitinition of virtio?
I think the reason is that you can in theory have the same region twice, once in an IO bar and once in an MMIO bar, and then the guest could prefer the IO bar if possible and use the MMIO bar otherwise (PCIe slot without IO address window for example). I think that was never actually done in practice,
There's an option to enable that for AMD CPUs where MMIO faults are slower than on intel CPUs.
Quoted 3 lines
and for (prefetchable) memory bars it doesn't make sense at all. So that would unlikely be a problem in practice.
Quoted 7 lines
Quoted 4 lines
Running out of capability IDs could become a real problem though.
cheers,
GerdWe can always add more bits if we run out of these. there's no real limit on capability size.
I thought it was defined as 8 bits by the capability-linked list structure in PCI?
Dave
Quoted 19 lines
Signature · 17 lines
MST This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 42 lines · Dr. David Alan Gilbert
Quoted 41 lines · * Michael S. Tsirkin ([email protected]) wrote:
Quoted 20 lines · Gerd Hoffmann
Hi,
Quoted 9 lines
Quoted 6 lines
So, I'm wondering whenever it makes sense to just do the same for your device. Just use one pci bar as shared memory umbrella, specify that one using the virtio vendor cap, then have sub-regions within that bar for the various regions you have. Manage them dynamically (using device-specific virtio commands) or just have a static configuration (in device-specific config space).
Ours are static subdivisions; so it felt easier to declare them; it's a shame to make that device specific.
Shared memory handling is device specific anyway, so I fail to see why this is a problem. Or do you want place virtio queues there (which could be common ground for multiple device types) ?
Quoted 4 lines
Quoted 3 lines
That avoids the problem with multiple capabilities of the same kind, and it also avoids exhausting the cap IDs quicky if every device defines their own
VIRTIO_FOO_DEVICE_PCI_SHMCAP_ID_BAR_REGION.Quoted 8 lines
Quoted 2 lines
Is having multiple capabilities of the same type actually a problem, or is it just historical in the defitinition of virtio?
I think the reason is that you can in theory have the same region twice, once in an IO bar and once in an MMIO bar, and then the guest could prefer the IO bar if possible and use the MMIO bar otherwise (PCIe slot without IO address window for example). I think that was never actually done in practice,
There's an option to enable that for AMD CPUs where MMIO faults are slower than on intel CPUs.
Quoted 7 lines
and for (prefetchable) memory bars it doesn't make sense at all. So that would unlikely be a problem in practice.
Running out of capability IDs could become a real problem though.
cheers,
Gerd
Quoted 7 lines
Quoted 2 lines
We can always add more bits if we run out of these. there's no real limit on capability size.
I thought it was defined as 8 bits by the capability-linked list structure in PCI?
Dave
Maybe I misunderstand. Don't you mean u8 cfg_type field type in struct virtio_pci_cap? If so then what I was pointing out is that if we ever need more than 256 types then we
can always have a special cfg_type value(s) meaning "different format" and add more types this way.
Quoted 19 lines
Quoted 19 lines
Signature · 17 lines
MST This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
Quoted 2 lines
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/