Skip to contentSkip to conversation

PCI cap for larger offsets/lengths

virtio-comment Stefan Hajnoczi 3 participants 6 messages Nov 26, 2018 – Nov 28, 2018 3 messages not in the archive

lore
Share & tools
lore
#1 lore raw
In reply to a message not in the archive
Quoted 3 lines · Dr. David Alan Gilbert

Hi,
We've got an experimental virtio device (using vhost-user) we're playing with
that would like to share multiple large mappings from the client back to qemu.

CCing Michael Tsirkin and Gerd Hoffman. Gerd could use this for virtio-gpu where some memory must be owned by the host.

Quoted 28 lines

'virtio_pci_cap' only has 32bit offset and length fields and so I've got a different capability to express larger regions:

/* Additional shared memory capability */
#define VIRTIO_PCI_CAP_SHARED_MEMORY_CFG 8

struct virtio_pci_shm_cap {
struct virtio_pci_cap cap;
le32 offset_hi; /* Most sig 32 bits of offset */
le32 length_hi; /* Most sig 32 bits of length */
u8 id; /* To distinguish shm chunks */
};

One oddity is that I'm allowing multiple instances of this capability on one device, distinguished by their 'id' field which I've made device type specific, e.g.:

#define VIRTIO_MYDEV_PCI_SHMCAP_ID_CACHE 0
#define VIRTIO_MYDEV_PCI_SHMCAP_ID_JOURNAL 1

Having more than one capability of the same type is a little different from what the virtio-spec currently suggests; it currently talks about 'order of preference suggested by the device' - i.e. two capabilities where the user picks one; in this case I'm using all of the instances for different things.

Quoted 3 lines

If this makes sense to people, I'd be happy to write this up into a more formal change; if it doesn't I'd be interested to hear about alternative suggestions.

Another option is to define a unique PCI capability for each type:

  struct virtio_pci_shm_cap {
         struct virtio_pci_cap cap;
         le32 offset_hi;      /* Most significant 32 bits of offset */
         le32 length_hi;      /* Most significant 32 bits of length */
  };

  #define VIRTIO_PCI_CAP_MYDEV_SHM_CACHE 8
  #define VIRTIO_PCI_CAP_MYDEV_SHM_JOURNAL 9

This way the "order of preference" semantics can be kept. The drawback is consuming more PCI capability numbers.

Stefan

#3 lore raw
In reply to a message not in the archive
Quoted 42 lines · * Gerd Hoffmann ([email protected]) wrote:
Quoted 7 lines · Stefan Hajnoczi
Quoted 3 lines · Dr. David Alan Gilbert

Hi,
We've got an experimental virtio device (using vhost-user) we're playing with
that would like to share multiple large mappings from the client back to qemu.

CCing Michael Tsirkin and Gerd Hoffman. Gerd could use this for virtio-gpu where some memory must be owned by the host.

Yep. For virtio-gpu I want be able to map host gpu resources (which must be allocated by the host gpu driver) into the guest address space.

Quoted 14 lines
Quoted 14 lines

'virtio_pci_cap' only has 32bit offset and length fields and so I've got a different capability to express larger regions:

/* Additional shared memory capability */
#define VIRTIO_PCI_CAP_SHARED_MEMORY_CFG 8

struct virtio_pci_shm_cap {
struct virtio_pci_cap cap;
le32 offset_hi; /* Most sig 32 bits of offset */
le32 length_hi; /* Most sig 32 bits of length */
u8 id; /* To distinguish shm chunks */
};

Quoted 6 lines
Quoted 6 lines

One oddity is that I'm allowing multiple instances of this capability on one device, distinguished by their 'id' field which I've made device type specific, e.g.:

#define VIRTIO_MYDEV_PCI_SHMCAP_ID_CACHE 0
#define VIRTIO_MYDEV_PCI_SHMCAP_ID_JOURNAL 1

For my experimental virtio-gpu code I use one pci bar to reserve address space. It is a separate pci bar. First, because it is a 64bit bar. Second, because it is declared as prefetchable (unlike the mmio bar which is not). I also simply use the whole bar, so no offset/length is needed.

gpu resources are sub-regions within that pci bar, and they are managed using device-specific commands.

Quoted 6 lines

So, I'm wondering whenever it makes sense to just do the same for your device. Just use one pci bar as shared memory umbrella, specify that one using the virtio vendor cap, then have sub-regions within that bar for the various regions you have. Manage them dynamically (using device-specific virtio commands) or just have a static configuration (in device-specific config space).

Ours are static subdivisions; so it felt easier to declare them; it's a shame to make that device specific.

Quoted 3 lines

That avoids the problem with multiple capabilities of the same kind, and it also avoids exhausting the cap IDs quicky if every device defines their own VIRTIO_FOO_DEVICE_PCI_SHMCAP_ID_BAR_REGION.

Is having multiple capabilities of the same type actually a problem, or is it just historical in the defitinition of virtio?

Dave

Quoted 3 lines

cheers,
Gerd

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#5 lore raw
In reply to a message not in the archive
Quoted 23 lines · Gerd Hoffmann

Hi,

Quoted 9 lines
Quoted 6 lines

So, I'm wondering whenever it makes sense to just do the same for your device. Just use one pci bar as shared memory umbrella, specify that one using the virtio vendor cap, then have sub-regions within that bar for the various regions you have. Manage them dynamically (using device-specific virtio commands) or just have a static configuration (in device-specific config space).

Ours are static subdivisions; so it felt easier to declare them; it's a shame to make that device specific.

Shared memory handling is device specific anyway, so I fail to see why this is a problem. Or do you want place virtio queues there (which could be common ground for multiple device types) ?

Quoted 6 lines
Quoted 3 lines

That avoids the problem with multiple capabilities of the same kind, and it also avoids exhausting the cap IDs quicky if every device defines their own VIRTIO_FOO_DEVICE_PCI_SHMCAP_ID_BAR_REGION.

Is having multiple capabilities of the same type actually a problem, or is it just historical in the defitinition of virtio?

Quoted 8 lines

I think the reason is that you can in theory have the same region twice, once in an IO bar and once in an MMIO bar, and then the guest could prefer the IO bar if possible and use the MMIO bar otherwise (PCIe slot without IO address window for example). I think that was never actually done in practice, and for (prefetchable) memory bars it doesn't make sense at all. So that would unlikely be a problem in practice.

Running out of capability IDs could become a real problem though.

If we use a dedicated prefetchable MMIO BAR for shared memory resources, then that BAR is unavailable. For example, if the virtio-pci transport itself ever wants to use a prefetchable MMIO BAR too.

I would keep the indirection offered by PCI capabilities so that devices retain the ability to decide how exactly BARs are used without requiring spec or driver changes.

Stefan

#6 lore raw
In reply to a message not in the archive
Quoted 20 lines · Gerd Hoffmann

Hi,

Quoted 9 lines
Quoted 6 lines

So, I'm wondering whenever it makes sense to just do the same for your device. Just use one pci bar as shared memory umbrella, specify that one using the virtio vendor cap, then have sub-regions within that bar for the various regions you have. Manage them dynamically (using device-specific virtio commands) or just have a static configuration (in device-specific config space).

Ours are static subdivisions; so it felt easier to declare them; it's a shame to make that device specific.

Shared memory handling is device specific anyway, so I fail to see why this is a problem. Or do you want place virtio queues there (which could be common ground for multiple device types) ?

Quoted 4 lines
Quoted 3 lines

That avoids the problem with multiple capabilities of the same kind, and it also avoids exhausting the cap IDs quicky if every device defines their own VIRTIO_FOO_DEVICE_PCI_SHMCAP_ID_BAR_REGION.

Quoted 8 lines
Quoted 2 lines

Is having multiple capabilities of the same type actually a problem, or is it just historical in the defitinition of virtio?

I think the reason is that you can in theory have the same region twice, once in an IO bar and once in an MMIO bar, and then the guest could prefer the IO bar if possible and use the MMIO bar otherwise (PCIe slot without IO address window for example). I think that was never actually done in practice,

There's an option to enable that for AMD CPUs where MMIO faults are slower than on intel CPUs.

Quoted 7 lines

and for (prefetchable) memory bars it doesn't make sense at all. So that would unlikely be a problem in practice.

Running out of capability IDs could become a real problem though.

cheers,
Gerd

We can always add more bits if we run out of these. there's no real limit on capability size.

Signature · 17 lines
MST

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#7 lore raw
Quoted 36 lines · * Michael S. Tsirkin ([email protected]) wrote:
Quoted 20 lines · Gerd Hoffmann

Hi,

Quoted 9 lines
Quoted 6 lines

So, I'm wondering whenever it makes sense to just do the same for your device. Just use one pci bar as shared memory umbrella, specify that one using the virtio vendor cap, then have sub-regions within that bar for the various regions you have. Manage them dynamically (using device-specific virtio commands) or just have a static configuration (in device-specific config space).

Ours are static subdivisions; so it felt easier to declare them; it's a shame to make that device specific.

Shared memory handling is device specific anyway, so I fail to see why this is a problem. Or do you want place virtio queues there (which could be common ground for multiple device types) ?

Quoted 4 lines
Quoted 3 lines

That avoids the problem with multiple capabilities of the same kind, and it also avoids exhausting the cap IDs quicky if every device defines their own VIRTIO_FOO_DEVICE_PCI_SHMCAP_ID_BAR_REGION.

Quoted 8 lines
Quoted 2 lines

Is having multiple capabilities of the same type actually a problem, or is it just historical in the defitinition of virtio?

I think the reason is that you can in theory have the same region twice, once in an IO bar and once in an MMIO bar, and then the guest could prefer the IO bar if possible and use the MMIO bar otherwise (PCIe slot without IO address window for example). I think that was never actually done in practice,

There's an option to enable that for AMD CPUs where MMIO faults are slower than on intel CPUs.

Quoted 3 lines

and for (prefetchable) memory bars it doesn't make sense at all. So that would unlikely be a problem in practice.

Quoted 7 lines
Quoted 4 lines

Running out of capability IDs could become a real problem though.

cheers,
Gerd

We can always add more bits if we run out of these. there's no real limit on capability size.

I thought it was defined as 8 bits by the capability-linked list structure in PCI?

Dave

Quoted 19 lines
Signature · 17 lines
MST

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#8 lore raw
Quoted 42 lines · Dr. David Alan Gilbert
Quoted 41 lines · * Michael S. Tsirkin ([email protected]) wrote:
Quoted 20 lines · Gerd Hoffmann

Hi,

Quoted 9 lines
Quoted 6 lines

So, I'm wondering whenever it makes sense to just do the same for your device. Just use one pci bar as shared memory umbrella, specify that one using the virtio vendor cap, then have sub-regions within that bar for the various regions you have. Manage them dynamically (using device-specific virtio commands) or just have a static configuration (in device-specific config space).

Ours are static subdivisions; so it felt easier to declare them; it's a shame to make that device specific.

Shared memory handling is device specific anyway, so I fail to see why this is a problem. Or do you want place virtio queues there (which could be common ground for multiple device types) ?

Quoted 4 lines
Quoted 3 lines

That avoids the problem with multiple capabilities of the same kind, and it also avoids exhausting the cap IDs quicky if every device defines their own VIRTIO_FOO_DEVICE_PCI_SHMCAP_ID_BAR_REGION.

Quoted 8 lines
Quoted 2 lines

Is having multiple capabilities of the same type actually a problem, or is it just historical in the defitinition of virtio?

I think the reason is that you can in theory have the same region twice, once in an IO bar and once in an MMIO bar, and then the guest could prefer the IO bar if possible and use the MMIO bar otherwise (PCIe slot without IO address window for example). I think that was never actually done in practice,

There's an option to enable that for AMD CPUs where MMIO faults are slower than on intel CPUs.

Quoted 7 lines

and for (prefetchable) memory bars it doesn't make sense at all. So that would unlikely be a problem in practice.

Running out of capability IDs could become a real problem though.

cheers,
Gerd

Quoted 7 lines
Quoted 2 lines

We can always add more bits if we run out of these. there's no real limit on capability size.

I thought it was defined as 8 bits by the capability-linked list structure in PCI?

Dave

Maybe I misunderstand. Don't you mean u8 cfg_type field type in struct virtio_pci_cap? If so then what I was pointing out is that if we ever need more than 256 types then we
can always have a special cfg_type value(s) meaning "different format" and add more types this way.

Quoted 19 lines
Quoted 19 lines
Signature · 17 lines
MST

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 2 lines

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/