Skip to contentSkip to conversation

IVSHMEM version 2 device for QEMU

RFC PATCH Superseded Jan Kiszka 6 participants 23 messages Nov 11, 2019 – Dec 6, 2019 1 message not in the archive Kbuild

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Jan Kiszka

author
#0 lore raw

cover IVSHMEM version 2 device for QEMU

To get the ball rolling after my presentation of the topic at KVM Forum
[1] and many fruitful discussions around it, this is a first concrete code series. As discussed, I'm starting with the IVSHMEM implementation of a QEMU device and server. It's RFC because, besides specification and implementation details, there will still be some decisions needed about how to integrate the new version best into the existing code bases.

If you want to play with this, the basic setup of the shared memory device is described in patch 1 and 3. UIO driver and also the virtio-ivshmem prototype can be found at

http://git.kiszka.org/?p=linux.git;a=shortlog;h=refs/heads/queues/ivshmem2

Accessing the device via UIO is trivial enough. If you want to use it for virtio, this is additionally to the description in patch 3 needed on the virtio console backend side:

    modprobe uio_ivshmem
    echo "1af4 1110 1af4 1100 ffc003 ffffff" > /sys/bus/pci/drivers/uio_ivshmem/new_id
    linux/tools/virtio/virtio-ivshmem-console /dev/uio0

And for virtio block:

echo "1af4 1110 1af4 1100 ffc002 ffffff" > /sys/bus/pci/drivers/uio_ivshmem/new_id linux/tools/virtio/virtio-ivshmem-console /dev/uio0 /path/to/disk.img

After that, you can start the QEMU frontend instance with the virtio-ivshmem driver installed which can use the new /dev/hvc* or /dev/vda* as usual.

Any feedback welcome!

Jan

PS: Let me know if I missed someone potentially interested in this topic on CC - or if you would like to be dropped from the list.

PPS: The Jailhouse queues are currently out of sync /wrt minor details of this one, primarily the device ID. Will update them when the general direction is clear.

[1] https://kvmforum2019.sched.com/event/TmxI

Jan Kiszka (3):
hw/misc: Add implementation of ivshmem revision 2 device docs/specs: Add specification of ivshmem device revision 2
contrib: Add server for ivshmem revision 2

 Makefile                                  |    3 +
 Makefile.objs                             |    1 +
 configure                                 |    1 +
 contrib/ivshmem2-server/Makefile.objs     |    1 +
 contrib/ivshmem2-server/ivshmem2-server.c |  462 ++++++++++++
 contrib/ivshmem2-server/ivshmem2-server.h |  158 +++++
 contrib/ivshmem2-server/main.c            |  313 +++++++++
 docs/specs/ivshmem-2-device-spec.md       |  333 +++++++++
 hw/misc/Makefile.objs                     |    2 +-
 hw/misc/ivshmem2.c                        | 1091 +++++++++++++++++++++++++++++
 include/hw/misc/ivshmem2.h                |   48 ++
 11 files changed, 2412 insertions(+), 1 deletion(-)
 create mode 100644 contrib/ivshmem2-server/Makefile.objs
 create mode 100644 contrib/ivshmem2-server/ivshmem2-server.c
 create mode 100644 contrib/ivshmem2-server/ivshmem2-server.h
 create mode 100644 contrib/ivshmem2-server/main.c
 create mode 100644 docs/specs/ivshmem-2-device-spec.md
 create mode 100644 hw/misc/ivshmem2.c
 create mode 100644 include/hw/misc/ivshmem2.h
Signature · 1 line
2.16.4
author
#1 lore raw

RFC PATCH 1/3 hw/misc: Add implementation of ivshmem revision 2 device

From: Jan Kiszka <[email protected]>

This adds a reimplementation of ivshmem in its new revision 2 as separate device. The goal of this is not to enable sharing with v1, rather to allow explore the properties and potential limitation of the new version prior to discussing its integration with the existing code.

v2 always requires a server to interconnect two more more QEMU instances because it provides signaling between peers unconditionally. Therefore, only the interconnecting chardev, master mode, and the usage of ioeventfd can be configured at device level. All other parameters are defined by the server instance.

A new server protocol is introduced along this. Its primary difference is the introduction of a single welcome message that contains all peer parameters, rather than a series of single-word messages pushing them piece by piece.

A complicating difference in interrupt handling, compare to v1, is the auto-disable mode of v2: When this is active, interrupt delivery is disabled by the device after each interrupt event. This prevents the usage of irqfd on the receiving side, but it lowers the handling cost for guests that implemented interrupt throttling this way (specifically when exposing the device via UIO).

No changes have been made to the ivshmem device regarding migration:
Only the master can live-migrate, slave peers have to hot-unplug the device first.

The details of the device model will be specified in a succeeding commit. Drivers for this device can currently be found under

http://git.kiszka.org/?p=linux.git;a=shortlog;h=refs/heads/queues/ivshmem2

To instantiate a ivshmem v2 device, just add

... -chardev socket,path=/tmp/ivshmem_socket,id=ivshmem \
-device ivshmem,chardev=ivshmem

provided the server opened its socket under the default path.

---
 hw/misc/Makefile.objs      |    2 +-
 hw/misc/ivshmem2.c         | 1091 ++++++++++++++++++++++++++++++++++++++++++++
 include/hw/misc/ivshmem2.h |   48 ++
 3 files changed, 1140 insertions(+), 1 deletion(-)
 create mode 100644 hw/misc/ivshmem2.c
 create mode 100644 include/hw/misc/ivshmem2.h
hw/misc/Makefile.objs +1 -1
index ba898a5781..90a4a6608c 100644
@@ -26,7 +26,7 @@ common-obj-$(CONFIG_PUV3) += puv3_pm.o
 
 common-obj-$(CONFIG_MACIO) += macio/
 
-common-obj-$(CONFIG_IVSHMEM_DEVICE) += ivshmem.o
+common-obj-$(CONFIG_IVSHMEM_DEVICE) += ivshmem.o ivshmem2.o
 
 common-obj-$(CONFIG_REALVIEW) += arm_sysctl.o
 common-obj-$(CONFIG_NSERIES) += cbus.o
hw/misc/ivshmem2.c new file +1091 -0

1,091 lines not on this page · raw message

include/hw/misc/ivshmem2.h new file +48 -0

48 lines not on this page · raw message

Signature · 1 line
2.16.4
author
#2 lore raw

RFC PATCH 2/3 docs/specs: Add specification of ivshmem device revision 2

In reply to Jan Kiszka

From: Jan Kiszka <[email protected]>

This imports the ivshmem v2 specification draft from Jailhouse. Its final home is to be decided, this shall just simplify the review process at this stage.

Note that specifically the Features register (offset 08h) is still under consideration. In particular, its bit 0 seems useless now as its benefit to guests, specifically when they want to be portable, is close to zero. Maybe the register should still be kept, with all bits RsvdZ, for easier extensibility.

The rest appears now rather mature and reasonably implementable, as proven also by a version for Jailhouse.

---
 docs/specs/ivshmem-2-device-spec.md | 333 ++++++++++++++++++++++++++++++++++++
 1 file changed, 333 insertions(+)
 create mode 100644 docs/specs/ivshmem-2-device-spec.md
docs/specs/ivshmem-2-device-spec.md new file +333 -0
new file mode 100644
index 0000000000..98cfde585a
@@ -0,0 +1,333 @@
+IVSHMEM Device Specification
+============================
+
+** NOTE: THIS IS WORK-IN-PROGRESS, NOT YET A STABLE INTERFACE SPECIFICATION! **
+
+The Inter-VM Shared Memory device provides the following features to its users:
+
+- Interconnection between up to 65536 peers
+
+- Multi-purpose shared memory region
+
+    - common read/writable section
+
+    - unidirectional sections that are read/writable for one peer and only
+      readable for the others
+
+    - section with peer states
+
+- Event signaling via interrupt to remote sides
+
+- Support for life-cycle management via state value exchange and interrupt
+  notification on changes, backed by a shared memory section
+
+- Free choice of protocol to be used on top
+
+- Protocol type declaration
+
+- Unprivileged access to memory-mapped or I/O registers feasible
+
+- Discoverable and configurable via standard PCI mechanisms
+
+
+Hypervisor Model
+----------------
+
+In order to provide a consistent link between peers, all connected instances of
+IVSHMEM devices need to be configured, created and run by the hypervisor
+according to the following requirements:
+
+- The instances of the device need to be accessible via PCI programming
+  interfaces on all sides.
+
+- The read/write shared memory section has to be of the same size for all
+  peers and, if non-zero, has to reflect the same memory content for them.
+
+- If output sections are present (non-zero section size), there must be one
+  reserved for each peer with exclusive write access. All output sections
+  must have the same size and must be readable for all peers. They have to
+  reflect the same memory content for all peers.
+
+- The State Table must have the same size for all peers, must be large enough to
+  hold a state values of all peers, and must be read-only for the user.
+
+- State register changes (explicit writes, peer resets) have to be propagated
+  to the other peers by updating the corresponding State Table entry and issuing
+  an interrupt to all other peers if they enabled reception.
+
+- Interrupts events triggered by a peer have to be delivered to the target peer,
+  provided the receiving side is valid and has enabled the reception.
+
+- All peers must have the same interrupt delivery features available, i.e. MSI-X
+  with the same maximum number of vectors on platforms supporting this
+  mechanism, otherwise INTx with one vector.
+
+
+Guest-side Programming Model
+----------------------------
+
+An IVSHMEM device appears as a PCI device to its users. Unless otherwise noted,
+it conforms to the PCI Local Bus Specification, Revision 3.0 As such, it is
+discoverable via the PCI configuration space and provides a number of standard
+and custom PCI configuration registers.
+
+### Shared Memory Region Layout
+
+The shared memory region is divided into several sections.
+
+    +-----------------------------+   -
+    |                             |   :
+    | Output Section for peer n-1 |   : Output Section Size
+    |     (n = Maximum Peers)     |   :
+    +-----------------------------+   -
+    :                             :
+    :                             :
+    :                             :
+    +-----------------------------+   -
+    |                             |   :
+    |  Output Section for peer 1  |   : Output Section Size
+    |                             |   :
+    +-----------------------------+   -
+    |                             |   :
+    |  Output Section for peer 0  |   : Output Section Size
+    |                             |   :
+    +-----------------------------+   -
+    |                             |   :
+    |     Read/Write Section      |   : R/W Section Size
+    |                             |   :
+    +-----------------------------+   -
+    |                             |   :
+    |         State Table         |   : State Table Size
+    |                             |   :
+    +-----------------------------+   <-- Shared memory base address
+
+The first section consists of the mandatory State Table. Its size is defined by
+the State Table Size register and cannot be zero. This section is read-only for
+all peers.
+
+The second section consists of shared memory that is read/writable for all
+peers. Its size is defined by the R/W Section Size register. A size of zero is
+permitted.
+
+The third and following sections are unidirectional output sections, one for
+each peer. Their sizes are all identical. The size of a single output section is
+defined by the Output Section Size register. An output section is read/writable
+for the corresponding peer and read-only for all other peers. E.g., only the
+peer with ID 3 can write to the fourths output section, but all peers can read
+from this section.
+
+All sizes have to be rounded up to multiples of a mappable page in order to
+allow access control according to the section restrictions.
+
+### Configuration Space Registers
+
+#### Header Registers
+
+| Offset | Register               | Content                                              |
+|-------:|:-----------------------|:-----------------------------------------------------|
+|    00h | Vendor ID              | 1AF4h                                                |
+|    02h | Device ID              | 1110h                                                |
+|    04h | Command Register       | 0000h on reset, implementing bits 1, 2, 10           |
+|    06h | Status Register        | 0010h, static value (bit 3 not implemented)          |
+|    08h | Revision ID            | 02h                                                  |
+|    09h | Class Code, Interface  | Protocol Type bits 0-7, see [Protocols](#Protocols)  |
+|    0Ah | Class Code, Sub-Class  | Protocol Type bits 8-15, see [Protocols](#Protocols) |
+|    0Bh | Class Code, Base Class | FFh                                                  |
+|    0Eh | Header Type            | 00h                                                  |
+|    10h | BAR 0                  | MMIO or I/O register region                          |
+|    14h | BAR 1                  | MSI-X region                                         |
+|    18h | BAR 2 (with BAR 3)     | optional: 64-bit shared memory region                |
+|    2Ch | Subsystem Vendor ID    | same as Vendor ID, or provider-specific value        |
+|    2Eh | Subsystem ID           | same as Device ID, or provider-specific value        |
+|    34h | Capability Pointer     | First capability                                     |
+|    3Eh | Interrupt Pin          | 01h-04h, must be 00h if MSI-X is available           |
+
+If BAR 2 is not present, the shared memory region is not relocatable by the
+user. In that case, the hypervisor has to implement the Base Address register in
+the vendor-specific capability.
+
+Other header registers may not be implemented. If not implemented, they return 0
+on read and ignore write accesses.
+
+#### Vendor Specific Capability (ID 09h)
+
+| Offset | Register            | Content                                        |
+|-------:|:--------------------|:-----------------------------------------------|
+|    00h | ID                  | 09h                                            |
+|    01h | Next Capability     | Pointer to next capability or 00h              |
+|    02h | Length              | 18h or 20h                                     |
+|    03h | Privileged Control  | Bit 0 (read/write): one-shot interrupt mode    |
+|        |                     | Bits 1-7: RsvdZ                                |
+|    04h | State Table Size    | 32-bit size of read-only State Table           |
+|    08h | R/W Section Size    | 64-bit size of common read/write section       |
+|    10h | Output Section Size | 64-bit size of unidirectional output sections  |
+|    18h | Base Address        | optional: 64-bit base address of shared memory |
+
+All registers are read-only, except for bit 0 of the Privileged Control
+register.
+
+When bit 0 in the Privileged Control register is set to 1, the device clears
+bit 0 in the Interrupt Control register on each interrupt delivery. This enables
+automatic interrupt throttling when re-enabling shall be performed by a
+scheduled unprivileged instance on the user side.
+
+If an IVSHMEM device does not support a relocatable shared memory region, BAR 2
+must not be implemented by the provider. Instead, the Base Address register has
+to be implemented to report the location of the shared memory region in the
+user's address space.
+
+A non-existing shared memory section has to report zero in its Section Size
+register.
+
+#### MSI-X Capability (ID 11h)
+
+On platforms supporting MSI-X, IVSHMEM has to provide interrupt delivery via
+this mechanism. In that case, the legacy INTx delivery mechanism is not
+available, and the Interrupt Pin configuration register returns 0.
+
+The IVSHMEM device has no notion of pending interrupts. Therefore, reading from
+the MSI-X Pending Bit Array will always return 0.
+
+The corresponding MSI-X MMIO region is configured via BAR 1.
+
+The MSI-X table size reported by the MSI-X capability structure is identical for
+all peers.
+
+### Register Region
+
+The register region may be implemented as MMIO or I/O.
+
+When implementing it as MMIO, the hypervisor has to ensure that the register
+region can be mapped as a single page into the address space of the user. Write
+accesses to MMIO region offsets that are not backed by registers have to be
+ignored, read accesses have to return 0. This enables the user to hand out the
+complete region, along with the shared memory, to an unprivileged instance.
+
+The region location in the user's physical address space is configured via BAR
+0. The following table visualizes the region layout:
+
+| Offset | Register                                                            |
+|-------:|:--------------------------------------------------------------------|
+|    00h | ID                                                                  |
+|    04h | Maximum Peers                                                       |
+|    08h | Features                                                            |
+|    0Ch | Interrupt Control                                                   |
+|    10h | Doorbell                                                            |
+|    14h | State                                                               |
+
+All registers support only aligned 32-bit accesses.
+
+#### ID Register (Offset 00h)
+
+Read-only register that reports the ID of the local device. It is unique for all
+of the connected devices and remains unchanged over their lifetime.
+
+#### Maximum Peers Register (Offset 04h)
+
+Read-only register that reports the maximum number of possible peers (including
+the local one). The supported range is between 2 and 65536 and remains constant
+over the lifetime of all peers.
+
+#### Features Register (Offset 08h)
+
+Read-only register that reports features of the local device or the connected
+peers. Its content remains constant over the lifetime of all peers.
+
+| Bits | Content                                                               |
+|-----:|:----------------------------------------------------------------------|
+|    0 | 1: Synchronized shared memory base address                            |
+| 1-31 | RsvdZ                                                                 |
+
+If "synchronized shared memory base address" is reported (bit 0 is set), the
+shared memory region is mapped at the same address into the user address spaces
+of all connected peers. Thus, peers can use physical addresses as pointers when
+exchanging information via the shared memory. This feature flag is never set
+when the shared memory region is relocatable via BAR 2.
+
+#### Interrupt Control Register (Offset 0Ch)
+
+This read/write register controls the generation of interrupts whenever a peer
+writes to the Doorbell register or changes its state.
+
+| Bits | Content                                                               |
+|-----:|:----------------------------------------------------------------------|
+|    0 | 1: Enable interrupt generation                                        |
+| 1-31 | RsvdZ                                                                 |
+
+Note that bit 0 is reset to 0 on interrupt delivery if one-shot interrupt mode
+is enabled in the Enhanced Features register.
+
+The value of this register after device reset is 0.
+
+#### Doorbell Register (Offset 10h)
+
+Write-only register that triggers an interrupt vector in the target device if it
+is enabled there.
+
+| Bits  | Content                                                              |
+|------:|:---------------------------------------------------------------------|
+|  0-15 | Vector number                                                        |
+| 16-31 | Target ID                                                            |
+
+Writing a vector number that is not enabled by the target has no effect. The
+peers can derive the number of available vectors from their own device
+capabilities and are expected to define or negotiate the used ones via the
+selected protocol.
+
+Addressing a non-existing or inactive target has no effect. Peers can identify
+active targets via the State Table.
+
+The behavior on reading from this register is undefined.
+
+#### State Register (Offset 14h)
+
+Read/write register that defines the state of the local device. Writing to this
+register sets the state and triggers interrupt vector 0 on the remote device if
+the written state value differs from the previous one. The user of the remote
+device can read the value written to this register from the State Table.
+
+The value of this register after device reset is 0.
+
+### State Table
+
+The State Table is a read-only section at the beginning of the shared memory
+region. It contains a 32-bit state value for each of the peers. Locating the
+table in shared memory allows fast checking of remote states without register
+accesses.
+
+The table is updated on each state change of a peers. Whenever a user of an
+IVSHMEM device writes a value to the Local State register, this value is copied
+into the corresponding entry of the State Table. When a IVSHMEM device is reset
+or disconnected from the other peers, zero is written into the corresponding
+table entry. The initial content of the table is all zeros.
+
+    +--------------------------------+
+    | 32-bit state value of peer n-1 |
+    +--------------------------------+
+    :                                :
+    +--------------------------------+
+    | 32-bit state value of peer 1   |
+    +--------------------------------+
+    | 32-bit state value of peer 0   |
+    +--------------------------------+ <-- Shared memory base address
+
+
+Protocols
+---------
+
+The IVSHMEM device shall support the peers of a connection in agreeing on the
+protocol used over the shared memory devices. For that purpose, the interface
+byte (offset 09h) and the sub-class byte (offset 0Ah) of the Class Code register
+encodes a 16-bit protocol type for the users. The following type values are
+defined:
+
+| Protocol Type | Description                                                  |
+|--------------:|:-------------------------------------------------------------|
+|         0000h | Undefined type                                               |
+|         0001h | Virtual peer-to-peer Ethernet                                |
+|   0002h-3FFFh | Reserved                                                     |
+|   4000h-7FFFh | User-defined protocols                                       |
+|   8000h-BFFFh | Virtio over Shared Memory, front-end peer                    |
+|   C000h-FFFFh | Virtio over Shared Memory, back-end peer                     |
+
+Details of the protocols are not in the scope of this specification.
Signature · 1 line
2.16.4
#3 lore raw

Re: [RFC][PATCH 2/3] docs/specs: Add specification of ivshmem device revision 2

Quoted 4 lines · Jan Kiszka
+| Offset | Register               | Content                                              |
+|-------:|:-----------------------|:-----------------------------------------------------|
+|    00h | Vendor ID              | 1AF4h                                                |
+|    02h | Device ID              | 1110h                                                |

Given it's a virtio vendor ID, please reserve a device ID with the virtio TC.

Thanks!

Signature · 1 line
MST
author
#4 lore raw

Re: [RFC][PATCH 2/3] docs/specs: Add specification of ivshmem device revision 2

Quoted 8 lines · Michael S. Tsirkin
Quoted 4 lines · Jan Kiszka
+| Offset | Register               | Content                                              |
+|-------:|:-----------------------|:-----------------------------------------------------|
+|    00h | Vendor ID              | 1AF4h                                                |
+|    02h | Device ID              | 1110h                                                |

Given it's a virtio vendor ID, please reserve a device ID with the virtio TC.

Yeah, QEMU's IVSHMEM was always using that. I'm happy to make this finally official.

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
#5 lore raw

Re: [RFC][PATCH 2/3] docs/specs: Add specification of ivshmem device revision 2

Quoted 7 lines · Jan Kiszka
Quoted 6 lines · Michael S. Tsirkin
Quoted 4 lines · Jan Kiszka
+| Offset | Register               | Content                                              |
+|-------:|:-----------------------|:-----------------------------------------------------|
+|    00h | Vendor ID              | 1AF4h                                                |
+|    02h | Device ID              | 1110h                                                |
Quoted 7 lines
Quoted 2 lines

Given it's a virtio vendor ID, please reserve a device ID with the virtio TC.

Yeah, QEMU's IVSHMEM was always using that. I'm happy to make this finally official.

Jan

And I guess we will just mark it reserved or something right? Since at least IVSHMEM 1 isn't a virtio device.
And will you be reusing same ID for IVSHMEM 2 or a new one?

Quoted 4 lines
Signature · 3 lines
-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
#6 lore raw

Re: [RFC][PATCH 2/3] docs/specs: Add specification of ivshmem device revision 2

Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines · Jan Kiszka
Quoted 8 lines · Michael S. Tsirkin
Quoted 4 lines · Jan Kiszka
+| Offset | Register               | Content                                              |
+|-------:|:-----------------------|:-----------------------------------------------------|
+|    00h | Vendor ID              | 1AF4h                                                |
+|    02h | Device ID              | 1110h                                                |

Given it's a virtio vendor ID, please reserve a device ID with the virtio TC.

Quoted 7 lines
Quoted 3 lines

Yeah, QEMU's IVSHMEM was always using that. I'm happy to make this finally official.

And I guess we will just mark it reserved or something right? Since at least IVSHMEM 1 isn't a virtio device.
And will you be reusing same ID for IVSHMEM 2 or a new one?

1110h isn't under either of the virtio PCI device ID allowed ranges according to the spec:

"Any PCI device with PCI Vendor ID 0x1AF4, and PCI Device
ID 0x1000 through 0x107F inclusive is a virtio device.
...
Additionally, devices MAY utilize a Transitional PCI Device ID range, 0x1000 to 0x103F depending on the device type. "

So there's no need to reserve 0x1110h from the virtio spec POV.

I have, however, ensured it is assigned to ivshmem from POV of Red Hat's own internal tracking of allocated device IDs, under its vendor ID.

If ivshmem 2 is now a virtio device, then it is a good thing that it will get a new/different PCI device ID, to show that it is not compatible with the old device impl.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
author
#7 lore raw

Re: [RFC][PATCH 2/3] docs/specs: Add specification of ivshmem device revision 2

Quoted 23 lines · Daniel P. Berrangé
Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines · Jan Kiszka
Quoted 8 lines · Michael S. Tsirkin
Quoted 4 lines · Jan Kiszka
+| Offset | Register               | Content                                              |
+|-------:|:-----------------------|:-----------------------------------------------------|
+|    00h | Vendor ID              | 1AF4h                                                |
+|    02h | Device ID              | 1110h                                                |

Given it's a virtio vendor ID, please reserve a device ID with the virtio TC.

Quoted 7 lines
Quoted 3 lines

Yeah, QEMU's IVSHMEM was always using that. I'm happy to make this finally official.

And I guess we will just mark it reserved or something right? Since at least IVSHMEM 1 isn't a virtio device.
And will you be reusing same ID for IVSHMEM 2 or a new one?

1110h isn't under either of the virtio PCI device ID allowed ranges according to the spec:

Quoted 7 lines

"Any PCI device with PCI Vendor ID 0x1AF4, and PCI Device

    ID 0x1000 through 0x107F inclusive is a virtio device.
    ...
    Additionally, devices MAY utilize a Transitional PCI Device
    ID range, 0x1000 to 0x103F depending on the device type. "

So there's no need to reserve 0x1110h from the virtio spec POV.

Indeed.

Quoted 8 lines

I have, however, ensured it is assigned to ivshmem from POV of Red Hat's own internal tracking of allocated device IDs, under its vendor ID.

If ivshmem 2 is now a virtio device, then it is a good thing that it will get a new/different PCI device ID, to show that it is not compatible with the old device impl.

At this stage, it is just a PCI device that may be used in combination with virtio (stacked on top), but it is not designed like a normal virtio (PCI) device. That's because it lacks many properties of regular virtio devices, like queues.

So, if such a device could be come part of the virtio spec, it would be separate from the rest, and having an ID from the regular range would likely not be helpful in this regard.

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
#8 lore raw

Re: [RFC][PATCH 2/3] docs/specs: Add specification of ivshmem device revision 2

Quoted 48 lines · Jan Kiszka
Quoted 23 lines · Daniel P. Berrangé
Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines · Jan Kiszka
Quoted 8 lines · Michael S. Tsirkin
Quoted 4 lines · Jan Kiszka
+| Offset | Register               | Content                                              |
+|-------:|:-----------------------|:-----------------------------------------------------|
+|    00h | Vendor ID              | 1AF4h                                                |
+|    02h | Device ID              | 1110h                                                |

Given it's a virtio vendor ID, please reserve a device ID with the virtio TC.

Quoted 7 lines
Quoted 3 lines

Yeah, QEMU's IVSHMEM was always using that. I'm happy to make this finally official.

And I guess we will just mark it reserved or something right? Since at least IVSHMEM 1 isn't a virtio device.
And will you be reusing same ID for IVSHMEM 2 or a new one?

1110h isn't under either of the virtio PCI device ID allowed ranges according to the spec:

Quoted 7 lines

"Any PCI device with PCI Vendor ID 0x1AF4, and PCI Device

    ID 0x1000 through 0x107F inclusive is a virtio device.
    ...
    Additionally, devices MAY utilize a Transitional PCI Device
    ID range, 0x1000 to 0x103F depending on the device type. "

So there's no need to reserve 0x1110h from the virtio spec POV.

Indeed.

Quoted 8 lines

I have, however, ensured it is assigned to ivshmem from POV of Red Hat's own internal tracking of allocated device IDs, under its vendor ID.

If ivshmem 2 is now a virtio device, then it is a good thing that it will get a new/different PCI device ID, to show that it is not compatible with the old device impl.

At this stage, it is just a PCI device that may be used in combination with virtio (stacked on top), but it is not designed like a normal virtio (PCI) device. That's because it lacks many properties of regular virtio devices, like queues.

Quoted 5 lines

So, if such a device could be come part of the virtio spec, it would be separate from the rest, and having an ID from the regular range would likely not be helpful in this regard.

Jan

I agree it needs a separate ID not from the regular range. It's a distinct transport.
Maybe even a distinct vendor ID - we could easily get another one if needed.

Quoted 3 lines
Signature · 2 lines
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
author
#9 lore raw

Re: [RFC][PATCH 2/3] docs/specs: Add specification of ivshmem device revision 2

Quoted 53 lines · Michael S. Tsirkin
Quoted 52 lines · Jan Kiszka
Quoted 23 lines · Daniel P. Berrangé
Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines · Jan Kiszka
Quoted 8 lines · Michael S. Tsirkin
Quoted 4 lines · Jan Kiszka
+| Offset | Register               | Content                                              |
+|-------:|:-----------------------|:-----------------------------------------------------|
+|    00h | Vendor ID              | 1AF4h                                                |
+|    02h | Device ID              | 1110h                                                |

Given it's a virtio vendor ID, please reserve a device ID with the virtio TC.

Quoted 7 lines
Quoted 3 lines

Yeah, QEMU's IVSHMEM was always using that. I'm happy to make this finally official.

And I guess we will just mark it reserved or something right? Since at least IVSHMEM 1 isn't a virtio device.
And will you be reusing same ID for IVSHMEM 2 or a new one?

1110h isn't under either of the virtio PCI device ID allowed ranges according to the spec:

Quoted 7 lines
    "Any PCI device with PCI Vendor ID 0x1AF4, and PCI Device
     ID 0x1000 through 0x107F inclusive is a virtio device.
     ...
     Additionally, devices MAY utilize a Transitional PCI Device
     ID range, 0x1000 to 0x103F depending on the device type. "

So there's no need to reserve 0x1110h from the virtio spec POV.

Indeed.

Quoted 8 lines

I have, however, ensured it is assigned to ivshmem from POV of Red Hat's own internal tracking of allocated device IDs, under its vendor ID.

If ivshmem 2 is now a virtio device, then it is a good thing that it will get a new/different PCI device ID, to show that it is not compatible with the old device impl.

At this stage, it is just a PCI device that may be used in combination with virtio (stacked on top), but it is not designed like a normal virtio (PCI) device. That's because it lacks many properties of regular virtio devices, like queues.

So, if such a device could be come part of the virtio spec, it would be separate from the rest, and having an ID from the regular range would likely not be helpful in this regard.

Quoted 6 lines
Quoted 1 line

Jan

I agree it needs a separate ID not from the regular range. It's a distinct transport.
Maybe even a distinct vendor ID - we could easily get another one if needed.

That might be useful because I've seen the kernel's virtio-pci driver grabbing ivshmem devices from time to time. OTOH, that could likely also be improved in Linux, at least for future versions.

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
#10 lore raw

Re: [RFC][PATCH 2/3] docs/specs: Add specification of ivshmem device revision 2

In reply to Daniel P. Berrangé
Quoted 23 lines · Daniel P. Berrangé
Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines · Jan Kiszka
Quoted 8 lines · Michael S. Tsirkin
Quoted 4 lines · Jan Kiszka
+| Offset | Register               | Content                                              |
+|-------:|:-----------------------|:-----------------------------------------------------|
+|    00h | Vendor ID              | 1AF4h                                                |
+|    02h | Device ID              | 1110h                                                |

Given it's a virtio vendor ID, please reserve a device ID with the virtio TC.

Quoted 7 lines
Quoted 3 lines

Yeah, QEMU's IVSHMEM was always using that. I'm happy to make this finally official.

And I guess we will just mark it reserved or something right? Since at least IVSHMEM 1 isn't a virtio device.
And will you be reusing same ID for IVSHMEM 2 or a new one?

1110h isn't under either of the virtio PCI device ID allowed ranges according to the spec:

Quoted 7 lines

"Any PCI device with PCI Vendor ID 0x1AF4, and PCI Device
ID 0x1000 through 0x107F inclusive is a virtio device.
...
Additionally, devices MAY utilize a Transitional PCI Device ID range, 0x1000 to 0x103F depending on the device type. "

So there's no need to reserve 0x1110h from the virtio spec POV.

Well we do have:

B.3
What Device Number?
Device numbers can be reserved by the OASIS committee: email [email protected] to secure a unique one.
Meanwhile for experimental drivers, use 65535 and work backwards.

So it seems it can in theory conflict at least with experimental virtio devices.

Really it's messy that people are reusing the virtio vendor ID for random stuff - getting a vendor ID is only hard for a hobbyist, any big company already has an ID - but if it is a hobbyist and they at least register then doesn't cause much harm.

E.g. Red Hat switched to 1b36 for new non virtio devices and I think that's nicer.

Quoted 3 lines

I have, however, ensured it is assigned to ivshmem from POV of Red Hat's own internal tracking of allocated device IDs, under its vendor ID.

Thanks!

Quoted 10 lines

If ivshmem 2 is now a virtio device, then it is a good thing that it will get a new/different PCI device ID, to show that it is not compatible with the old device impl.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
author
#11 lore raw

Re: [RFC][PATCH 2/3] docs/specs: Add specification of ivshmem device revision 2

Quoted 40 lines · Michael S. Tsirkin
Quoted 23 lines · Daniel P. Berrangé
Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines · Jan Kiszka
Quoted 8 lines · Michael S. Tsirkin
Quoted 4 lines · Jan Kiszka
+| Offset | Register               | Content                                              |
+|-------:|:-----------------------|:-----------------------------------------------------|
+|    00h | Vendor ID              | 1AF4h                                                |
+|    02h | Device ID              | 1110h                                                |

Given it's a virtio vendor ID, please reserve a device ID with the virtio TC.

Quoted 7 lines
Quoted 3 lines

Yeah, QEMU's IVSHMEM was always using that. I'm happy to make this finally official.

And I guess we will just mark it reserved or something right? Since at least IVSHMEM 1 isn't a virtio device.
And will you be reusing same ID for IVSHMEM 2 or a new one?

1110h isn't under either of the virtio PCI device ID allowed ranges according to the spec:

Quoted 7 lines

"Any PCI device with PCI Vendor ID 0x1AF4, and PCI Device

    ID 0x1000 through 0x107F inclusive is a virtio device.
    ...
    Additionally, devices MAY utilize a Transitional PCI Device
    ID range, 0x1000 to 0x103F depending on the device type. "

So there's no need to reserve 0x1110h from the virtio spec POV.

Well we do have:

B.3
What Device Number?
Device numbers can be reserved by the OASIS committee: email [email protected] to secure a unique one.
Meanwhile for experimental drivers, use 65535 and work backwards.

Quoted 6 lines

So it seems it can in theory conflict at least with experimental virtio devices.

Really it's messy that people are reusing the virtio vendor ID for random stuff - getting a vendor ID is only hard for a hobbyist, any big company already has an ID - but if it is a hobbyist and they at least register then doesn't cause much harm.

Note that ivshmem came from a research environment. I do know if there was a check for the IDs at the point the code was merged.

That said, I may get a device ID here as well, provided I can explain that not a single "product" will own it, but rather an open specification.

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
#12 lore raw

Re: [RFC][PATCH 2/3] docs/specs: Add specification of ivshmem device revision 2

Quoted 48 lines · Jan Kiszka
Quoted 40 lines · Michael S. Tsirkin
Quoted 23 lines · Daniel P. Berrangé
Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines · Jan Kiszka
Quoted 8 lines · Michael S. Tsirkin
Quoted 4 lines · Jan Kiszka
+| Offset | Register               | Content                                              |
+|-------:|:-----------------------|:-----------------------------------------------------|
+|    00h | Vendor ID              | 1AF4h                                                |
+|    02h | Device ID              | 1110h                                                |

Given it's a virtio vendor ID, please reserve a device ID with the virtio TC.

Quoted 7 lines
Quoted 3 lines

Yeah, QEMU's IVSHMEM was always using that. I'm happy to make this finally official.

And I guess we will just mark it reserved or something right? Since at least IVSHMEM 1 isn't a virtio device.
And will you be reusing same ID for IVSHMEM 2 or a new one?

1110h isn't under either of the virtio PCI device ID allowed ranges according to the spec:

Quoted 7 lines

"Any PCI device with PCI Vendor ID 0x1AF4, and PCI Device

    ID 0x1000 through 0x107F inclusive is a virtio device.
    ...
    Additionally, devices MAY utilize a Transitional PCI Device
    ID range, 0x1000 to 0x103F depending on the device type. "

So there's no need to reserve 0x1110h from the virtio spec POV.

Well we do have:

B.3
What Device Number?
Device numbers can be reserved by the OASIS committee: email [email protected] to secure a unique one.
Meanwhile for experimental drivers, use 65535 and work backwards.

Quoted 6 lines

So it seems it can in theory conflict at least with experimental virtio devices.

Really it's messy that people are reusing the virtio vendor ID for random stuff - getting a vendor ID is only hard for a hobbyist, any big company already has an ID - but if it is a hobbyist and they at least register then doesn't cause much harm.

Quoted 7 lines

Note that ivshmem came from a research environment. I do know if there was a check for the IDs at the point the code was merged.

That said, I may get a device ID here as well, provided I can explain that not a single "product" will own it, but rather an open specification.

Jan

OK, up to you - if you decide you want an ID reserved, pls let us know.

At this point I'm not sure I have a good grasp which IDs are registered where anymore. If someone can write it up, that would be great too!

Quoted 3 lines
Signature · 2 lines
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
author
#13 lore raw

Re: [RFC][PATCH 2/3] docs/specs: Add specification of ivshmem device revision 2

Quoted 52 lines · Michael S. Tsirkin
Quoted 51 lines · Jan Kiszka
Quoted 40 lines · Michael S. Tsirkin
Quoted 23 lines · Daniel P. Berrangé
Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines · Jan Kiszka
Quoted 8 lines · Michael S. Tsirkin
Quoted 4 lines · Jan Kiszka
+| Offset | Register               | Content                                              |
+|-------:|:-----------------------|:-----------------------------------------------------|
+|    00h | Vendor ID              | 1AF4h                                                |
+|    02h | Device ID              | 1110h                                                |

Given it's a virtio vendor ID, please reserve a device ID with the virtio TC.

Quoted 7 lines
Quoted 3 lines

Yeah, QEMU's IVSHMEM was always using that. I'm happy to make this finally official.

And I guess we will just mark it reserved or something right? Since at least IVSHMEM 1 isn't a virtio device.
And will you be reusing same ID for IVSHMEM 2 or a new one?

1110h isn't under either of the virtio PCI device ID allowed ranges according to the spec:

Quoted 7 lines
    "Any PCI device with PCI Vendor ID 0x1AF4, and PCI Device
     ID 0x1000 through 0x107F inclusive is a virtio device.
     ...
     Additionally, devices MAY utilize a Transitional PCI Device
     ID range, 0x1000 to 0x103F depending on the device type. "

So there's no need to reserve 0x1110h from the virtio spec POV.

Well we do have:

B.3
What Device Number?
Device numbers can be reserved by the OASIS committee: email [email protected] to secure a unique one.
Meanwhile for experimental drivers, use 65535 and work backwards.

Quoted 6 lines

So it seems it can in theory conflict at least with experimental virtio devices.

Really it's messy that people are reusing the virtio vendor ID for random stuff - getting a vendor ID is only hard for a hobbyist, any big company already has an ID - but if it is a hobbyist and they at least register then doesn't cause much harm.

Note that ivshmem came from a research environment. I do know if there was a check for the IDs at the point the code was merged.

Quoted 7 lines
Quoted 4 lines

That said, I may get a device ID here as well, provided I can explain that not a single "product" will own it, but rather an open specification.

Jan

OK, up to you - if you decide you want an ID reserved, pls let us know.

Turned out to be much simpler than expect:

I reserved device ID 4106h under the vendor ID 110Ah (Siemens AG) for the purpose of specifying a shared memory device via the VIRTIO TC. Will update this "detail" in the next revision of the patches, also resetting the device revision ID to 0 as no longer need to tell us apart from the current implementation this way.

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
#14 lore raw

Re: [RFC][PATCH 2/3] docs/specs: Add specification of ivshmem device revision 2

In reply to Jan Kiszka

This has been on the list for more than three weeks already. I apologize for the delay.

Quoted 47 lines · Jan Kiszka

From: Jan Kiszka <[email protected]>

This imports the ivshmem v2 specification draft from Jailhouse. Its final home is to be decided, this shall just simplify the review process at this stage.

Note that specifically the Features register (offset 08h) is still under consideration. In particular, its bit 0 seems useless now as its benefit to guests, specifically when they want to be portable, is close to zero. Maybe the register should still be kept, with all bits RsvdZ, for easier extensibility.

The rest appears now rather mature and reasonably implementable, as proven also by a version for Jailhouse.

---
 docs/specs/ivshmem-2-device-spec.md | 333 ++++++++++++++++++++++++++++++++++++
 1 file changed, 333 insertions(+)
 create mode 100644 docs/specs/ivshmem-2-device-spec.md
docs/specs/ivshmem-2-device-spec.md new file +20 -0
new file mode 100644
index 0000000000..98cfde585a
@@ -0,0 +1,333 @@
+IVSHMEM Device Specification
+============================
+
+** NOTE: THIS IS WORK-IN-PROGRESS, NOT YET A STABLE INTERFACE SPECIFICATION! **
+
+The Inter-VM Shared Memory device provides the following features to its users:
+
+- Interconnection between up to 65536 peers
+
+- Multi-purpose shared memory region
+
+    - common read/writable section
+
+    - unidirectional sections that are read/writable for one peer and only
+      readable for the others
+
+    - section with peer states
+
+- Event signaling via interrupt to remote sides
+

docs/specs/ivshmem-2-device-spec.md line 30

Quoted 10 lines
docs/specs/ivshmem-2-device-spec.md new file +10 -0
+- Support for life-cycle management via state value exchange and interrupt
+  notification on changes, backed by a shared memory section
+
+- Free choice of protocol to be used on top
+
+- Protocol type declaration
+
+- Unprivileged access to memory-mapped or I/O registers feasible
+
+- Discoverable and configurable via standard PCI mechanisms

Stating requirements is much appreciated. Design rationale would be even better :)

As pointed out many times, shared memory is not a solution to any communication problem, it's merely a building block for building such
solutions: you invariably have to layer some protocol on top. In your KVM Forum talk, you mention layering virtio on top. Makes sense to me. But why does *this* virtio transport have to be an independent device? Other transports aren't.

Now let me indulge in spec nitpicking :)

docs/specs/ivshmem-2-device-spec.md line 41

Quoted 11 lines
docs/specs/ivshmem-2-device-spec.md new file +11 -0
+
+
+Hypervisor Model
+----------------
+
+In order to provide a consistent link between peers, all connected instances of
+IVSHMEM devices need to be configured, created and run by the hypervisor
+according to the following requirements:
+
+- The instances of the device need to be accessible via PCI programming
+  interfaces on all sides.

What does that mean?

docs/specs/ivshmem-2-device-spec.md line 44

Quoted 3 lines
docs/specs/ivshmem-2-device-spec.md new file +3 -0
+
+- The read/write shared memory section has to be of the same size for all
+  peers and, if non-zero, has to reflect the same memory content for them.

Isn't "same memory content" redundant with "shared memory"?

docs/specs/ivshmem-2-device-spec.md line 49

Quoted 5 lines
docs/specs/ivshmem-2-device-spec.md new file +5 -0
+
+- If output sections are present (non-zero section size), there must be one
+  reserved for each peer with exclusive write access. All output sections
+  must have the same size and must be readable for all peers. They have to
+  reflect the same memory content for all peers.

Are these the "unidirectional sections" mentioned previously?

docs/specs/ivshmem-2-device-spec.md line 52

Quoted 3 lines
docs/specs/ivshmem-2-device-spec.md new file +3 -0
+
+- The State Table must have the same size for all peers, must be large enough to
+  hold a state values of all peers, and must be read-only for the user.

"the state values", I guess.

docs/specs/ivshmem-2-device-spec.md line 63

Quoted 11 lines
docs/specs/ivshmem-2-device-spec.md new file +11 -0
+
+- State register changes (explicit writes, peer resets) have to be propagated
+  to the other peers by updating the corresponding State Table entry and issuing
+  an interrupt to all other peers if they enabled reception.
+
+- Interrupts events triggered by a peer have to be delivered to the target peer,
+  provided the receiving side is valid and has enabled the reception.
+
+- All peers must have the same interrupt delivery features available, i.e. MSI-X
+  with the same maximum number of vectors on platforms supporting this
+  mechanism, otherwise INTx with one vector.

Use case for legacy INTx?

For what it's worth, we removed INTx support from ivshmem v1 in rev 1, QEMU 2.6.

docs/specs/ivshmem-2-device-spec.md line 70

Quoted 7 lines
docs/specs/ivshmem-2-device-spec.md new file +7 -0
+
+
+Guest-side Programming Model
+----------------------------
+
+An IVSHMEM device appears as a PCI device to its users. Unless otherwise noted,
+it conforms to the PCI Local Bus Specification, Revision 3.0 As such, it is

Uh, is there anything in it that does *not* conform to this spec?

Quoted 49 lines
docs/specs/ivshmem-2-device-spec.md new file +49 -0
+discoverable via the PCI configuration space and provides a number of standard
+and custom PCI configuration registers.
+
+### Shared Memory Region Layout
+
+The shared memory region is divided into several sections.
+
+    +-----------------------------+   -
+    |                             |   :
+    | Output Section for peer n-1 |   : Output Section Size
+    |     (n = Maximum Peers)     |   :
+    +-----------------------------+   -
+    :                             :
+    :                             :
+    :                             :
+    +-----------------------------+   -
+    |                             |   :
+    |  Output Section for peer 1  |   : Output Section Size
+    |                             |   :
+    +-----------------------------+   -
+    |                             |   :
+    |  Output Section for peer 0  |   : Output Section Size
+    |                             |   :
+    +-----------------------------+   -
+    |                             |   :
+    |     Read/Write Section      |   : R/W Section Size
+    |                             |   :
+    +-----------------------------+   -
+    |                             |   :
+    |         State Table         |   : State Table Size
+    |                             |   :
+    +-----------------------------+   <-- Shared memory base address
+
+The first section consists of the mandatory State Table. Its size is defined by
+the State Table Size register and cannot be zero. This section is read-only for
+all peers.
+
+The second section consists of shared memory that is read/writable for all
+peers. Its size is defined by the R/W Section Size register. A size of zero is
+permitted.
+
+The third and following sections are unidirectional output sections, one for
+each peer. Their sizes are all identical. The size of a single output section is
+defined by the Output Section Size register. An output section is read/writable
+for the corresponding peer and read-only for all other peers. E.g., only the
+peer with ID 3 can write to the fourths output section, but all peers can read
+from this section.
+
+All sizes have to be rounded up to multiples of a mappable page in order to

docs/specs/ivshmem-2-device-spec.md line 129

Quoted 10 lines
docs/specs/ivshmem-2-device-spec.md new file +10 -0
+allow access control according to the section restrictions.
+
+### Configuration Space Registers
+
+#### Header Registers
+
+| Offset | Register               | Content                                              |
+|-------:|:-----------------------|:-----------------------------------------------------|
+|    00h | Vendor ID              | 1AF4h                                                |
+|    02h | Device ID              | 1110h                                                |

Same as ivshmem v1. Revision ID (offset 08h) disambiguates.

docs/specs/ivshmem-2-device-spec.md line 130

Quoted 1 line
docs/specs/ivshmem-2-device-spec.md new file +1 -0
+|    04h | Command Register       | 0000h on reset, implementing bits 1, 2, 10           |

What does "implementing bits ..." mean?

docs/specs/ivshmem-2-device-spec.md line 131

Quoted 1 line
docs/specs/ivshmem-2-device-spec.md new file +1 -0
+|    06h | Status Register        | 0010h, static value (bit 3 not implemented)          |

What does "bit 3 not implemented" mean?

docs/specs/ivshmem-2-device-spec.md line 132

Quoted 1 line
docs/specs/ivshmem-2-device-spec.md new file +1 -0
+|    08h | Revision ID            | 02h                                                  |

ivshmem v1 is rev 0 before QEMU 2.6, rev 1 since. Rev 1 is backward-compatible to rev 0 for guest software. Is rev 2 intended to be backward-compatible, too?

You should probably explan how your v2 relates to v1 in more detail, possibly in its own top-level section.

docs/specs/ivshmem-2-device-spec.md line 135

Quoted 3 lines
docs/specs/ivshmem-2-device-spec.md new file +3 -0
+|    09h | Class Code, Interface  | Protocol Type bits 0-7, see [Protocols](#Protocols)  |
+|    0Ah | Class Code, Sub-Class  | Protocol Type bits 8-15, see [Protocols](#Protocols) |
+|    0Bh | Class Code, Base Class | FFh                                                  |

FFh means "device does not fit any defined class." For what it's worth, ivshmem v1 uses 05h "Memory Controller", with sub-class and interface 00h "RAM Controller".

docs/specs/ivshmem-2-device-spec.md line 137

Quoted 2 lines
docs/specs/ivshmem-2-device-spec.md new file +2 -0
+|    0Eh | Header Type            | 00h                                                  |
+|    10h | BAR 0                  | MMIO or I/O register region                          |

Use case for I/O?

For what it's worth, ivshmem v1 never supported I/O.

docs/specs/ivshmem-2-device-spec.md line 139

Quoted 2 lines
docs/specs/ivshmem-2-device-spec.md new file +2 -0
+|    14h | BAR 1                  | MSI-X region                                         |
+|    18h | BAR 2 (with BAR 3)     | optional: 64-bit shared memory region                |

What does "(with BAR 3)" mean?

docs/specs/ivshmem-2-device-spec.md line 141

Quoted 2 lines
docs/specs/ivshmem-2-device-spec.md new file +2 -0
+|    2Ch | Subsystem Vendor ID    | same as Vendor ID, or provider-specific value        |
+|    2Eh | Subsystem ID           | same as Device ID, or provider-specific value        |

ivshmem v1 leaves these blank.

docs/specs/ivshmem-2-device-spec.md line 143

Quoted 2 lines
docs/specs/ivshmem-2-device-spec.md new file +2 -0
+|    34h | Capability Pointer     | First capability                                     |
+|    3Eh | Interrupt Pin          | 01h-04h, must be 00h if MSI-X is available           |

"If MSI-X is available"?

A PCIe device always provides MSI-X, and may additionally provide legacy INTx. A conventional PCI device may provide either or both. In any case, the Interrupt Pin register is zero when legacy INTx is not provided,

docs/specs/ivshmem-2-device-spec.md line 150

Quoted 7 lines
docs/specs/ivshmem-2-device-spec.md new file +7 -0
+
+If BAR 2 is not present, the shared memory region is not relocatable by the
+user. In that case, the hypervisor has to implement the Base Address register in
+the vendor-specific capability.
+
+Other header registers may not be implemented. If not implemented, they return 0
+on read and ignore write accesses.

Is this an example of where the device does not conform to the PCI Local Bus Specification?

docs/specs/ivshmem-2-device-spec.md line 160

Quoted 10 lines
docs/specs/ivshmem-2-device-spec.md new file +10 -0
+
+#### Vendor Specific Capability (ID 09h)
+
+| Offset | Register            | Content                                        |
+|-------:|:--------------------|:-----------------------------------------------|
+|    00h | ID                  | 09h                                            |
+|    01h | Next Capability     | Pointer to next capability or 00h              |
+|    02h | Length              | 18h or 20h                                     |
+|    03h | Privileged Control  | Bit 0 (read/write): one-shot interrupt mode    |
+|        |                     | Bits 1-7: RsvdZ                                |

Please define RsvdZ somewhere, or use plainer language.

docs/specs/ivshmem-2-device-spec.md line 164

Quoted 4 lines
docs/specs/ivshmem-2-device-spec.md new file +4 -0
+|    04h | State Table Size    | 32-bit size of read-only State Table           |
+|    08h | R/W Section Size    | 64-bit size of common read/write section       |
+|    10h | Output Section Size | 64-bit size of unidirectional output sections  |
+|    18h | Base Address        | optional: 64-bit base address of shared memory |

Length is 20h when Base Adress is present, else 18h. Worth spelling that out?

docs/specs/ivshmem-2-device-spec.md line 167

Quoted 3 lines
docs/specs/ivshmem-2-device-spec.md new file +3 -0
+
+All registers are read-only, except for bit 0 of the Privileged Control
+register.

Well, the other bits of that register are writable, they're just ignored.

For what it's worth, existing ivshmem-spec.txt uses the terms read-only, write-only, read/write and reserved bits rigorously: "Software should only access the registers as specified [...] Reserved bits should be ignored on read, and preserved on write."

docs/specs/ivshmem-2-device-spec.md line 177

Quoted 10 lines
docs/specs/ivshmem-2-device-spec.md new file +10 -0
+
+When bit 0 in the Privileged Control register is set to 1, the device clears
+bit 0 in the Interrupt Control register on each interrupt delivery. This enables
+automatic interrupt throttling when re-enabling shall be performed by a
+scheduled unprivileged instance on the user side.
+
+If an IVSHMEM device does not support a relocatable shared memory region, BAR 2
+must not be implemented by the provider. Instead, the Base Address register has
+to be implemented to report the location of the shared memory region in the
+user's address space.

Rationale for not wanting to support relocatable shared memory?

docs/specs/ivshmem-2-device-spec.md line 180

Quoted 3 lines
docs/specs/ivshmem-2-device-spec.md new file +3 -0
+
+A non-existing shared memory section has to report zero in its Section Size
+register.

This vendor-specific capability must always be present, I presume. Worth spelling out.

docs/specs/ivshmem-2-device-spec.md line 186

Quoted 6 lines
docs/specs/ivshmem-2-device-spec.md new file +6 -0
+
+#### MSI-X Capability (ID 11h)
+
+On platforms supporting MSI-X, IVSHMEM has to provide interrupt delivery via
+this mechanism. In that case, the legacy INTx delivery mechanism is not
+available, and the Interrupt Pin configuration register returns 0.

I'm confused. Does that mean the device shall support either MSI-X or legacy INTx, but never both?

docs/specs/ivshmem-2-device-spec.md line 189

Quoted 3 lines
docs/specs/ivshmem-2-device-spec.md new file +3 -0
+
+The IVSHMEM device has no notion of pending interrupts. Therefore, reading from
+the MSI-X Pending Bit Array will always return 0.

I guess this means a polling mode of operation is not possible. Correct?

docs/specs/ivshmem-2-device-spec.md line 201

Quoted 12 lines
docs/specs/ivshmem-2-device-spec.md new file +12 -0
+
+The corresponding MSI-X MMIO region is configured via BAR 1.
+
+The MSI-X table size reported by the MSI-X capability structure is identical for
+all peers.
+
+### Register Region
+
+The register region may be implemented as MMIO or I/O.
+
+When implementing it as MMIO, the hypervisor has to ensure that the register
+region can be mapped as a single page into the address space of the user. Write

"can be mapped as a single page" depends on the host system, not the device.

For what it's worth, ivshmem v1 fixes the size of BAR0 to 256 bytes. Any particular reason to keep its size so loosely specified in v2?

Quoted 7 lines
docs/specs/ivshmem-2-device-spec.md new file +7 -0
+accesses to MMIO region offsets that are not backed by registers have to be
+ignored, read accesses have to return 0. This enables the user to hand out the
+complete region, along with the shared memory, to an unprivileged instance.
+
+The region location in the user's physical address space is configured via BAR
+0. The following table visualizes the region layout:
+

docs/specs/ivshmem-2-device-spec.md line 218

Quoted 10 lines
docs/specs/ivshmem-2-device-spec.md new file +10 -0
+| Offset | Register                                                            |
+|-------:|:--------------------------------------------------------------------|
+|    00h | ID                                                                  |
+|    04h | Maximum Peers                                                       |
+|    08h | Features                                                            |
+|    0Ch | Interrupt Control                                                   |
+|    10h | Doorbell                                                            |
+|    14h | State                                                               |
+
+All registers support only aligned 32-bit accesses.

Definitely not backwards compatible to rev 1. I figure that means v2 should use a different Vendor ID / Device ID, not just bump the Revsion ID.

No interrupt status register?

docs/specs/ivshmem-2-device-spec.md line 229

Quoted 11 lines
docs/specs/ivshmem-2-device-spec.md new file +11 -0
+
+#### ID Register (Offset 00h)
+
+Read-only register that reports the ID of the local device. It is unique for all
+of the connected devices and remains unchanged over their lifetime.
+
+#### Maximum Peers Register (Offset 04h)
+
+Read-only register that reports the maximum number of possible peers (including
+the local one). The supported range is between 2 and 65536 and remains constant
+over the lifetime of all peers.

Value 1 would be boring, but is it really impossible?

Quoted 21 lines
docs/specs/ivshmem-2-device-spec.md new file +21 -0
+
+#### Features Register (Offset 08h)
+
+Read-only register that reports features of the local device or the connected
+peers. Its content remains constant over the lifetime of all peers.
+
+| Bits | Content                                                               |
+|-----:|:----------------------------------------------------------------------|
+|    0 | 1: Synchronized shared memory base address                            |
+| 1-31 | RsvdZ                                                                 |
+
+If "synchronized shared memory base address" is reported (bit 0 is set), the
+shared memory region is mapped at the same address into the user address spaces
+of all connected peers. Thus, peers can use physical addresses as pointers when
+exchanging information via the shared memory. This feature flag is never set
+when the shared memory region is relocatable via BAR 2.
+
+#### Interrupt Control Register (Offset 0Ch)
+
+This read/write register controls the generation of interrupts whenever a peer
+writes to the Doorbell register or changes its state.

docs/specs/ivshmem-2-device-spec.md line 260

Quoted 10 lines
docs/specs/ivshmem-2-device-spec.md new file +10 -0
+
+| Bits | Content                                                               |
+|-----:|:----------------------------------------------------------------------|
+|    0 | 1: Enable interrupt generation                                        |
+| 1-31 | RsvdZ                                                                 |
+
+Note that bit 0 is reset to 0 on interrupt delivery if one-shot interrupt mode
+is enabled in the Enhanced Features register.
+
+The value of this register after device reset is 0.

I presume this applies only to legacy INTx. Correct?

ivshmem v1 calls this Interrupt Mask.

Quoted 5 lines
docs/specs/ivshmem-2-device-spec.md new file +5 -0
+
+#### Doorbell Register (Offset 10h)
+
+Write-only register that triggers an interrupt vector in the target device if it
+is enabled there.

docs/specs/ivshmem-2-device-spec.md line 275

Quoted 10 lines
docs/specs/ivshmem-2-device-spec.md new file +10 -0
+
+| Bits  | Content                                                              |
+|------:|:---------------------------------------------------------------------|
+|  0-15 | Vector number                                                        |
+| 16-31 | Target ID                                                            |
+
+Writing a vector number that is not enabled by the target has no effect. The
+peers can derive the number of available vectors from their own device
+capabilities and are expected to define or negotiate the used ones via the
+selected protocol.

That's because all peers use the same number of vectors. Worth spelling out?

docs/specs/ivshmem-2-device-spec.md line 285

Quoted 10 lines
docs/specs/ivshmem-2-device-spec.md new file +10 -0
+
+Addressing a non-existing or inactive target has no effect. Peers can identify
+active targets via the State Table.
+
+The behavior on reading from this register is undefined.
+
+#### State Register (Offset 14h)
+
+Read/write register that defines the state of the local device. Writing to this
+register sets the state and triggers interrupt vector 0 on the remote device if

"the remote device"? Do you mean all peers?

"interrupt vector 0" assumes MSI-X. With legacy INTx, it simply triggers an interrupt.

How can guest software distinguish between "interrupt due to remote state change" and "interrupt due to doorbell"?

Things become so much easier when you ditch INTx: reserve vector 0 for state change, done.

docs/specs/ivshmem-2-device-spec.md line 289

Quoted 4 lines
docs/specs/ivshmem-2-device-spec.md new file +4 -0
+the written state value differs from the previous one. The user of the remote
+device can read the value written to this register from the State Table.
+
+The value of this register after device reset is 0.

I guess the meaning of state values depends on the protocol type. Correct?

Quoted 34 lines
docs/specs/ivshmem-2-device-spec.md new file +34 -0
+
+### State Table
+
+The State Table is a read-only section at the beginning of the shared memory
+region. It contains a 32-bit state value for each of the peers. Locating the
+table in shared memory allows fast checking of remote states without register
+accesses.
+
+The table is updated on each state change of a peers. Whenever a user of an
+IVSHMEM device writes a value to the Local State register, this value is copied
+into the corresponding entry of the State Table. When a IVSHMEM device is reset
+or disconnected from the other peers, zero is written into the corresponding
+table entry. The initial content of the table is all zeros.
+
+    +--------------------------------+
+    | 32-bit state value of peer n-1 |
+    +--------------------------------+
+    :                                :
+    +--------------------------------+
+    | 32-bit state value of peer 1   |
+    +--------------------------------+
+    | 32-bit state value of peer 0   |
+    +--------------------------------+ <-- Shared memory base address
+
+
+Protocols
+---------
+
+The IVSHMEM device shall support the peers of a connection in agreeing on the
+protocol used over the shared memory devices. For that purpose, the interface
+byte (offset 09h) and the sub-class byte (offset 0Ah) of the Class Code register
+encodes a 16-bit protocol type for the users. The following type values are
+defined:
+

docs/specs/ivshmem-2-device-spec.md line 333

Quoted 10 lines
docs/specs/ivshmem-2-device-spec.md new file +10 -0
+| Protocol Type | Description                                                  |
+|--------------:|:-------------------------------------------------------------|
+|         0000h | Undefined type                                               |
+|         0001h | Virtual peer-to-peer Ethernet                                |
+|   0002h-3FFFh | Reserved                                                     |
+|   4000h-7FFFh | User-defined protocols                                       |
+|   8000h-BFFFh | Virtio over Shared Memory, front-end peer                    |
+|   C000h-FFFFh | Virtio over Shared Memory, back-end peer                     |
+
+Details of the protocols are not in the scope of this specification.

Are you sure this use of PCI class code is kosher?

Final request: please break your lines around column 70 for readability.

author
#15 lore raw

Re: [RFC][PATCH 2/3] docs/specs: Add specification of ivshmem device revision 2

Quoted 2 lines · Markus Armbruster

This has been on the list for more than three weeks already. I apologize for the delay.

No problem! Your feedback is highly appreciated.

Quoted 55 lines
Quoted 52 lines · Jan Kiszka

From: Jan Kiszka <[email protected]>

This imports the ivshmem v2 specification draft from Jailhouse. Its final home is to be decided, this shall just simplify the review process at this stage.

Note that specifically the Features register (offset 08h) is still under consideration. In particular, its bit 0 seems useless now as its benefit to guests, specifically when they want to be portable, is close to zero. Maybe the register should still be kept, with all bits RsvdZ, for easier extensibility.

The rest appears now rather mature and reasonably implementable, as proven also by a version for Jailhouse.

---
 docs/specs/ivshmem-2-device-spec.md | 333 ++++++++++++++++++++++++++++++++++++
 1 file changed, 333 insertions(+)
 create mode 100644 docs/specs/ivshmem-2-device-spec.md
docs/specs/ivshmem-2-device-spec.md new file +25 -0
new file mode 100644
index 0000000000..98cfde585a
@@ -0,0 +1,333 @@
+IVSHMEM Device Specification
+============================
+
+** NOTE: THIS IS WORK-IN-PROGRESS, NOT YET A STABLE INTERFACE SPECIFICATION! **
+
+The Inter-VM Shared Memory device provides the following features to its users:
+
+- Interconnection between up to 65536 peers
+
+- Multi-purpose shared memory region
+
+    - common read/writable section
+
+    - unidirectional sections that are read/writable for one peer and only
+      readable for the others
+
+    - section with peer states
+
+- Event signaling via interrupt to remote sides
+
+- Support for life-cycle management via state value exchange and interrupt
+  notification on changes, backed by a shared memory section
+
+- Free choice of protocol to be used on top
+
Quoted 8 lines

docs/specs/ivshmem-2-device-spec.md line 30

Quoted 5 lines
docs/specs/ivshmem-2-device-spec.md new file +5 -0
+- Protocol type declaration
+
+- Unprivileged access to memory-mapped or I/O registers feasible
+
+- Discoverable and configurable via standard PCI mechanisms

Stating requirements is much appreciated. Design rationale would be even better :)

The idea is to avoid having to model also a platform device with platform discovery mechanism. Based on our experiences with providing a generic virtual PCI host controller, such a complication of specification and implementation is simply not needed.

Quoted 7 lines

As pointed out many times, shared memory is not a solution to any communication problem, it's merely a building block for building such
solutions: you invariably have to layer some protocol on top. In your KVM Forum talk, you mention layering virtio on top. Makes sense to me. But why does *this* virtio transport have to be an independent device? Other transports aren't.

This device is not only a virtio transport. As you correctly point out, it is a foundation for communication, not the full solution by itself.

Its goal is to define the minimal piece that a hypervisor has to provide to its guests in order to enable them building full communication solutions of their preferred flavor on top. It does not want to deal with those details, may they be virtio, may they be something much simpler, much older or much more sophisticated (or complicated). All that shall not be the business of a hypervisor.

Quoted 16 lines

Now let me indulge in spec nitpicking :)

docs/specs/ivshmem-2-device-spec.md line 41

Quoted 11 lines
docs/specs/ivshmem-2-device-spec.md new file +11 -0
+
+
+Hypervisor Model
+----------------
+
+In order to provide a consistent link between peers, all connected instances of
+IVSHMEM devices need to be configured, created and run by the hypervisor
+according to the following requirements:
+
+- The instances of the device need to be accessible via PCI programming
+  interfaces on all sides.

What does that mean?

"From a guest point of view, this shall be a PCI device."

Quoted 6 lines

docs/specs/ivshmem-2-device-spec.md line 44

Quoted 4 lines
docs/specs/ivshmem-2-device-spec.md new file +3 -0
+
+- The read/write shared memory section has to be of the same size for all
+  peers and, if non-zero, has to reflect the same memory content for them.

Isn't "same memory content" redundant with "shared memory"?

Probably.

Quoted 8 lines

docs/specs/ivshmem-2-device-spec.md line 49

Quoted 6 lines
docs/specs/ivshmem-2-device-spec.md new file +5 -0
+
+- If output sections are present (non-zero section size), there must be one
+  reserved for each peer with exclusive write access. All output sections
+  must have the same size and must be readable for all peers. They have to
+  reflect the same memory content for all peers.

Are these the "unidirectional sections" mentioned previously?

Yep. The terms have some history, maybe worth consolidating them.

Quoted 6 lines

docs/specs/ivshmem-2-device-spec.md line 52

Quoted 4 lines
docs/specs/ivshmem-2-device-spec.md new file +3 -0
+
+- The State Table must have the same size for all peers, must be large enough to
+  hold a state values of all peers, and must be read-only for the user.

"the state values", I guess.

Yes.

Quoted 14 lines

docs/specs/ivshmem-2-device-spec.md line 63

Quoted 12 lines
docs/specs/ivshmem-2-device-spec.md new file +11 -0
+
+- State register changes (explicit writes, peer resets) have to be propagated
+  to the other peers by updating the corresponding State Table entry and issuing
+  an interrupt to all other peers if they enabled reception.
+
+- Interrupts events triggered by a peer have to be delivered to the target peer,
+  provided the receiving side is valid and has enabled the reception.
+
+- All peers must have the same interrupt delivery features available, i.e. MSI-X
+  with the same maximum number of vectors on platforms supporting this
+  mechanism, otherwise INTx with one vector.

Use case for legacy INTx?

Platforms without support for MSI-X injection. Quite a few current ARM platforms still fall into this category, but they get less, indeed.

Quoted 3 lines

For what it's worth, we removed INTx support from ivshmem v1 in rev 1, QEMU 2.6.

Maybe worth to look into this again, specifically when we are done with ITS emulation on the Jailhouse side. Maybe it will be the same story as with our virtual PCI host controller there: After we had to add PCI logic for devices with real controller anyway, the virtual one was literally for free (not a single line of code added).

Quoted 10 lines

docs/specs/ivshmem-2-device-spec.md line 70

Quoted 8 lines
docs/specs/ivshmem-2-device-spec.md new file +7 -0
+
+
+Guest-side Programming Model
+----------------------------
+
+An IVSHMEM device appears as a PCI device to its users. Unless otherwise noted,
+it conforms to the PCI Local Bus Specification, Revision 3.0 As such, it is

Uh, is there anything in it that does *not* conform to this spec?

See below.

Quoted 54 lines
Quoted 54 lines
docs/specs/ivshmem-2-device-spec.md new file +53 -0
+discoverable via the PCI configuration space and provides a number of standard
+and custom PCI configuration registers.
+
+### Shared Memory Region Layout
+
+The shared memory region is divided into several sections.
+
+    +-----------------------------+   -
+    |                             |   :
+    | Output Section for peer n-1 |   : Output Section Size
+    |     (n = Maximum Peers)     |   :
+    +-----------------------------+   -
+    :                             :
+    :                             :
+    :                             :
+    +-----------------------------+   -
+    |                             |   :
+    |  Output Section for peer 1  |   : Output Section Size
+    |                             |   :
+    +-----------------------------+   -
+    |                             |   :
+    |  Output Section for peer 0  |   : Output Section Size
+    |                             |   :
+    +-----------------------------+   -
+    |                             |   :
+    |     Read/Write Section      |   : R/W Section Size
+    |                             |   :
+    +-----------------------------+   -
+    |                             |   :
+    |         State Table         |   : State Table Size
+    |                             |   :
+    +-----------------------------+   <-- Shared memory base address
+
+The first section consists of the mandatory State Table. Its size is defined by
+the State Table Size register and cannot be zero. This section is read-only for
+all peers.
+
+The second section consists of shared memory that is read/writable for all
+peers. Its size is defined by the R/W Section Size register. A size of zero is
+permitted.
+
+The third and following sections are unidirectional output sections, one for
+each peer. Their sizes are all identical. The size of a single output section is
+defined by the Output Section Size register. An output section is read/writable
+for the corresponding peer and read-only for all other peers. E.g., only the
+peer with ID 3 can write to the fourths output section, but all peers can read
+from this section.
+
+All sizes have to be rounded up to multiples of a mappable page in order to
+allow access control according to the section restrictions.
+
+### Configuration Space Registers
+
Quoted 8 lines

docs/specs/ivshmem-2-device-spec.md line 129

Quoted 6 lines
docs/specs/ivshmem-2-device-spec.md new file +6 -0
+#### Header Registers
+
+| Offset | Register               | Content                                              |
+|-------:|:-----------------------|:-----------------------------------------------------|
+|    00h | Vendor ID              | 1AF4h                                                |
+|    02h | Device ID              | 1110h                                                |

Same as ivshmem v1. Revision ID (offset 08h) disambiguates.

Will become 110Ah (Siemens) / 4106h (freshly reserved for a new ivshmem) in the next round.

Quoted 4 lines

docs/specs/ivshmem-2-device-spec.md line 130

Quoted 2 lines
docs/specs/ivshmem-2-device-spec.md new file +1 -0
+|    04h | Command Register       | 0000h on reset, implementing bits 1, 2, 10           |

What does "implementing bits ..." mean?

Ignore writes to the other bits.

Quoted 4 lines

docs/specs/ivshmem-2-device-spec.md line 131

Quoted 2 lines
docs/specs/ivshmem-2-device-spec.md new file +1 -0
+|    06h | Status Register        | 0010h, static value (bit 3 not implemented)          |

What does "bit 3 not implemented" mean?

Will stay at 0, no "pending INTx" information available.

Quoted 9 lines

docs/specs/ivshmem-2-device-spec.md line 132

Quoted 2 lines
docs/specs/ivshmem-2-device-spec.md new file +1 -0
+|    08h | Revision ID            | 02h                                                  |

ivshmem v1 is rev 0 before QEMU 2.6, rev 1 since. Rev 1 is backward-compatible to rev 0 for guest software. Is rev 2 intended to be backward-compatible, too?

You should probably explan how your v2 relates to v1 in more detail, possibly in its own top-level section.

With that new, separate device IT, this register will start at 0 in the next revision of this document.

Quoted 8 lines

docs/specs/ivshmem-2-device-spec.md line 135

Quoted 4 lines
docs/specs/ivshmem-2-device-spec.md new file +3 -0
+|    09h | Class Code, Interface  | Protocol Type bits 0-7, see [Protocols](#Protocols)  |
+|    0Ah | Class Code, Sub-Class  | Protocol Type bits 8-15, see [Protocols](#Protocols) |
+|    0Bh | Class Code, Base Class | FFh                                                  |

FFh means "device does not fit any defined class." For what it's worth, ivshmem v1 uses 05h "Memory Controller", with sub-class and interface 00h "RAM Controller".

I know. But using that scheme would needlessly wast 8 bits for protocol ID encoding. In turn, having the protocol encoded into the class code helps a lot with automatic driver probing on Linux e.g.

Quoted 5 lines

docs/specs/ivshmem-2-device-spec.md line 137

Quoted 3 lines
docs/specs/ivshmem-2-device-spec.md new file +2 -0
+|    0Eh | Header Type            | 00h                                                  |
+|    10h | BAR 0                  | MMIO or I/O register region                          |

Use case for I/O?

Faster VM exit handling. But, granted, that was not exploited so far in existing implementations. I just wanted to keep that door open.

Quoted 7 lines

For what it's worth, ivshmem v1 never supported I/O.

docs/specs/ivshmem-2-device-spec.md line 139

Quoted 2 lines
docs/specs/ivshmem-2-device-spec.md new file +2 -0
+|    14h | BAR 1                  | MSI-X region                                         |
+|    18h | BAR 2 (with BAR 3)     | optional: 64-bit shared memory region                |

What does "(with BAR 3)" mean?

64-bit resource regions always use two BARs, see PCI spec.

Quoted 5 lines

docs/specs/ivshmem-2-device-spec.md line 141

Quoted 3 lines
docs/specs/ivshmem-2-device-spec.md new file +2 -0
+|    2Ch | Subsystem Vendor ID    | same as Vendor ID, or provider-specific value        |
+|    2Eh | Subsystem ID           | same as Device ID, or provider-specific value        |

ivshmem v1 leaves these blank.

Same story as with virtio: This should encode the ID of the provider (hypervisor), also for the case we ever face deviating implementations, due to quirks of specification vagueness.

Quoted 10 lines

docs/specs/ivshmem-2-device-spec.md line 143

Quoted 3 lines
docs/specs/ivshmem-2-device-spec.md new file +2 -0
+|    34h | Capability Pointer     | First capability                                     |
+|    3Eh | Interrupt Pin          | 01h-04h, must be 00h if MSI-X is available           |

"If MSI-X is available"?

A PCIe device always provides MSI-X, and may additionally provide legacy INTx. A conventional PCI device may provide either or both. In any case, the Interrupt Pin register is zero when legacy INTx is not provided,

All known. This shall just "encourage" to provide more powerfull MSI-x when the platform provides that for other devices anyway. Only if the platform this device is injected to does not have such support, INTx shall be there.

Quoted 11 lines

docs/specs/ivshmem-2-device-spec.md line 150

Quoted 8 lines
docs/specs/ivshmem-2-device-spec.md new file +7 -0
+
+If BAR 2 is not present, the shared memory region is not relocatable by the
+user. In that case, the hypervisor has to implement the Base Address register in
+the vendor-specific capability.
+
+Other header registers may not be implemented. If not implemented, they return 0
+on read and ignore write accesses.

Is this an example of where the device does not conform to the PCI Local Bus Specification?

- INTx is an edge interrupt
- there is no INTx status bit
- there is no pending MSI-X bitmap

Quoted 13 lines

docs/specs/ivshmem-2-device-spec.md line 160

Quoted 11 lines
docs/specs/ivshmem-2-device-spec.md new file +10 -0
+
+#### Vendor Specific Capability (ID 09h)
+
+| Offset | Register            | Content                                        |
+|-------:|:--------------------|:-----------------------------------------------|
+|    00h | ID                  | 09h                                            |
+|    01h | Next Capability     | Pointer to next capability or 00h              |
+|    02h | Length              | 18h or 20h                                     |
+|    03h | Privileged Control  | Bit 0 (read/write): one-shot interrupt mode    |
+|        |                     | Bits 1-7: RsvdZ                                |

Please define RsvdZ somewhere, or use plainer language.

OK.

Quoted 8 lines

docs/specs/ivshmem-2-device-spec.md line 164

Quoted 5 lines
docs/specs/ivshmem-2-device-spec.md new file +4 -0
+|    04h | State Table Size    | 32-bit size of read-only State Table           |
+|    08h | R/W Section Size    | 64-bit size of common read/write section       |
+|    10h | Output Section Size | 64-bit size of unidirectional output sections  |
+|    18h | Base Address        | optional: 64-bit base address of shared memory |

Length is 20h when Base Adress is present, else 18h. Worth spelling that out?

Sure.

Quoted 8 lines

docs/specs/ivshmem-2-device-spec.md line 167

Quoted 4 lines
docs/specs/ivshmem-2-device-spec.md new file +3 -0
+
+All registers are read-only, except for bit 0 of the Privileged Control
+register.

Well, the other bits of that register are writable, they're just ignored.

Quoted 4 lines

For what it's worth, existing ivshmem-spec.txt uses the terms read-only, write-only, read/write and reserved bits rigorously: "Software should only access the registers as specified [...] Reserved bits should be ignored on read, and preserved on write."

Will change.

Quoted 13 lines

docs/specs/ivshmem-2-device-spec.md line 177

Quoted 11 lines
docs/specs/ivshmem-2-device-spec.md new file +10 -0
+
+When bit 0 in the Privileged Control register is set to 1, the device clears
+bit 0 in the Interrupt Control register on each interrupt delivery. This enables
+automatic interrupt throttling when re-enabling shall be performed by a
+scheduled unprivileged instance on the user side.
+
+If an IVSHMEM device does not support a relocatable shared memory region, BAR 2
+must not be implemented by the provider. Instead, the Base Address register has
+to be implemented to report the location of the shared memory region in the
+user's address space.

Rationale for not wanting to support relocatable shared memory?

Locked-down second stage page tables during runtime. We can even check-sum their content periodically when we do not allow guest to influence the shared memory guest location. Relevant for safety and possibly also security scenarios.

Quoted 7 lines

docs/specs/ivshmem-2-device-spec.md line 180

Quoted 4 lines
docs/specs/ivshmem-2-device-spec.md new file +3 -0
+
+A non-existing shared memory section has to report zero in its Section Size
+register.

This vendor-specific capability must always be present, I presume. Worth spelling out.

Ack.

Quoted 10 lines

docs/specs/ivshmem-2-device-spec.md line 186

Quoted 7 lines
docs/specs/ivshmem-2-device-spec.md new file +6 -0
+
+#### MSI-X Capability (ID 11h)
+
+On platforms supporting MSI-X, IVSHMEM has to provide interrupt delivery via
+this mechanism. In that case, the legacy INTx delivery mechanism is not
+available, and the Interrupt Pin configuration register returns 0.

I'm confused. Does that mean the device shall support either MSI-X or legacy INTx, but never both?

Correct, see above. There is no point in requesting both and requiring the hypervisor to deal with the guest switching between things during runtime.

Quoted 7 lines

docs/specs/ivshmem-2-device-spec.md line 189

Quoted 4 lines
docs/specs/ivshmem-2-device-spec.md new file +3 -0
+
+The IVSHMEM device has no notion of pending interrupts. Therefore, reading from
+the MSI-X Pending Bit Array will always return 0.

I guess this means a polling mode of operation is not possible. Correct?

Of course, polling remains possible, but the guest driver should rather obtain the interrupt-triggering state information from the much faster shared memory. There is simply no value in also mirroring that into interrupt registers for this particular device.

Quoted 16 lines

docs/specs/ivshmem-2-device-spec.md line 201

Quoted 13 lines
docs/specs/ivshmem-2-device-spec.md new file +12 -0
+
+The corresponding MSI-X MMIO region is configured via BAR 1.
+
+The MSI-X table size reported by the MSI-X capability structure is identical for
+all peers.
+
+### Register Region
+
+The register region may be implemented as MMIO or I/O.
+
+When implementing it as MMIO, the hypervisor has to ensure that the register
+region can be mapped as a single page into the address space of the user. Write

"can be mapped as a single page" depends on the host system, not the device.

When host system provides this device, it must ensure that the MMIO register region can be mapped by its guest as one page, without potential overlap with other resources.

Quoted 3 lines

For what it's worth, ivshmem v1 fixes the size of BAR0 to 256 bytes. Any particular reason to keep its size so loosely specified in v2?

UIO. The v1 device can't be handed out to untrusted UIO applications without intercepting the MMIO writes in the guest kernel in order to restrict them to the valid range. This device supports that.

Quoted 14 lines
Quoted 14 lines
docs/specs/ivshmem-2-device-spec.md new file +13 -0
+accesses to MMIO region offsets that are not backed by registers have to be
+ignored, read accesses have to return 0. This enables the user to hand out the
+complete region, along with the shared memory, to an unprivileged instance.
+
+The region location in the user's physical address space is configured via BAR
+0. The following table visualizes the region layout:
+
+| Offset | Register                                                            |
+|-------:|:--------------------------------------------------------------------|
+|    00h | ID                                                                  |
+|    04h | Maximum Peers                                                       |
+|    08h | Features                                                            |
+|    0Ch | Interrupt Control                                                   |
Quoted 8 lines

docs/specs/ivshmem-2-device-spec.md line 218

Quoted 4 lines
docs/specs/ivshmem-2-device-spec.md new file +4 -0
+|    10h | Doorbell                                                            |
+|    14h | State                                                               |
+
+All registers support only aligned 32-bit accesses.

Definitely not backwards compatible to rev 1. I figure that means v2 should use a different Vendor ID / Device ID, not just bump the Revsion ID.

Yep, see above.

Quoted 2 lines

No interrupt status register?

Yep, see above.

Quoted 14 lines

docs/specs/ivshmem-2-device-spec.md line 229

Quoted 12 lines
docs/specs/ivshmem-2-device-spec.md new file +11 -0
+
+#### ID Register (Offset 00h)
+
+Read-only register that reports the ID of the local device. It is unique for all
+of the connected devices and remains unchanged over their lifetime.
+
+#### Maximum Peers Register (Offset 04h)
+
+Read-only register that reports the maximum number of possible peers (including
+the local one). The supported range is between 2 and 65536 and remains constant
+over the lifetime of all peers.

Value 1 would be boring, but is it really impossible?

Likely not, didn't think about it so far. Would we gain anything by permitting this?

Quoted 26 lines
Quoted 26 lines
docs/specs/ivshmem-2-device-spec.md new file +25 -0
+
+#### Features Register (Offset 08h)
+
+Read-only register that reports features of the local device or the connected
+peers. Its content remains constant over the lifetime of all peers.
+
+| Bits | Content                                                               |
+|-----:|:----------------------------------------------------------------------|
+|    0 | 1: Synchronized shared memory base address                            |
+| 1-31 | RsvdZ                                                                 |
+
+If "synchronized shared memory base address" is reported (bit 0 is set), the
+shared memory region is mapped at the same address into the user address spaces
+of all connected peers. Thus, peers can use physical addresses as pointers when
+exchanging information via the shared memory. This feature flag is never set
+when the shared memory region is relocatable via BAR 2.
+
+#### Interrupt Control Register (Offset 0Ch)
+
+This read/write register controls the generation of interrupts whenever a peer
+writes to the Doorbell register or changes its state.
+
+| Bits | Content                                                               |
+|-----:|:----------------------------------------------------------------------|
+|    0 | 1: Enable interrupt generation                                        |
Quoted 8 lines

docs/specs/ivshmem-2-device-spec.md line 260

Quoted 6 lines
docs/specs/ivshmem-2-device-spec.md new file +6 -0
+| 1-31 | RsvdZ                                                                 |
+
+Note that bit 0 is reset to 0 on interrupt delivery if one-shot interrupt mode
+is enabled in the Enhanced Features register.
+
+The value of this register after device reset is 0.

I presume this applies only to legacy INTx. Correct?

Nope, this applies to both INTx and MSI-x. It's a building block for UIO devices, see one-shot interrupt mode. In that mode, bit 0 will be cleared by the device after each interrupt delivery, avoiding that the guest UIO driver needs to do that via a vmexit.

Quoted 2 lines

ivshmem v1 calls this Interrupt Mask.

As "set" does not mean "masked", I chose "enable" here.

Quoted 10 lines
Quoted 10 lines
docs/specs/ivshmem-2-device-spec.md new file +9 -0
+
+#### Doorbell Register (Offset 10h)
+
+Write-only register that triggers an interrupt vector in the target device if it
+is enabled there.
+
+| Bits  | Content                                                              |
+|------:|:---------------------------------------------------------------------|
+|  0-15 | Vector number                                                        |
Quoted 8 lines

docs/specs/ivshmem-2-device-spec.md line 275

Quoted 6 lines
docs/specs/ivshmem-2-device-spec.md new file +6 -0
+| 16-31 | Target ID                                                            |
+
+Writing a vector number that is not enabled by the target has no effect. The
+peers can derive the number of available vectors from their own device
+capabilities and are expected to define or negotiate the used ones via the
+selected protocol.

That's because all peers use the same number of vectors. Worth spelling out?

I thought I did that somewhere. Will check.

Quoted 13 lines

docs/specs/ivshmem-2-device-spec.md line 285

Quoted 11 lines
docs/specs/ivshmem-2-device-spec.md new file +10 -0
+
+Addressing a non-existing or inactive target has no effect. Peers can identify
+active targets via the State Table.
+
+The behavior on reading from this register is undefined.
+
+#### State Register (Offset 14h)
+
+Read/write register that defines the state of the local device. Writing to this
+register sets the state and triggers interrupt vector 0 on the remote device if

"the remote device"? Do you mean all peers?

Historic singular (an earlier revision only supported 2 peers). Will fix.

Quoted 3 lines

"interrupt vector 0" assumes MSI-X. With legacy INTx, it simply triggers an interrupt.

Right, INTx only has vector 0.

Quoted 3 lines

How can guest software distinguish between "interrupt due to remote state change" and "interrupt due to doorbell"?

By checking those states. Both are in shared memory, thus quickly readable.

Quoted 3 lines

Things become so much easier when you ditch INTx: reserve vector 0 for state change, done.

That aspect does not make things complicated with INTx, but I agree that we should look at its "TCO" again.

Quoted 8 lines

docs/specs/ivshmem-2-device-spec.md line 289

Quoted 5 lines
docs/specs/ivshmem-2-device-spec.md new file +4 -0
+the written state value differs from the previous one. The user of the remote
+device can read the value written to this register from the State Table.
+
+The value of this register after device reset is 0.

I guess the meaning of state values depends on the protocol type. Correct?

Yes, except for value 0 which means "reset state". The protocols are free to define all other values according to their needs.

Quoted 39 lines
Quoted 39 lines
docs/specs/ivshmem-2-device-spec.md new file +38 -0
+
+### State Table
+
+The State Table is a read-only section at the beginning of the shared memory
+region. It contains a 32-bit state value for each of the peers. Locating the
+table in shared memory allows fast checking of remote states without register
+accesses.
+
+The table is updated on each state change of a peers. Whenever a user of an
+IVSHMEM device writes a value to the Local State register, this value is copied
+into the corresponding entry of the State Table. When a IVSHMEM device is reset
+or disconnected from the other peers, zero is written into the corresponding
+table entry. The initial content of the table is all zeros.
+
+    +--------------------------------+
+    | 32-bit state value of peer n-1 |
+    +--------------------------------+
+    :                                :
+    +--------------------------------+
+    | 32-bit state value of peer 1   |
+    +--------------------------------+
+    | 32-bit state value of peer 0   |
+    +--------------------------------+ <-- Shared memory base address
+
+
+Protocols
+---------
+
+The IVSHMEM device shall support the peers of a connection in agreeing on the
+protocol used over the shared memory devices. For that purpose, the interface
+byte (offset 09h) and the sub-class byte (offset 0Ah) of the Class Code register
+encodes a 16-bit protocol type for the users. The following type values are
+defined:
+
+| Protocol Type | Description                                                  |
+|--------------:|:-------------------------------------------------------------|
+|         0000h | Undefined type                                               |
+|         0001h | Virtual peer-to-peer Ethernet                                |
Quoted 8 lines

docs/specs/ivshmem-2-device-spec.md line 333

Quoted 6 lines
docs/specs/ivshmem-2-device-spec.md new file +6 -0
+|   0002h-3FFFh | Reserved                                                     |
+|   4000h-7FFFh | User-defined protocols                                       |
+|   8000h-BFFFh | Virtio over Shared Memory, front-end peer                    |
+|   C000h-FFFFh | Virtio over Shared Memory, back-end peer                     |
+
+Details of the protocols are not in the scope of this specification.

Are you sure this use of PCI class code is kosher?

Well, we are in the 0xff range. To my understanding, there are no constraints on what you put in the other bits for that class.

Quoted 3 lines

Final request: please break your lines around column 70 for readability.

Sure, can do.

Thanks a lot,
Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
#16 lore raw

Re: [RFC][PATCH 2/3] docs/specs: Add specification of ivshmem device revision 2

Quoted 70 lines · Jan Kiszka
Quoted 2 lines · Markus Armbruster

This has been on the list for more than three weeks already. I apologize for the delay.

No problem! Your feedback is highly appreciated.

Quoted 55 lines
Quoted 52 lines · Jan Kiszka

From: Jan Kiszka <[email protected]>

This imports the ivshmem v2 specification draft from Jailhouse. Its final home is to be decided, this shall just simplify the review process at this stage.

Note that specifically the Features register (offset 08h) is still under consideration. In particular, its bit 0 seems useless now as its benefit to guests, specifically when they want to be portable, is close to zero. Maybe the register should still be kept, with all bits RsvdZ, for easier extensibility.

The rest appears now rather mature and reasonably implementable, as proven also by a version for Jailhouse.

---
 docs/specs/ivshmem-2-device-spec.md | 333 ++++++++++++++++++++++++++++++++++++
 1 file changed, 333 insertions(+)
 create mode 100644 docs/specs/ivshmem-2-device-spec.md
docs/specs/ivshmem-2-device-spec.md new file +25 -0
new file mode 100644
index 0000000000..98cfde585a
@@ -0,0 +1,333 @@
+IVSHMEM Device Specification
+============================
+
+** NOTE: THIS IS WORK-IN-PROGRESS, NOT YET A STABLE INTERFACE SPECIFICATION! **
+
+The Inter-VM Shared Memory device provides the following features to its users:
+
+- Interconnection between up to 65536 peers
+
+- Multi-purpose shared memory region
+
+    - common read/writable section
+
+    - unidirectional sections that are read/writable for one peer and only
+      readable for the others
+
+    - section with peer states
+
+- Event signaling via interrupt to remote sides
+
+- Support for life-cycle management via state value exchange and interrupt
+  notification on changes, backed by a shared memory section
+
+- Free choice of protocol to be used on top
+
Quoted 8 lines

docs/specs/ivshmem-2-device-spec.md line 30

Quoted 5 lines
docs/specs/ivshmem-2-device-spec.md new file +5 -0
+- Protocol type declaration
+
+- Unprivileged access to memory-mapped or I/O registers feasible
+
+- Discoverable and configurable via standard PCI mechanisms

Stating requirements is much appreciated. Design rationale would be even better :)

Quoted 4 lines

The idea is to avoid having to model also a platform device with platform discovery mechanism. Based on our experiences with providing a generic virtual PCI host controller, such a complication of specification and implementation is simply not needed.

Work that into the document please.

Quoted 10 lines
Quoted 6 lines

As pointed out many times, shared memory is not a solution to any communication problem, it's merely a building block for building such
solutions: you invariably have to layer some protocol on top. In your KVM Forum talk, you mention layering virtio on top. Makes sense to me. But why does *this* virtio transport have to be an independent device? Other transports aren't.

This device is not only a virtio transport. As you correctly point out, it is a foundation for communication, not the full solution by itself.

Quoted 6 lines

Its goal is to define the minimal piece that a hypervisor has to provide to its guests in order to enable them building full communication solutions of their preferred flavor on top. It does not want to deal with those details, may they be virtio, may they be something much simpler, much older or much more sophisticated (or complicated). All that shall not be the business of a hypervisor.

Likewise.

Quoted 16 lines
Quoted 15 lines

Now let me indulge in spec nitpicking :)

docs/specs/ivshmem-2-device-spec.md line 41

Quoted 11 lines
docs/specs/ivshmem-2-device-spec.md new file +11 -0
+
+
+Hypervisor Model
+----------------
+
+In order to provide a consistent link between peers, all connected instances of
+IVSHMEM devices need to be configured, created and run by the hypervisor
+according to the following requirements:
+
+- The instances of the device need to be accessible via PCI programming
+  interfaces on all sides.

What does that mean?

Quoted 1 line

"From a guest point of view, this shall be a PCI device."

Clearer.

Quoted 15 lines
Quoted 5 lines

docs/specs/ivshmem-2-device-spec.md line 44

Quoted 3 lines
docs/specs/ivshmem-2-device-spec.md new file +3 -0
+
+- The read/write shared memory section has to be of the same size for all
+  peers and, if non-zero, has to reflect the same memory content for them.

Isn't "same memory content" redundant with "shared memory"?

Probably.

Quoted 7 lines
Quoted 6 lines
docs/specs/ivshmem-2-device-spec.md new file +5 -0
+
+- If output sections are present (non-zero section size), there must be one
+  reserved for each peer with exclusive write access. All output sections
+  must have the same size and must be readable for all peers. They have to
+  reflect the same memory content for all peers.
Quoted 3 lines
Quoted 1 line

Are these the "unidirectional sections" mentioned previously?

Yep. The terms have some history, maybe worth consolidating them.

Yes, please. Consistent terminology helps.

Quoted 27 lines
Quoted 5 lines

docs/specs/ivshmem-2-device-spec.md line 52

Quoted 3 lines
docs/specs/ivshmem-2-device-spec.md new file +3 -0
+
+- The State Table must have the same size for all peers, must be large enough to
+  hold a state values of all peers, and must be read-only for the user.

"the state values", I guess.

Yes.

Quoted 14 lines

docs/specs/ivshmem-2-device-spec.md line 63

Quoted 12 lines
docs/specs/ivshmem-2-device-spec.md new file +11 -0
+
+- State register changes (explicit writes, peer resets) have to be propagated
+  to the other peers by updating the corresponding State Table entry and issuing
+  an interrupt to all other peers if they enabled reception.
+
+- Interrupts events triggered by a peer have to be delivered to the target peer,
+  provided the receiving side is valid and has enabled the reception.
+
+- All peers must have the same interrupt delivery features available, i.e. MSI-X
+  with the same maximum number of vectors on platforms supporting this
+  mechanism, otherwise INTx with one vector.

Use case for legacy INTx?

Platforms without support for MSI-X injection. Quite a few current ARM platforms still fall into this category, but they get less, indeed.

Quoted 1 line
Quoted 8 lines
Quoted 2 lines

For what it's worth, we removed INTx support from ivshmem v1 in rev 1, QEMU 2.6.

Maybe worth to look into this again, specifically when we are done with ITS emulation on the Jailhouse side. Maybe it will be the same story as with our virtual PCI host controller there: After we had to add PCI logic for devices with real controller anyway, the virtual one was literally for free (not a single line of code added).

I'd like to encourage you to consider this once more. Dropping legacy INTx support will simplify this document quite a bit.

Quoted 11 lines
Quoted 9 lines

docs/specs/ivshmem-2-device-spec.md line 70

Quoted 7 lines
docs/specs/ivshmem-2-device-spec.md new file +7 -0
+
+
+Guest-side Programming Model
+----------------------------
+
+An IVSHMEM device appears as a PCI device to its users. Unless otherwise noted,
+it conforms to the PCI Local Bus Specification, Revision 3.0 As such, it is

Uh, is there anything in it that does *not* conform to this spec?

See below.

I'm not opposed to reasoned deviations. I'd like to see the "unless otherwise noted" promise honored with explicit notes, though.

Quoted 63 lines
Quoted 54 lines
Quoted 54 lines
docs/specs/ivshmem-2-device-spec.md new file +53 -0
+discoverable via the PCI configuration space and provides a number of standard
+and custom PCI configuration registers.
+
+### Shared Memory Region Layout
+
+The shared memory region is divided into several sections.
+
+    +-----------------------------+   -
+    |                             |   :
+    | Output Section for peer n-1 |   : Output Section Size
+    |     (n = Maximum Peers)     |   :
+    +-----------------------------+   -
+    :                             :
+    :                             :
+    :                             :
+    +-----------------------------+   -
+    |                             |   :
+    |  Output Section for peer 1  |   : Output Section Size
+    |                             |   :
+    +-----------------------------+   -
+    |                             |   :
+    |  Output Section for peer 0  |   : Output Section Size
+    |                             |   :
+    +-----------------------------+   -
+    |                             |   :
+    |     Read/Write Section      |   : R/W Section Size
+    |                             |   :
+    +-----------------------------+   -
+    |                             |   :
+    |         State Table         |   : State Table Size
+    |                             |   :
+    +-----------------------------+   <-- Shared memory base address
+
+The first section consists of the mandatory State Table. Its size is defined by
+the State Table Size register and cannot be zero. This section is read-only for
+all peers.
+
+The second section consists of shared memory that is read/writable for all
+peers. Its size is defined by the R/W Section Size register. A size of zero is
+permitted.
+
+The third and following sections are unidirectional output sections, one for
+each peer. Their sizes are all identical. The size of a single output section is
+defined by the Output Section Size register. An output section is read/writable
+for the corresponding peer and read-only for all other peers. E.g., only the
+peer with ID 3 can write to the fourths output section, but all peers can read
+from this section.
+
+All sizes have to be rounded up to multiples of a mappable page in order to
+allow access control according to the section restrictions.
+
+### Configuration Space Registers
+
Quoted 8 lines

docs/specs/ivshmem-2-device-spec.md line 129

Quoted 6 lines
docs/specs/ivshmem-2-device-spec.md new file +6 -0
+#### Header Registers
+
+| Offset | Register               | Content                                              |
+|-------:|:-----------------------|:-----------------------------------------------------|
+|    00h | Vendor ID              | 1AF4h                                                |
+|    02h | Device ID              | 1110h                                                |

Same as ivshmem v1. Revision ID (offset 08h) disambiguates.

Quoted 2 lines

Will become 110Ah (Siemens) / 4106h (freshly reserved for a new ivshmem) in the next round.

Better.

Quoted 20 lines
Quoted 3 lines

docs/specs/ivshmem-2-device-spec.md line 130

Quoted 1 line
docs/specs/ivshmem-2-device-spec.md new file +1 -0
+|    04h | Command Register       | 0000h on reset, implementing bits 1, 2, 10           |

What does "implementing bits ..." mean?

Ignore writes to the other bits.

Quoted 4 lines

docs/specs/ivshmem-2-device-spec.md line 131

Quoted 2 lines
docs/specs/ivshmem-2-device-spec.md new file +1 -0
+|    06h | Status Register        | 0010h, static value (bit 3 not implemented)          |

What does "bit 3 not implemented" mean?

Will stay at 0, no "pending INTx" information available.

Quoted 7 lines

docs/specs/ivshmem-2-device-spec.md line 132

Quoted 2 lines
docs/specs/ivshmem-2-device-spec.md new file +1 -0
+|    08h | Revision ID            | 02h                                                  |

ivshmem v1 is rev 0 before QEMU 2.6, rev 1 since. Rev 1 is backward-compatible to rev 0 for guest software. Is rev 2 intended to be backward-compatible, too?

Quoted 5 lines
Quoted 2 lines

You should probably explan how your v2 relates to v1 in more detail, possibly in its own top-level section.

With that new, separate device IT, this register will start at 0 in the next revision of this document.

By making the device clearly different that way, you don't need to discuss differences in detail. In particular, backward compatibility can be treated as non-issue. But I'd still like to see a higher level discussion of the differences. In your KVM Forum presentation, you covered v1's deficits (slide 5), and v2's key differences (slide 6).

docs/specs/ivshmem-spec.txt documents numerous v1 deficits. Referring to that document might be helpful.

Quoted 11 lines
Quoted 7 lines

docs/specs/ivshmem-2-device-spec.md line 135

Quoted 3 lines
docs/specs/ivshmem-2-device-spec.md new file +3 -0
+|    09h | Class Code, Interface  | Protocol Type bits 0-7, see [Protocols](#Protocols)  |
+|    0Ah | Class Code, Sub-Class  | Protocol Type bits 8-15, see [Protocols](#Protocols) |
+|    0Bh | Class Code, Base Class | FFh                                                  |

FFh means "device does not fit any defined class." For what it's worth, ivshmem v1 uses 05h "Memory Controller", with sub-class and interface 00h "RAM Controller".

I know. But using that scheme would needlessly wast 8 bits for protocol ID encoding. In turn, having the protocol encoded into the class code helps a lot with automatic driver probing on Linux e.g.

Less of an issue (or none-issue?) now that you use a fresh Vendor ID / Device ID pair.

Quoted 8 lines
Quoted 5 lines

docs/specs/ivshmem-2-device-spec.md line 137

Quoted 3 lines
docs/specs/ivshmem-2-device-spec.md new file +2 -0
+|    0Eh | Header Type            | 00h                                                  |
+|    10h | BAR 0                  | MMIO or I/O register region                          |

Use case for I/O?

Faster VM exit handling. But, granted, that was not exploited so far in existing implementations. I just wanted to keep that door open.

Write that down, please :)

*Why* things are designed a certain way is often as useful to know as the actual design.

Quoted 8 lines
Quoted 6 lines

For what it's worth, ivshmem v1 never supported I/O.

docs/specs/ivshmem-2-device-spec.md line 139

Quoted 2 lines
docs/specs/ivshmem-2-device-spec.md new file +2 -0
+|    14h | BAR 1                  | MSI-X region                                         |
+|    18h | BAR 2 (with BAR 3)     | optional: 64-bit shared memory region                |

What does "(with BAR 3)" mean?

64-bit resource regions always use two BARs, see PCI spec.

Right. Perhaps we can convey that more clearly to readers who forgot (or never knew) this PCI detail, but I admit I'm drawing blanks right now.

Quoted 8 lines
Quoted 4 lines

docs/specs/ivshmem-2-device-spec.md line 141

Quoted 2 lines
docs/specs/ivshmem-2-device-spec.md new file +2 -0
+|    2Ch | Subsystem Vendor ID    | same as Vendor ID, or provider-specific value        |
+|    2Eh | Subsystem ID           | same as Device ID, or provider-specific value        |

ivshmem v1 leaves these blank.

Same story as with virtio: This should encode the ID of the provider (hypervisor), also for the case we ever face deviating implementations, due to quirks of specification vagueness.

This intended use makes sense. Write it down, please.

Quoted 10 lines
Quoted 9 lines

docs/specs/ivshmem-2-device-spec.md line 143

Quoted 2 lines
docs/specs/ivshmem-2-device-spec.md new file +2 -0
+|    34h | Capability Pointer     | First capability                                     |
+|    3Eh | Interrupt Pin          | 01h-04h, must be 00h if MSI-X is available           |

"If MSI-X is available"?

A PCIe device always provides MSI-X, and may additionally provide legacy INTx. A conventional PCI device may provide either or both. In any case, the Interrupt Pin register is zero when legacy INTx is not provided,

Quoted 4 lines

All known. This shall just "encourage" to provide more powerfull MSI-x when the platform provides that for other devices anyway. Only if the platform this device is injected to does not have such support, INTx shall be there.

In case you keep legacy INTx support: I'd phrase this as a (strong) recommendation, not as (vague) specification.

Quoted 11 lines
Quoted 10 lines

docs/specs/ivshmem-2-device-spec.md line 150

Quoted 7 lines
docs/specs/ivshmem-2-device-spec.md new file +7 -0
+
+If BAR 2 is not present, the shared memory region is not relocatable by the
+user. In that case, the hypervisor has to implement the Base Address register in
+the vendor-specific capability.
+
+Other header registers may not be implemented. If not implemented, they return 0
+on read and ignore write accesses.

Is this an example of where the device does not conform to the PCI Local Bus Specification?

Quoted 3 lines

- INTx is an edge interrupt
- there is no INTx status bit
- there is no pending MSI-X bitmap

Two more reasons to ditch legacy INTx.

For each of these, I'd like to see an clear notice of deviation from the PCI spec, and rationale (why are deviating).

Quoted 34 lines
Quoted 4 lines
Quoted 4 lines
docs/specs/ivshmem-2-device-spec.md new file +4 -0
+
+#### Vendor Specific Capability (ID 09h)
+
+| Offset | Register            | Content                                        |
Quoted 8 lines

docs/specs/ivshmem-2-device-spec.md line 160

Quoted 6 lines
docs/specs/ivshmem-2-device-spec.md new file +6 -0
+|-------:|:--------------------|:-----------------------------------------------|
+|    00h | ID                  | 09h                                            |
+|    01h | Next Capability     | Pointer to next capability or 00h              |
+|    02h | Length              | 18h or 20h                                     |
+|    03h | Privileged Control  | Bit 0 (read/write): one-shot interrupt mode    |
+|        |                     | Bits 1-7: RsvdZ                                |

Please define RsvdZ somewhere, or use plainer language.

OK.

Quoted 8 lines

docs/specs/ivshmem-2-device-spec.md line 164

Quoted 5 lines
docs/specs/ivshmem-2-device-spec.md new file +4 -0
+|    04h | State Table Size    | 32-bit size of read-only State Table           |
+|    08h | R/W Section Size    | 64-bit size of common read/write section       |
+|    10h | Output Section Size | 64-bit size of unidirectional output sections  |
+|    18h | Base Address        | optional: 64-bit base address of shared memory |

Length is 20h when Base Adress is present, else 18h. Worth spelling that out?

Sure.

Quoted 8 lines

docs/specs/ivshmem-2-device-spec.md line 167

Quoted 4 lines
docs/specs/ivshmem-2-device-spec.md new file +3 -0
+
+All registers are read-only, except for bit 0 of the Privileged Control
+register.

Well, the other bits of that register are writable, they're just ignored.

Quoted 6 lines
Quoted 4 lines

For what it's worth, existing ivshmem-spec.txt uses the terms read-only, write-only, read/write and reserved bits rigorously: "Software should only access the registers as specified [...] Reserved bits should be ignored on read, and preserved on write."

Will change.

Could also help with documenting Command and Status register more clearly.

Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
docs/specs/ivshmem-2-device-spec.md new file +4 -0
+
+When bit 0 in the Privileged Control register is set to 1, the device clears
+bit 0 in the Interrupt Control register on each interrupt delivery. This enables
+automatic interrupt throttling when re-enabling shall be performed by a
Quoted 8 lines

docs/specs/ivshmem-2-device-spec.md line 177

Quoted 6 lines
docs/specs/ivshmem-2-device-spec.md new file +6 -0
+scheduled unprivileged instance on the user side.
+
+If an IVSHMEM device does not support a relocatable shared memory region, BAR 2
+must not be implemented by the provider. Instead, the Base Address register has
+to be implemented to report the location of the shared memory region in the
+user's address space.

Rationale for not wanting to support relocatable shared memory?

Quoted 4 lines

Locked-down second stage page tables during runtime. We can even check-sum their content periodically when we do not allow guest to influence the shared memory guest location. Relevant for safety and possibly also security scenarios.

Write that down, please.

Quoted 20 lines
Quoted 6 lines

docs/specs/ivshmem-2-device-spec.md line 180

Quoted 3 lines
docs/specs/ivshmem-2-device-spec.md new file +3 -0
+
+A non-existing shared memory section has to report zero in its Section Size
+register.

This vendor-specific capability must always be present, I presume. Worth spelling out.

Ack.

Quoted 10 lines

docs/specs/ivshmem-2-device-spec.md line 186

Quoted 7 lines
docs/specs/ivshmem-2-device-spec.md new file +6 -0
+
+#### MSI-X Capability (ID 11h)
+
+On platforms supporting MSI-X, IVSHMEM has to provide interrupt delivery via
+this mechanism. In that case, the legacy INTx delivery mechanism is not
+available, and the Interrupt Pin configuration register returns 0.

I'm confused. Does that mean the device shall support either MSI-X or legacy INTx, but never both?

Quoted 3 lines

Correct, see above. There is no point in requesting both and requiring the hypervisor to deal with the guest switching between things during runtime.

The PCI spec is cool with devices that only implement one of the two. But shipping one kind of device come in two such different flavours while still sharing the same Vendor ID, Device ID is something that would get a hardware vendor yelled at, and rightfully so.

Now, this isn't physical hardware. And you might have a high tolerance for getting yelled at.

Still, yet another reason to ditch legacy INTx.

If not, I'd advocate for a more prominent "caveat emptor" sticker on this particular, unusual feature.

Quoted 11 lines
Quoted 6 lines

docs/specs/ivshmem-2-device-spec.md line 189

Quoted 3 lines
docs/specs/ivshmem-2-device-spec.md new file +3 -0
+
+The IVSHMEM device has no notion of pending interrupts. Therefore, reading from
+the MSI-X Pending Bit Array will always return 0.

I guess this means a polling mode of operation is not possible. Correct?

Of course, polling remains possible, but the guest driver should rather obtain the interrupt-triggering state information from the much faster shared memory. There is simply no value in also mirroring that into interrupt registers for this particular device.

I see. Mirroring it just for spec compliance may not be worthwhile. Please explain this in this document.

Quoted 21 lines
Quoted 7 lines
Quoted 7 lines
docs/specs/ivshmem-2-device-spec.md new file +7 -0
+
+The corresponding MSI-X MMIO region is configured via BAR 1.
+
+The MSI-X table size reported by the MSI-X capability structure is identical for
+all peers.
+
+### Register Region
Quoted 8 lines

docs/specs/ivshmem-2-device-spec.md line 201

Quoted 5 lines
docs/specs/ivshmem-2-device-spec.md new file +5 -0
+
+The register region may be implemented as MMIO or I/O.
+
+When implementing it as MMIO, the hypervisor has to ensure that the register
+region can be mapped as a single page into the address space of the user. Write

"can be mapped as a single page" depends on the host system, not the device.

When host system provides this device, it must ensure that the MMIO register region can be mapped by its guest as one page, without potential overlap with other resources.

Quoted 1 line
Quoted 6 lines
Quoted 2 lines

For what it's worth, ivshmem v1 fixes the size of BAR0 to 256 bytes. Any particular reason to keep its size so loosely specified in v2?

UIO. The v1 device can't be handed out to untrusted UIO applications without intercepting the MMIO writes in the guest kernel in order to restrict them to the valid range. This device supports that.

Sad. BARs changing size while Vendor ID, Device ID stays the same is unusual, but the other BARs *have* be that way, so this one hardly makes it more unusual.

Please write this down.

Quoted 22 lines
Quoted 13 lines
Quoted 13 lines
docs/specs/ivshmem-2-device-spec.md new file +13 -0
+accesses to MMIO region offsets that are not backed by registers have to be
+ignored, read accesses have to return 0. This enables the user to hand out the
+complete region, along with the shared memory, to an unprivileged instance.
+
+The region location in the user's physical address space is configured via BAR
+0. The following table visualizes the region layout:
+
+| Offset | Register                                                            |
+|-------:|:--------------------------------------------------------------------|
+|    00h | ID                                                                  |
+|    04h | Maximum Peers                                                       |
+|    08h | Features                                                            |
+|    0Ch | Interrupt Control                                                   |
Quoted 8 lines

docs/specs/ivshmem-2-device-spec.md line 218

Quoted 4 lines
docs/specs/ivshmem-2-device-spec.md new file +4 -0
+|    10h | Doorbell                                                            |
+|    14h | State                                                               |
+
+All registers support only aligned 32-bit accesses.

Definitely not backwards compatible to rev 1. I figure that means v2 should use a different Vendor ID / Device ID, not just bump the Revsion ID.

Quoted 6 lines

Yep, see above.

Quoted 2 lines

No interrupt status register?

Yep, see above.

Where? Is it "guest driver should rather obtain the interrupt-triggering state information from the much faster shared memory"?

Quoted 15 lines
Quoted 14 lines

docs/specs/ivshmem-2-device-spec.md line 229

Quoted 12 lines
docs/specs/ivshmem-2-device-spec.md new file +11 -0
+
+#### ID Register (Offset 00h)
+
+Read-only register that reports the ID of the local device. It is unique for all
+of the connected devices and remains unchanged over their lifetime.
+
+#### Maximum Peers Register (Offset 04h)
+
+Read-only register that reports the maximum number of possible peers (including
+the local one). The supported range is between 2 and 65536 and remains constant
+over the lifetime of all peers.

Value 1 would be boring, but is it really impossible?

Quoted 2 lines

Likely not, didn't think about it so far. Would we gain anything by permitting this?

Running a lone guest with an ivshmem device wouldn't be out of spec.

Never stipulate an error you don't want to detect and handle :)

Quoted 34 lines
Quoted 25 lines
Quoted 25 lines
docs/specs/ivshmem-2-device-spec.md new file +25 -0
+
+#### Features Register (Offset 08h)
+
+Read-only register that reports features of the local device or the connected
+peers. Its content remains constant over the lifetime of all peers.
+
+| Bits | Content                                                               |
+|-----:|:----------------------------------------------------------------------|
+|    0 | 1: Synchronized shared memory base address                            |
+| 1-31 | RsvdZ                                                                 |
+
+If "synchronized shared memory base address" is reported (bit 0 is set), the
+shared memory region is mapped at the same address into the user address spaces
+of all connected peers. Thus, peers can use physical addresses as pointers when
+exchanging information via the shared memory. This feature flag is never set
+when the shared memory region is relocatable via BAR 2.
+
+#### Interrupt Control Register (Offset 0Ch)
+
+This read/write register controls the generation of interrupts whenever a peer
+writes to the Doorbell register or changes its state.
+
+| Bits | Content                                                               |
+|-----:|:----------------------------------------------------------------------|
+|    0 | 1: Enable interrupt generation                                        |
Quoted 8 lines

docs/specs/ivshmem-2-device-spec.md line 260

Quoted 6 lines
docs/specs/ivshmem-2-device-spec.md new file +6 -0
+| 1-31 | RsvdZ                                                                 |
+
+Note that bit 0 is reset to 0 on interrupt delivery if one-shot interrupt mode
+is enabled in the Enhanced Features register.
+
+The value of this register after device reset is 0.

I presume this applies only to legacy INTx. Correct?

Quoted 4 lines

Nope, this applies to both INTx and MSI-x. It's a building block for UIO devices, see one-shot interrupt mode. In that mode, bit 0 will be cleared by the device after each interrupt delivery, avoiding that the guest UIO driver needs to do that via a vmexit.

How does this play with MSI-X per-vector masking?

Quoted 3 lines
Quoted 1 line

ivshmem v1 calls this Interrupt Mask.

As "set" does not mean "masked", I chose "enable" here.

I'm confused. Can you explain the difference?

Quoted 34 lines
Quoted 9 lines
Quoted 9 lines
docs/specs/ivshmem-2-device-spec.md new file +9 -0
+
+#### Doorbell Register (Offset 10h)
+
+Write-only register that triggers an interrupt vector in the target device if it
+is enabled there.
+
+| Bits  | Content                                                              |
+|------:|:---------------------------------------------------------------------|
+|  0-15 | Vector number                                                        |
Quoted 8 lines

docs/specs/ivshmem-2-device-spec.md line 275

Quoted 6 lines
docs/specs/ivshmem-2-device-spec.md new file +6 -0
+| 16-31 | Target ID                                                            |
+
+Writing a vector number that is not enabled by the target has no effect. The
+peers can derive the number of available vectors from their own device
+capabilities and are expected to define or negotiate the used ones via the
+selected protocol.

That's because all peers use the same number of vectors. Worth spelling out?

I thought I did that somewhere. Will check.

Quoted 13 lines

docs/specs/ivshmem-2-device-spec.md line 285

Quoted 11 lines
docs/specs/ivshmem-2-device-spec.md new file +10 -0
+
+Addressing a non-existing or inactive target has no effect. Peers can identify
+active targets via the State Table.
+
+The behavior on reading from this register is undefined.
+
+#### State Register (Offset 14h)
+
+Read/write register that defines the state of the local device. Writing to this
+register sets the state and triggers interrupt vector 0 on the remote device if

"the remote device"? Do you mean all peers?

Quoted 7 lines

Historic singular (an earlier revision only supported 2 peers). Will fix.

Quoted 3 lines

"interrupt vector 0" assumes MSI-X. With legacy INTx, it simply triggers an interrupt.

Right, INTx only has vector 0.

INTx has no concept of vectors, it uses pins. Suggest to be a bit more verbose to avoid confusion.

Quoted 4 lines
Quoted 2 lines

How can guest software distinguish between "interrupt due to remote state change" and "interrupt due to doorbell"?

By checking those states. Both are in shared memory, thus quickly readable.

I think that's worth pointing out somewhere.

Quoted 70 lines
Quoted 2 lines

Things become so much easier when you ditch INTx: reserve vector 0 for state change, done.

That aspect does not make things complicated with INTx, but I agree that we should look at its "TCO" again.

Quoted 8 lines

docs/specs/ivshmem-2-device-spec.md line 289

Quoted 5 lines
docs/specs/ivshmem-2-device-spec.md new file +4 -0
+the written state value differs from the previous one. The user of the remote
+device can read the value written to this register from the State Table.
+
+The value of this register after device reset is 0.

I guess the meaning of state values depends on the protocol type. Correct?

Yes, except for value 0 which means "reset state". The protocols are free to define all other values according to their needs.

Quoted 39 lines
Quoted 39 lines
docs/specs/ivshmem-2-device-spec.md new file +38 -0
+
+### State Table
+
+The State Table is a read-only section at the beginning of the shared memory
+region. It contains a 32-bit state value for each of the peers. Locating the
+table in shared memory allows fast checking of remote states without register
+accesses.
+
+The table is updated on each state change of a peers. Whenever a user of an
+IVSHMEM device writes a value to the Local State register, this value is copied
+into the corresponding entry of the State Table. When a IVSHMEM device is reset
+or disconnected from the other peers, zero is written into the corresponding
+table entry. The initial content of the table is all zeros.
+
+    +--------------------------------+
+    | 32-bit state value of peer n-1 |
+    +--------------------------------+
+    :                                :
+    +--------------------------------+
+    | 32-bit state value of peer 1   |
+    +--------------------------------+
+    | 32-bit state value of peer 0   |
+    +--------------------------------+ <-- Shared memory base address
+
+
+Protocols
+---------
+
+The IVSHMEM device shall support the peers of a connection in agreeing on the
+protocol used over the shared memory devices. For that purpose, the interface
+byte (offset 09h) and the sub-class byte (offset 0Ah) of the Class Code register
+encodes a 16-bit protocol type for the users. The following type values are
+defined:
+
+| Protocol Type | Description                                                  |
+|--------------:|:-------------------------------------------------------------|
+|         0000h | Undefined type                                               |
+|         0001h | Virtual peer-to-peer Ethernet                                |
Quoted 8 lines

docs/specs/ivshmem-2-device-spec.md line 333

Quoted 6 lines
docs/specs/ivshmem-2-device-spec.md new file +6 -0
+|   0002h-3FFFh | Reserved                                                     |
+|   4000h-7FFFh | User-defined protocols                                       |
+|   8000h-BFFFh | Virtio over Shared Memory, front-end peer                    |
+|   C000h-FFFFh | Virtio over Shared Memory, back-end peer                     |
+
+Details of the protocols are not in the scope of this specification.

Are you sure this use of PCI class code is kosher?

Well, we are in the 0xff range. To my understanding, there are no constraints on what you put in the other bits for that class.

Quoted 1 line
Quoted 6 lines
Quoted 2 lines

Final request: please break your lines around column 70 for readability.

Sure, can do.

Thanks a lot,

You're welcome!

author
#17 lore raw

RFC PATCH 3/3 contrib: Add server for ivshmem revision 2

In reply to Jan Kiszka

From: Jan Kiszka <[email protected]>

This implements the server process for ivshmem v2 device models of QEMU. Again, no effort has been spent yet on sharing code with the v1 server. Parts have been copied, others were rewritten.

In addition to parameters of v1, this server now also specifies

- the maximum number of peers to be connected (required to know in advance because of v2's state table)
- the size of the output sections (can be 0)
- the protocol ID to be published to all peers

When a virtio protocol ID is chosen, only 2 peers can be connected. Furthermore, the server will signal the backend variant of the ID to the master instance and the frontend ID to the slave peer.

To start, e.g., a server that allows virtio console over ivshmem, call

ivshmem2-server -F -l 64K -n 2 -V 3 -P 0x8003

TODO: specify the new server protocol.

---
 Makefile                                  |   3 +
 Makefile.objs                             |   1 +
 configure                                 |   1 +
 contrib/ivshmem2-server/Makefile.objs     |   1 +
 contrib/ivshmem2-server/ivshmem2-server.c | 462 ++++++++++++++++++++++++++++++
 contrib/ivshmem2-server/ivshmem2-server.h | 158 ++++++++++
 contrib/ivshmem2-server/main.c            | 313 ++++++++++++++++++++
 7 files changed, 939 insertions(+)
 create mode 100644 contrib/ivshmem2-server/Makefile.objs
 create mode 100644 contrib/ivshmem2-server/ivshmem2-server.c
 create mode 100644 contrib/ivshmem2-server/ivshmem2-server.h
 create mode 100644 contrib/ivshmem2-server/main.c
Makefile +3 -0
index aa9d1a42aa..85b5e985ac 100644
@@ -430,6 +430,7 @@ dummy := $(call unnest-vars,, \
                 elf2dmp-obj-y \
                 ivshmem-client-obj-y \
                 ivshmem-server-obj-y \
+                ivshmem2-server-obj-y \
                 rdmacm-mux-obj-y \
                 libvhost-user-obj-y \
                 vhost-user-scsi-obj-y \
@@ -662,6 +663,8 @@ ivshmem-client$(EXESUF): $(ivshmem-client-obj-y) $(COMMON_LDADDS)
 	$(call LINK, $^)
 ivshmem-server$(EXESUF): $(ivshmem-server-obj-y) $(COMMON_LDADDS)
 	$(call LINK, $^)
+ivshmem2-server$(EXESUF): $(ivshmem2-server-obj-y) $(COMMON_LDADDS)
+	$(call LINK, $^)
 endif
 vhost-user-scsi$(EXESUF): $(vhost-user-scsi-obj-y) libvhost-user.a
 	$(call LINK, $^)
Makefile.objs +1 -0
index 11ba1a36bd..a4a729d808 100644
@@ -117,6 +117,7 @@ qga-vss-dll-obj-y = qga/
 elf2dmp-obj-y = contrib/elf2dmp/
 ivshmem-client-obj-$(CONFIG_IVSHMEM) = contrib/ivshmem-client/
 ivshmem-server-obj-$(CONFIG_IVSHMEM) = contrib/ivshmem-server/
+ivshmem2-server-obj-$(CONFIG_IVSHMEM) = contrib/ivshmem2-server/
 libvhost-user-obj-y = contrib/libvhost-user/
 vhost-user-scsi.o-cflags := $(LIBISCSI_CFLAGS)
 vhost-user-scsi.o-libs := $(LIBISCSI_LIBS)
configure +1 -0
index efe165edf9..1b466b2461 100755
@@ -6184,6 +6184,7 @@ if test "$want_tools" = "yes" ; then
   fi
   if [ "$ivshmem" = "yes" ]; then
     tools="ivshmem-client\$(EXESUF) ivshmem-server\$(EXESUF) $tools"
+    tools="ivshmem2-server\$(EXESUF) $tools"
   fi
   if [ "$curl" = "yes" ]; then
       tools="elf2dmp\$(EXESUF) $tools"
contrib/ivshmem2-server/Makefile.objs new file +1 -0
new file mode 100644
index 0000000000..d233e18ec8
@@ -0,0 +1 @@
+ivshmem2-server-obj-y = ivshmem2-server.o main.o
contrib/ivshmem2-server/ivshmem2-server.c new file +462 -0

462 lines not on this page · raw message

contrib/ivshmem2-server/ivshmem2-server.h new file +158 -0

158 lines not on this page · raw message

contrib/ivshmem2-server/main.c new file +313 -0

313 lines not on this page · raw message

Signature · 1 line
2.16.4
#18 lore raw
In reply to Jan Kiszka
Patchew URL: https://patchew.org/QEMU/[email protected]/

Patchew URL: https://patchew.org/QEMU/[email protected]/

Hi,

This series seems to have some coding style problems. See output below for more information:

Subject: [RFC][PATCH 0/3] IVSHMEM version 2 device for QEMU
Type: series
Message-id: [email protected]

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384 Switched to a new branch 'test'
45625de contrib: Add server for ivshmem revision 2 df18ce0 docs/specs: Add specification of ivshmem device revision 2 ff35318 hw/misc: Add implementation of ivshmem revision 2 device

=== OUTPUT BEGIN ===
1/3 Checking commit ff35318fdf84 (hw/misc: Add implementation of ivshmem revision 2 device)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating? #63:
new file mode 100644

ERROR: return is not a function, parentheses are not required #206: FILE: hw/misc/ivshmem2.c:139:
+ return (ivs->features & (1 << feature));

ERROR: memory barrier without comment
#250: FILE: hw/misc/ivshmem2.c:183:
+ smp_mb();

ERROR: braces {} are necessary for all arms of this statement #625: FILE: hw/misc/ivshmem2.c:558:
+ if (msg->vector == 0)
[...]

WARNING: Block comments use a leading /* on a separate line #775: FILE: hw/misc/ivshmem2.c:708:
+/* Select the MSI-X vectors used by device.

WARNING: Block comments use a trailing */ on a separate line #777: FILE: hw/misc/ivshmem2.c:710:
+ * we just enable all vectors on init and after reset. */

total: 3 errors, 3 warnings, 1147 lines checked

Patch 1/3 has style problems, please review. If any of these errors are false positives report them to the maintainer, see CHECKPATCH in MAINTAINERS.

2/3 Checking commit df18ce079161 (docs/specs: Add specification of ivshmem device revision 2)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#24:
new file mode 100644

total: 0 errors, 1 warnings, 333 lines checked

Patch 2/3 has style problems, please review. If any of these errors are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
3/3 Checking commit 45625def0d51 (contrib: Add server for ivshmem revision 2)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating? #77:
new file mode 100644

WARNING: Block comments use a leading /* on a separate line #174: FILE: contrib/ivshmem2-server/ivshmem2-server.c:86:
+/* free a peer when the server advertises a disconnection or when the

WARNING: Block comments use a trailing */ on a separate line #175: FILE: contrib/ivshmem2-server/ivshmem2-server.c:87:
+ * server is freed */

ERROR: memory barrier without comment
#194: FILE: contrib/ivshmem2-server/ivshmem2-server.c:106:
+ smp_mb();

WARNING: Block comments use a leading /* on a separate line #276: FILE: contrib/ivshmem2-server/ivshmem2-server.c:188:
+ /* XXX: this could use id allocation such as Linux IDA, or simply

WARNING: Block comments use a trailing */ on a separate line #277: FILE: contrib/ivshmem2-server/ivshmem2-server.c:189:
+ * a free-list */

WARNING: Block comments use a leading /* on a separate line #342: FILE: contrib/ivshmem2-server/ivshmem2-server.c:254:
+/* Try to ftruncate a file to next power of 2 of shmsize.

WARNING: Block comments use a trailing */ on a separate line #346: FILE: contrib/ivshmem2-server/ivshmem2-server.c:258:
+ * shm_size value. */

WARNING: Block comments use a leading /* on a separate line #619: FILE: contrib/ivshmem2-server/ivshmem2-server.h:63:
+ const char *shm_path; /**< Path to the shared memory; path

WARNING: Block comments use * on subsequent lines #620: FILE: contrib/ivshmem2-server/ivshmem2-server.h:64:
+ const char *shm_path; /**< Path to the shared memory; path
+ corresponds to a POSIX shm name or a

WARNING: Block comments use a trailing */ on a separate line #621: FILE: contrib/ivshmem2-server/ivshmem2-server.h:65:
+ hugetlbfs mount point. */

WARNING: Block comments use a leading /* on a separate line #622: FILE: contrib/ivshmem2-server/ivshmem2-server.h:66:
+ bool use_shm_open; /**< true to use shm_open, false for

WARNING: Block comments use * on subsequent lines #623: FILE: contrib/ivshmem2-server/ivshmem2-server.h:67:
+ bool use_shm_open; /**< true to use shm_open, false for
+ file-backed shared memory */

WARNING: Block comments use a trailing */ on a separate line #623: FILE: contrib/ivshmem2-server/ivshmem2-server.h:67:
+ file-backed shared memory */

ERROR: spaces required around that '*' (ctx:VxV)
#742: FILE: contrib/ivshmem2-server/main.c:22:
+#define IVSHMEM_SERVER_DEFAULT_SHM_SIZE (4*1024*1024)

                                                 ^

ERROR: spaces required around that '*' (ctx:VxV)
#742: FILE: contrib/ivshmem2-server/main.c:22:
+#define IVSHMEM_SERVER_DEFAULT_SHM_SIZE (4*1024*1024)

                                                      ^

WARNING: Block comments use a leading /* on a separate line #906: FILE: contrib/ivshmem2-server/main.c:186:
+/* wait for events on listening server unix socket and connected client

WARNING: Block comments use a trailing */ on a separate line #907: FILE: contrib/ivshmem2-server/main.c:187:
+ * sockets */

WARNING: Block comments use a leading /* on a separate line #977: FILE: contrib/ivshmem2-server/main.c:257:
+ /* Ignore SIGPIPE, see this link for more info:

WARNING: Block comments use a trailing */ on a separate line #978: FILE: contrib/ivshmem2-server/main.c:258:
+ * http://www.mail-archive.com/[email protected]/msg01606.html */

total: 3 errors, 17 warnings, 963 lines checked

Patch 3/3 has style problems, please review. If any of these errors are false positives report them to the maintainer, see CHECKPATCH in MAINTAINERS.

=== OUTPUT END ===

Test command exited with code: 1

The full log is available at
http://patchew.org/logs/[email protected]/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to [email protected]

#19 lore raw
In reply to Jan Kiszka
Quoted 23 lines · Jan Kiszka

To get the ball rolling after my presentation of the topic at KVM Forum
[1] and many fruitful discussions around it, this is a first concrete code series. As discussed, I'm starting with the IVSHMEM implementation of a QEMU device and server. It's RFC because, besides specification and implementation details, there will still be some decisions needed about how to integrate the new version best into the existing code bases.

If you want to play with this, the basic setup of the shared memory device is described in patch 1 and 3. UIO driver and also the virtio-ivshmem prototype can be found at

http://git.kiszka.org/?p=linux.git;a=shortlog;h=refs/heads/queues/ivshmem2

Accessing the device via UIO is trivial enough. If you want to use it for virtio, this is additionally to the description in patch 3 needed on the virtio console backend side:

    modprobe uio_ivshmem
    echo "1af4 1110 1af4 1100 ffc003 ffffff" > /sys/bus/pci/drivers/uio_ivshmem/new_id
    linux/tools/virtio/virtio-ivshmem-console /dev/uio0

And for virtio block:

Quoted 8 lines

echo "1af4 1110 1af4 1100 ffc002 ffffff" > /sys/bus/pci/drivers/uio_ivshmem/new_id linux/tools/virtio/virtio-ivshmem-console /dev/uio0 /path/to/disk.img

After that, you can start the QEMU frontend instance with the virtio-ivshmem driver installed which can use the new /dev/hvc* or /dev/vda* as usual.

Any feedback welcome!

Hi, Jan,

I have been playing your code for last few weeks, mostly study and test, of course. Really nice work. I have a few questions here:

First, qemu part looks good, I tried test between couple VMs, and device could pop up correctly for all of them, but I had some problems when trying load driver. For example, if set up two VMs, vm1 and vm2, start ivshmem server as you suggested. vm1 could load uio_ivshmem and virtio_ivshmem correctly, vm2 could load uio_ivshmem but could not show up "/dev/uio0", virtio_ivshmem could not be loaded at all, these still exist even I switch the load sequence of vm1 and vm2, and sometimes reset "virtio_ivshmem" could crash both vm1 and vm2. Not quite sure this is bug or "Ivshmem Mode" issue, but I went through ivshmem-server code, did not related information.

I started some code work recently, such as fix code style issues and some work based on above testing, however I know you are also working on RFC V2, beside the protocol between server-client and client-client is not finalized yet either, things may change, so much appreciate if you could squeeze me into your develop schedule and share with me some plans, :-) Maybe I could send some pull request in your github repo?

I personally like this project a lot, there would be a lot of potential and user case for it, especially some devices like ivshmem-net/ivshmem-block. Anyway, thanks for adding me to the list, and looking forward to your reply.

Best,
Liang

Quoted 37 lines

Jan

PS: Let me know if I missed someone potentially interested in this topic on CC - or if you would like to be dropped from the list.

PPS: The Jailhouse queues are currently out of sync /wrt minor details of this one, primarily the device ID. Will update them when the general direction is clear.

[1] https://kvmforum2019.sched.com/event/TmxI

Jan Kiszka (3):
hw/misc: Add implementation of ivshmem revision 2 device docs/specs: Add specification of ivshmem device revision 2
contrib: Add server for ivshmem revision 2

 Makefile                                  |    3 +
 Makefile.objs                             |    1 +
 configure                                 |    1 +
 contrib/ivshmem2-server/Makefile.objs     |    1 +
 contrib/ivshmem2-server/ivshmem2-server.c |  462 ++++++++++++
 contrib/ivshmem2-server/ivshmem2-server.h |  158 +++++
 contrib/ivshmem2-server/main.c            |  313 +++++++++
 docs/specs/ivshmem-2-device-spec.md       |  333 +++++++++
 hw/misc/Makefile.objs                     |    2 +-
 hw/misc/ivshmem2.c                        | 1091 +++++++++++++++++++++++++++++
 include/hw/misc/ivshmem2.h                |   48 ++
 11 files changed, 2412 insertions(+), 1 deletion(-)
 create mode 100644 contrib/ivshmem2-server/Makefile.objs
 create mode 100644 contrib/ivshmem2-server/ivshmem2-server.c
 create mode 100644 contrib/ivshmem2-server/ivshmem2-server.h
 create mode 100644 contrib/ivshmem2-server/main.c
 create mode 100644 docs/specs/ivshmem-2-device-spec.md
 create mode 100644 hw/misc/ivshmem2.c
 create mode 100644 include/hw/misc/ivshmem2.h
author
#20 lore raw

Hi Liang,

Quoted 40 lines · Liang Yan
Quoted 23 lines · Jan Kiszka

To get the ball rolling after my presentation of the topic at KVM Forum
[1] and many fruitful discussions around it, this is a first concrete code series. As discussed, I'm starting with the IVSHMEM implementation of a QEMU device and server. It's RFC because, besides specification and implementation details, there will still be some decisions needed about how to integrate the new version best into the existing code bases.

If you want to play with this, the basic setup of the shared memory device is described in patch 1 and 3. UIO driver and also the virtio-ivshmem prototype can be found at

http://git.kiszka.org/?p=linux.git;a=shortlog;h=refs/heads/queues/ivshmem2

Accessing the device via UIO is trivial enough. If you want to use it for virtio, this is additionally to the description in patch 3 needed on the virtio console backend side:

     modprobe uio_ivshmem
     echo "1af4 1110 1af4 1100 ffc003 ffffff" > /sys/bus/pci/drivers/uio_ivshmem/new_id
     linux/tools/virtio/virtio-ivshmem-console /dev/uio0

And for virtio block:

Quoted 8 lines

echo "1af4 1110 1af4 1100 ffc002 ffffff" > /sys/bus/pci/drivers/uio_ivshmem/new_id linux/tools/virtio/virtio-ivshmem-console /dev/uio0 /path/to/disk.img

After that, you can start the QEMU frontend instance with the virtio-ivshmem driver installed which can use the new /dev/hvc* or /dev/vda* as usual.

Any feedback welcome!

Hi, Jan,

I have been playing your code for last few weeks, mostly study and test, of course. Really nice work. I have a few questions here:

Quoted 10 lines

First, qemu part looks good, I tried test between couple VMs, and device could pop up correctly for all of them, but I had some problems when trying load driver. For example, if set up two VMs, vm1 and vm2, start ivshmem server as you suggested. vm1 could load uio_ivshmem and virtio_ivshmem correctly, vm2 could load uio_ivshmem but could not show up "/dev/uio0", virtio_ivshmem could not be loaded at all, these still exist even I switch the load sequence of vm1 and vm2, and sometimes reset "virtio_ivshmem" could crash both vm1 and vm2. Not quite sure this is bug or "Ivshmem Mode" issue, but I went through ivshmem-server code, did not related information.

If we are only talking about one ivshmem link and vm1 is the master, there is not role for virtio_ivshmem on it as backend. That is purely a frontend driver. Vice versa for vm2: If you want to use its ivshmem instance as virtio frontend, uio_ivshmem plays no role.

The "crash" is would be interesting to understand: Do you see kernel panics of the guests? Or are they stuck? Or are the QEMU instances stuck? Do you know that you can debug the guest kernels via gdb (and gdb-scripts of the kernel)?

Quoted 7 lines

I started some code work recently, such as fix code style issues and some work based on above testing, however I know you are also working on RFC V2, beside the protocol between server-client and client-client is not finalized yet either, things may change, so much appreciate if you could squeeze me into your develop schedule and share with me some plans, :-) Maybe I could send some pull request in your github repo?

I'm currently working on a refresh of the Jailhouse queue and the kernel patches to incorporate just two smaller changes:

- use Siemens device ID
- drop "features" register from ivshmem device

I have not yet touched the QEMU code for that so far, thus no conflict yet. I would wait for your patches then.

If it helps us to work on this together, I can push things to github as well. Will drop you a note when done. We should just present the outcome frequently as new series to the list.

Quoted 5 lines

I personally like this project a lot, there would be a lot of potential and user case for it, especially some devices like ivshmem-net/ivshmem-block. Anyway, thanks for adding me to the list, and looking forward to your reply.

Thanks for the positive feedback. I'm looking forward to work on this together!

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
author
#21 lore raw
Quoted 68 lines · Jan Kiszka

Hi Liang,

Quoted 42 lines · Liang Yan
Quoted 25 lines · Jan Kiszka

To get the ball rolling after my presentation of the topic at KVM Forum
[1] and many fruitful discussions around it, this is a first concrete code series. As discussed, I'm starting with the IVSHMEM implementation of a QEMU device and server. It's RFC because, besides specification and implementation details, there will still be some decisions needed about how to integrate the new version best into the existing code bases.

If you want to play with this, the basic setup of the shared memory device is described in patch 1 and 3. UIO driver and also the virtio-ivshmem prototype can be found at

http://git.kiszka.org/?p=linux.git;a=shortlog;h=refs/heads/queues/ivshmem2

Accessing the device via UIO is trivial enough. If you want to use it for virtio, this is additionally to the description in patch 3 needed on the virtio console backend side:

     modprobe uio_ivshmem
     echo "1af4 1110 1af4 1100 ffc003 ffffff" > /sys/bus/pci/drivers/uio_ivshmem/new_id linux/tools/virtio/virtio-ivshmem-console /dev/uio0

And for virtio block:

Quoted 8 lines

     echo "1af4 1110 1af4 1100 ffc002 ffffff" > /sys/bus/pci/drivers/uio_ivshmem/new_id linux/tools/virtio/virtio-ivshmem-console /dev/uio0 /path/to/disk.img

After that, you can start the QEMU frontend instance with the virtio-ivshmem driver installed which can use the new /dev/hvc* or /dev/vda* as usual.

Any feedback welcome!

Hi, Jan,

I have been playing your code for last few weeks, mostly study and test, of course. Really nice work. I have a few questions here:

Quoted 10 lines

First, qemu part looks good, I tried test between couple VMs, and device could pop up correctly for all of them, but I had some problems when trying load driver. For example, if set up two VMs, vm1 and vm2, start ivshmem server as you suggested. vm1 could load uio_ivshmem and virtio_ivshmem correctly, vm2 could load uio_ivshmem but could not show up "/dev/uio0", virtio_ivshmem could not be loaded at all, these still exist even I switch the load sequence of vm1 and vm2, and sometimes reset "virtio_ivshmem" could crash both vm1 and vm2. Not quite sure this is bug or "Ivshmem Mode" issue, but I went through ivshmem-server code, did not related information.

If we are only talking about one ivshmem link and vm1 is the master, there is not role for virtio_ivshmem on it as backend. That is purely a frontend driver. Vice versa for vm2: If you want to use its ivshmem instance as virtio frontend, uio_ivshmem plays no role.

The "crash" is would be interesting to understand: Do you see kernel panics of the guests? Or are they stuck? Or are the QEMU instances stuck? Do you know that you can debug the guest kernels via gdb (and gdb-scripts of the kernel)?

Quoted 7 lines

I started some code work recently, such as fix code style issues and some work based on above testing, however I know you are also working on RFC V2, beside the protocol between server-client and client-client is not finalized yet either, things may change, so much appreciate if you could squeeze me into your develop schedule and share with me some plans, :-)  Maybe I could send some pull request in your github repo?

Quoted 8 lines

I'm currently working on a refresh of the Jailhouse queue and the kernel patches to incorporate just two smaller changes:

 - use Siemens device ID
 - drop "features" register from ivshmem device

I have not yet touched the QEMU code for that so far, thus no conflict yet. I would wait for your patches then.

If it helps us to work on this together, I can push things to github as well. Will drop you a note when done. We should just present the outcome frequently as new series to the list.

I've updated my queues, mostly small changes, mostly to the kernel bits. Besides the already announced places, you can also find them as PR targets on

https://github.com/siemens/qemu/commits/wip/ivshmem2
https://github.com/siemens/linux/commits/queues/ivshmem2

To give the whole thing broader coverage, I will now also move forward and integrate the current state into Jailhouse - at the risk of having to rework the interface there once again. But there are a number of users already requiring the extended features (or even using them), plus this gives a nice test coverage of key components and properties.

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

A message that is not in the archive (sent off-list or not yet received): [email protected]

author
#23 lore raw
Quoted 69 lines · Liang Yan
Quoted 62 lines · Jan Kiszka
Quoted 61 lines · Jan Kiszka

Hi Liang,

Quoted 57 lines · Liang Yan
Quoted 33 lines · Jan Kiszka

To get the ball rolling after my presentation of the topic at KVM Forum
[1] and many fruitful discussions around it, this is a first concrete code series. As discussed, I'm starting with the IVSHMEM implementation of a QEMU device and server. It's RFC because, besides specification and
implementation details, there will still be some decisions needed about how to integrate the new version best into the existing code bases.

If you want to play with this, the basic setup of the shared memory device is described in patch 1 and 3. UIO driver and also the virtio-ivshmem prototype can be found at


http://git.kiszka.org/?p=linux.git;a=shortlog;h=refs/heads/queues/ivshmem2

Accessing the device via UIO is trivial enough. If you want to use it for virtio, this is additionally to the description in patch 3 needed on
the virtio console backend side:

     modprobe uio_ivshmem
     echo "1af4 1110 1af4 1100 ffc003 ffffff" > /sys/bus/pci/drivers/uio_ivshmem/new_id linux/tools/virtio/virtio-ivshmem-console /dev/uio0

And for virtio block:

     echo "1af4 1110 1af4 1100 ffc002 ffffff" > /sys/bus/pci/drivers/uio_ivshmem/new_id linux/tools/virtio/virtio-ivshmem-console /dev/uio0 /path/to/disk.img

Quoted 5 lines

After that, you can start the QEMU frontend instance with the virtio-ivshmem driver installed which can use the new /dev/hvc* or /dev/vda* as usual.

Any feedback welcome!

Hi, Jan,

I have been playing your code for last few weeks, mostly study and test, of course. Really nice work. I have a few questions here:

First, qemu part looks good, I tried test between couple VMs, and device could pop up correctly for all of them, but I had some problems when trying load driver. For example, if set up two VMs, vm1 and vm2, start ivshmem server as you suggested. vm1 could load uio_ivshmem and virtio_ivshmem correctly, vm2 could load uio_ivshmem but could not show up "/dev/uio0", virtio_ivshmem could not be loaded at all, these still exist even I switch the load sequence of vm1 and vm2, and sometimes reset "virtio_ivshmem" could crash both vm1 and vm2. Not quite sure this is bug or "Ivshmem Mode" issue, but I went through ivshmem-server code, did not related information.

Quoted 5 lines
Quoted 5 lines

If we are only talking about one ivshmem link and vm1 is the master, there is not role for virtio_ivshmem on it as backend. That is purely a frontend driver. Vice versa for vm2: If you want to use its ivshmem instance as virtio frontend, uio_ivshmem plays no role.

Quoted 7 lines

Hi, Jan,

Sorry for the late response. Just came back from Thanksgiving holiday.

I have a few questions here.
First, how to decide master/slave node? I used two VMs here, they did not show same behavior even if I change the boot sequence.

The current mechanism works by selecting the VM gets ID 0 as the backend, thus sending it also a different protocol ID than the frontend gets. Could possibly be improved by allowing selection also on the VM side (QEMU command line parameter etc.).

Inherently, this only affects virtio over ivshmem. Other, symmetric protocols do not need this differentiation.

Quoted 7 lines

Second, in order to run virtio-ivshmem-console demo, VM1 connect to VM2 Console. So, need to install virtio frontend driver in VM2, then install uio frontend driver in VM1 to get "/dev/uio0", then run demo, right? Could you share your procedure?

Also, I could not get "/dev/uio0" all the time.

OK, should have collected this earlier. This is how I start the console demo right now:

- ivshmem2-server -F -l 64K -n 2 -V 3 -P 0x8003
- start backend qemu with something like "-chardev socket,path=/tmp/ivshmem_socket,id=ivshm -device ivshmem,chardev=ivshm" in its command line
- inside that VM
- modprobe uio_ivshmem
- echo "110a 4106 1af4 1100 ffc003 ffffff" > \
/sys/bus/pci/drivers/uio_ivshmem/new_id
- virtio-ivshmem-console /dev/uio0
- start frontend qemu (can be identical options)

Now the frontend VM should see the ivshmem-virtio transport device and attach a virtio console driver to it (/dev/hvc0). If you build the transport into the kernel, you can even do "console=hvc0".

Quoted 8 lines
Quoted 7 lines
Quoted 7 lines

The "crash" is would be interesting to understand: Do you see kernel panics of the guests? Or are they stuck? Or are the QEMU instances stuck? Do you know that you can debug the guest kernels via gdb (and gdb-scripts of the kernel)?

Quoted 6 lines

They are stuck, no kernel panics, It's like during console connection, I try to load/remove/reset module from the other side, then two VMs are stuck. One VM could still run if I killed the other VM. Like I said above, it may be just wrong operation from my side. I am working on ivshmem-block now, it is easier to understand for dual connection case.

As I said, would be good to have an exact description of steps how to reproduce - or you could attach gdb to the instances and do a some backtraces on where the VMs are stuck.

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux