Skip to contentSkip to conversation

Introduce a PCIe endpoint virtio console

RFC PATCH In review Shunsuke Mie 4 participants 18 messages Apr 27, 2023 – May 31, 2023

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v2 cover Introduce a PCIe endpoint virtio console

PCIe endpoint framework provides APIs to implement PCIe endpoint function. This framework allows defining various PCIe endpoint function behaviors in software. This patch extend the framework for virtio pci device. The virtio is defined to communicate guest on virtual machine and host side. Advantage of the virtio is the efficiency of data transfer and the conciseness of implementation device using software. It also be applied to PCIe endpoint function.

We designed and implemented a PCIe EP virtio console function driver using the extended PCIe endpoint framework for virtio. It can be communicate host and endpoint over virtio as console.

An architecture of the function driver is following:

 ┌────────────┐         ┌──────────────────────┬────────────┐
 │virtio      │         │                      │virtio      │
 │console drv │         ├───────────────┐      │console drv │
 ├────────────┤         │(virtio console│      ├────────────┤
 │ virtio bus │         │ device)       │◄────►│ virtio bus │
 ├────────────┤         ├---------------┤      └────────────┤
 │            │         │ pci ep virtio │                   │
 │  pci bus   │         │  console drv  │                   │
 │            │  pcie   ├───────────────┤                   │
 │            │ ◄─────► │  pci ep Bus   │                   │
 └────────────┘         └───────────────┴───────────────────┘
   PCIe Root              PCIe Endpoint

Introduced driver is pci ep virtio console drv in the figure. It works as ep function for PCIe root and virtual virtio console device for PCIe endpoint. Each side of virtio console driver has virtqueue, and introduced driver transfers data on the virtqueue to each other. A data on root tx queue is transfered to endpoint rx queue and vice versa.

This patchset is depend follwing patches which are under discussion.

- [RFC PATCH 0/3] Deal with alignment restriction on EP side

- [PATCH v3] vringh: IOMEM support

First of this patchset is introduce a helper function to realize pci virtio function using PCIe endpoint framework. The second one is adding a missing definition for virtio pci header. The last one is for PCIe endpoint virtio console driver.

This is tested on next-20230416 and RCar S4 board as PCIe endpoint.

Shunsuke Mie (3):
PCI: endpoint: introduce a helper to implement pci ep virtio function
virtio_pci: add a definition of queue flag in ISR
PCI: endpoint: Add EP function driver to provide virtio-console
functionality

 drivers/pci/endpoint/functions/Kconfig        |  19 +
 drivers/pci/endpoint/functions/Makefile       |   2 +
 drivers/pci/endpoint/functions/pci-epf-vcon.c | 596 ++++++++++++++++++
 .../pci/endpoint/functions/pci-epf-virtio.c   | 458 ++++++++++++++
 .../pci/endpoint/functions/pci-epf-virtio.h   | 126 ++++
 include/uapi/linux/virtio_pci.h               |   3 +
 6 files changed, 1204 insertions(+)
 create mode 100644 drivers/pci/endpoint/functions/pci-epf-vcon.c
 create mode 100644 drivers/pci/endpoint/functions/pci-epf-virtio.c
 create mode 100644 drivers/pci/endpoint/functions/pci-epf-virtio.h
Signature · 1 line
2.25.1
#1 lore raw

RFC PATCH v2 1/3 PCI: endpoint: introduce a helper to implement pci ep virtio function

The Linux PCIe Endpoint framework supports to implement PCIe endpoint functions using a PCIe controller operating in endpoint mode. It is possble to realize the behavior of PCIe device, such as virtio PCI device. This patch introduces a setof helper functions and data structures to implement a PCIe endpoint function that behaves as a virtio device.

Those functions enable the implementation PCIe endpoint function that comply with the virtio lecacy specification. Because modern virtio specifications require devices to implement custom PCIe capabilities, which are not currently supported by either PCIe controllers/drivers or the PCIe endpoint framework.

The helper functions work with the new struct epf_virtio, which is initialized and finalized using the following functions:

- int epf_virtio_init();
- void epf_virtio_final()

Once initialized, the PCIe configuration space can be read and written using the following functions:

- epf_virtio_cfg_{read,write}#size()
- epf_virtio_cfg_{set,clear}#size()
- epf_virtio_cfg_memcpy_toio()

The size is supported 8, 16 and 32bits. The content of configuration space varies depending on the type of virtio device.

After the setup, launch the kernel thread for negotiation with host virtio drivers and detection virtqueue notifications with the function:

- epf_virtio_launch_bgtask()

Also there are functions to shutdown and reset the kthread.

- epf_virtio_terminate_bgtask()
- epf_virtio_terminate_reset()

Data transfer function is also provide.

- epf_virtio_memcpy_kiov2kiov()

While this patch provides functions for negotiating with host drivers and copying data, each PCIe function driver must impl ement operations that depend on each specific device, such as network, block, etc.

---

Changes from v2:
- Improve the memcpy function between kiov and kiov on local ram and remote ram via pcie bus.

 drivers/pci/endpoint/functions/Kconfig        |   7 +
 drivers/pci/endpoint/functions/Makefile       |   1 +
 .../pci/endpoint/functions/pci-epf-virtio.c   | 458 ++++++++++++++++++
 .../pci/endpoint/functions/pci-epf-virtio.h   | 126 +++++
 4 files changed, 592 insertions(+)
 create mode 100644 drivers/pci/endpoint/functions/pci-epf-virtio.c
 create mode 100644 drivers/pci/endpoint/functions/pci-epf-virtio.h
drivers/pci/endpoint/functions/Kconfig +7 -0
index 9fd560886871..fa1a6a569a8f 100644
@@ -37,3 +37,10 @@ config PCI_EPF_VNTB
 	  between PCI Root Port and PCIe Endpoint.
 
 	  If in doubt, say "N" to disable Endpoint NTB driver.
+
+config PCI_EPF_VIRTIO
+	tristate
+	depends on PCI_ENDPOINT
+	select VHOST_RING_IOMEM
+	help
+	  Helpers to implement PCI virtio Endpoint function
drivers/pci/endpoint/functions/Makefile +1 -0
index 5c13001deaba..a96f127ce900 100644
@@ -6,3 +6,4 @@
 obj-$(CONFIG_PCI_EPF_TEST)		+= pci-epf-test.o
 obj-$(CONFIG_PCI_EPF_NTB)		+= pci-epf-ntb.o
 obj-$(CONFIG_PCI_EPF_VNTB) 		+= pci-epf-vntb.o
+obj-$(CONFIG_PCI_EPF_VIRTIO)		+= pci-epf-virtio.o
drivers/pci/endpoint/functions/pci-epf-virtio.c new file +458 -0

458 lines not on this page · raw message

drivers/pci/endpoint/functions/pci-epf-virtio.h new file +126 -0

126 lines not on this page · raw message

Signature · 6 lines
2.25.1

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
#2 lore raw

Re: [RFC PATCH v2 1/3] PCI: endpoint: introduce a helper to implement pci ep virtio function

Simple typos, don't repost until there's more substantive feedback.

Quoted 5 lines · Shunsuke Mie

The Linux PCIe Endpoint framework supports to implement PCIe endpoint functions using a PCIe controller operating in endpoint mode. It is possble to realize the behavior of PCIe device, such as virtio PCI device. This patch introduces a setof helper functions and data structures to implement a PCIe endpoint function that behaves as a virtio device.

s/possble/possible/
s/setof/set of/

Quoted 5 lines

Those functions enable the implementation PCIe endpoint function that comply with the virtio lecacy specification. Because modern virtio specifications require devices to implement custom PCIe capabilities, which are not currently supported by either PCIe controllers/drivers or the PCIe endpoint framework.

s/implementation PCIe endpoint function/implementation of PCIe endpoint functions/ s/lecacy/legacy/ (What does "legacy" mean? Is there a spec for this?)

I guess "legacy virtio" devices need not implement custom PCIe capabilities, but "modern virtio" devices must implement them?

Capitalize "Endpoint framework" consistently; sometimes it's "Endpoint", other times it's "endpoint".

Quoted 3 lines

While this patch provides functions for negotiating with host drivers and copying data, each PCIe function driver must impl ement operations that depend on each specific device, such as network, block, etc.

s/impl ement/implement/

drivers/pci/endpoint/functions/pci-epf-virtio.c line 7

Quoted 3 lines
drivers/pci/endpoint/functions/pci-epf-virtio.c new file +3 -0
+#include <linux/virtio_pci.h>
+#include <linux/virtio_config.h>
+#include <linux/kthread.h>

Typically the header includes would be alphabetized if possible.

drivers/pci/endpoint/functions/pci-epf-virtio.c line 34

Quoted 4 lines
drivers/pci/endpoint/functions/pci-epf-virtio.c new file +4 -0
+	vq_virt = pci_epc_map_addr(epf->epc, epf->func_no, epf->vfunc_no,
+				   vq_pci_addr, vq_phys, vq_size);
+	if (IS_ERR(vq_virt)) {
+		pr_err("Failed to map virtuqueue to local");

s/virtuqueue/virtqueue/

I know you probably don't have any way to use dev_err(), but this message really needs some context, like a driver name and instance or something.

Quoted 1 line

+#define VIRTIO_PCI_LEGACY_CFG_BAR 0

What makes this a "legacy cfg BAR"? Is there some spec that covers virtio BAR usage?

Quoted 4 lines

+ * epf_virtio_init - initialize struct epf_virtio and setup BAR for virtio
+ * @evio: struct epf_virtio to initialize.
+ * @hdr: pci configuration space to show remote host.
+ * @bar_size: pci BAR size it depends on the virtio device type.

s/pci/PCI/ (there were also a few more instances above in messages or comments)

Quoted 2 lines

+ * epf_virtio_final - finalize struct epf_virtio. it frees bar and memories
+ * @evio: struct epf_virtio to finalize.

s/bar/BAR/

drivers/pci/endpoint/functions/pci-epf-virtio.c line 288

Quoted 7 lines
drivers/pci/endpoint/functions/pci-epf-virtio.c new file +7 -0
+static void epf_virtio_monitor_qnotify(struct epf_virtio *evio)
+{
+	const u16 qn_default = evio->nvq;
+	u16 tmp;
+
+	/* Since there is no way to synchronize between the host and EP functions,
+	 * it is possible to miss multiple notifications.

Multi-line comment style.

drivers/pci/endpoint/functions/pci-epf-virtio.c line 309

Quoted 3 lines
drivers/pci/endpoint/functions/pci-epf-virtio.c new file +3 -0
+	err = epf_virtio_negotiate_vq(evio);
+	if (err < 0) {
+		pr_err("failed to negoticate configs with driver\n");

s/negoticate/negotiate/

Quoted 1 line

+ * epf_virtio_reset - reset virtio status

Some of the function descriptions end with a period (".") and others don't. Please figure out what the most common style is and use that consistently.

drivers/pci/endpoint/functions/pci-epf-virtio.c line 429

Quoted 5 lines
drivers/pci/endpoint/functions/pci-epf-virtio.c new file +5 -0
+			dst = pci_epc_map_addr(epf->epc, epf->func_no,
+					       epf->vfunc_no, dbase, &phys,
+					       slen);
+			if (IS_ERR(dst)) {
+				pr_err("failed to map pci mmoery spact to local\n");

s/pci/PCI/
s/mmoery/memory/
s/spact/space/ ?

Also below.

IIRC some previous messages started with a capital letter. Please make them all consistent.

drivers/pci/endpoint/functions/pci-epf-virtio.c line 455

Quoted 10 lines
drivers/pci/endpoint/functions/pci-epf-virtio.c new file +10 -0
+		if (dir == DMA_MEM_TO_DEV) {
+			pci_epc_unmap_addr(epf->epc, epf->func_no,
+					   epf->vfunc_no, phys, dst, slen);
+		} else {
+			pci_epc_unmap_addr(epf->epc, epf->func_no,
+					   epf->vfunc_no, phys, src, slen);
+		}
+	}
+
+	return 1;

I guess this function returns either a negative error code or the value 1? That seems sort of weird (I think "negative error code or *zero* is more typical), but maybe you're following some convention?

drivers/pci/endpoint/functions/pci-epf-virtio.h line 8

Quoted 4 lines
drivers/pci/endpoint/functions/pci-epf-virtio.h new file +4 -0
+#include <linux/pci-epf.h>
+#include <linux/pci-epc.h>
+#include <linux/vringh.h>
+#include <linux/dmaengine.h>

Alpha order if possible

drivers/pci/endpoint/functions/pci-epf-virtio.h line 33

Quoted 1 line
drivers/pci/endpoint/functions/pci-epf-virtio.h new file +1 -0
+	/* Virtual address of pci configuration space */

s/pci/PCI/

drivers/pci/endpoint/functions/pci-epf-virtio.h line 39

Quoted 4 lines
drivers/pci/endpoint/functions/pci-epf-virtio.h new file +4 -0
+	/* Callback function and parameter for queue notifcation
+	 * Note: PCI EP function cannot detect qnotify accurately, therefore this
+	 * callback function should check all of virtqueue's changes.
+	 */

Multi-line comment style.

Bjorn
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#3 lore raw

Re: [RFC PATCH v2 1/3] PCI: endpoint: introduce a helper to implement pci ep virtio function

I'll fix the typo and some styles you mentioned. In this e-mail, I reply to the other comments. 2023年4月28日(金) 3:28 Bjorn Helgaas <[email protected]>:

Quoted 13 lines

Simple typos, don't repost until there's more substantive feedback.

Quoted 5 lines · Shunsuke Mie

The Linux PCIe Endpoint framework supports to implement PCIe endpoint functions using a PCIe controller operating in endpoint mode. It is possble to realize the behavior of PCIe device, such as virtio PCI device. This patch introduces a setof helper functions and data structures to implement a PCIe endpoint function that behaves as a virtio device.

s/possble/possible/
s/setof/set of/

Quoted 8 lines
Quoted 5 lines

Those functions enable the implementation PCIe endpoint function that comply with the virtio lecacy specification. Because modern virtio specifications require devices to implement custom PCIe capabilities, which are not currently supported by either PCIe controllers/drivers or the PCIe endpoint framework.

s/implementation PCIe endpoint function/implementation of PCIe endpoint functions/ s/lecacy/legacy/ (What does "legacy" mean? Is there a spec for this?)

Yes, it is defined in spec.

Quoted 2 lines

I guess "legacy virtio" devices need not implement custom PCIe capabilities, but "modern virtio" devices must implement them?

That I wanted to explain here.

Quoted 2 lines

Capitalize "Endpoint framework" consistently; sometimes it's "Endpoint", other times it's "endpoint".

I'll take care to be consistent.

Quoted 11 lines
Quoted 3 lines

While this patch provides functions for negotiating with host drivers and copying data, each PCIe function driver must impl ement operations that depend on each specific device, such as network, block, etc.

s/impl ement/implement/

drivers/pci/endpoint/functions/pci-epf-virtio.c line 7

Quoted 3 lines
drivers/pci/endpoint/functions/pci-epf-virtio.c new file +3 -0
+#include <linux/virtio_pci.h>
+#include <linux/virtio_config.h>
+#include <linux/kthread.h>

Typically the header includes would be alphabetized if possible.

I'll try to reorder them.

Quoted 10 lines

drivers/pci/endpoint/functions/pci-epf-virtio.c line 34

Quoted 4 lines
drivers/pci/endpoint/functions/pci-epf-virtio.c new file +4 -0
+     vq_virt = pci_epc_map_addr(epf->epc, epf->func_no, epf->vfunc_no,
+                                vq_pci_addr, vq_phys, vq_size);
+     if (IS_ERR(vq_virt)) {
+             pr_err("Failed to map virtuqueue to local");

s/virtuqueue/virtqueue/

I know you probably don't have any way to use dev_err(), but this message really needs some context, like a driver name and instance or something.

I'll try to use the dev_* function appropriately If possible.

Quoted 4 lines
Quoted 1 line

+#define VIRTIO_PCI_LEGACY_CFG_BAR 0

What makes this a "legacy cfg BAR"? Is there some spec that covers virtio BAR usage?

Yes. Virtio Legacy interface defines the PCI BAR number to use.
https://docs.oasis-open.org/virtio/virtio/v1.2/csd01/virtio-v1.2-csd01.html#x1-14500010

Quoted 27 lines
Quoted 4 lines

+ * epf_virtio_init - initialize struct epf_virtio and setup BAR for virtio
+ * @evio: struct epf_virtio to initialize.
+ * @hdr: pci configuration space to show remote host.
+ * @bar_size: pci BAR size it depends on the virtio device type.

s/pci/PCI/ (there were also a few more instances above in messages or comments)

Quoted 2 lines

+ * epf_virtio_final - finalize struct epf_virtio. it frees bar and memories
+ * @evio: struct epf_virtio to finalize.

s/bar/BAR/

drivers/pci/endpoint/functions/pci-epf-virtio.c line 288

Quoted 7 lines
drivers/pci/endpoint/functions/pci-epf-virtio.c new file +7 -0
+static void epf_virtio_monitor_qnotify(struct epf_virtio *evio)
+{
+     const u16 qn_default = evio->nvq;
+     u16 tmp;
+
+     /* Since there is no way to synchronize between the host and EP functions,
+      * it is possible to miss multiple notifications.

Multi-line comment style.

drivers/pci/endpoint/functions/pci-epf-virtio.c line 309

Quoted 3 lines
drivers/pci/endpoint/functions/pci-epf-virtio.c new file +3 -0
+     err = epf_virtio_negotiate_vq(evio);
+     if (err < 0) {
+             pr_err("failed to negoticate configs with driver\n");
Quoted 7 lines

s/negoticate/negotiate/

Quoted 1 line

+ * epf_virtio_reset - reset virtio status

Some of the function descriptions end with a period (".") and others don't. Please figure out what the most common style is and use that consistently.

I agree. I'll fix to be consistent.

Quoted 6 lines

drivers/pci/endpoint/functions/pci-epf-virtio.c line 429

Quoted 5 lines
drivers/pci/endpoint/functions/pci-epf-virtio.c new file +5 -0
+                     dst = pci_epc_map_addr(epf->epc, epf->func_no,
+                                            epf->vfunc_no, dbase, &phys,
+                                            slen);
+                     if (IS_ERR(dst)) {
+                             pr_err("failed to map pci mmoery spact to local\n");
Quoted 8 lines

s/pci/PCI/
s/mmoery/memory/
s/spact/space/ ?

Also below.

IIRC some previous messages started with a capital letter. Please make them all consistent.

Sure.

Quoted 11 lines

drivers/pci/endpoint/functions/pci-epf-virtio.c line 455

Quoted 10 lines
drivers/pci/endpoint/functions/pci-epf-virtio.c new file +10 -0
+             if (dir == DMA_MEM_TO_DEV) {
+                     pci_epc_unmap_addr(epf->epc, epf->func_no,
+                                        epf->vfunc_no, phys, dst, slen);
+             } else {
+                     pci_epc_unmap_addr(epf->epc, epf->func_no,
+                                        epf->vfunc_no, phys, src, slen);
+             }
+     }
+
+     return 1;
Quoted 3 lines

I guess this function returns either a negative error code or the value 1? That seems sort of weird (I think "negative error code or *zero* is more typical), but maybe you're following some convention?

It has to be 0. It is my mistake.

Some vringh functions return 0 (data does not exist), 1 (it exists), or error. But this functions is not related.

Quoted 6 lines

drivers/pci/endpoint/functions/pci-epf-virtio.h line 8

Quoted 4 lines
drivers/pci/endpoint/functions/pci-epf-virtio.h new file +4 -0
+#include <linux/pci-epf.h>
+#include <linux/pci-epc.h>
+#include <linux/vringh.h>
+#include <linux/dmaengine.h>

Alpha order if possible

Yes.

Quoted 4 lines
Quoted 1 line

+ /* Virtual address of pci configuration space */

s/pci/PCI/

Quoted 8 lines

drivers/pci/endpoint/functions/pci-epf-virtio.h line 39

Quoted 4 lines
drivers/pci/endpoint/functions/pci-epf-virtio.h new file +4 -0
+     /* Callback function and parameter for queue notifcation
+      * Note: PCI EP function cannot detect qnotify accurately, therefore this
+      * callback function should check all of virtqueue's changes.
+      */

Multi-line comment style.

Bjorn

Best regards,
Shunsuke
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#4 lore raw

Re: [RFC PATCH v2 1/3] PCI: endpoint: introduce a helper to implement pci ep virtio function

In reply to Shunsuke Mie
Quoted 190 lines · Shunsuke Mie

The Linux PCIe Endpoint framework supports to implement PCIe endpoint functions using a PCIe controller operating in endpoint mode. It is possble to realize the behavior of PCIe device, such as virtio PCI device. This patch introduces a setof helper functions and data structures to implement a PCIe endpoint function that behaves as a virtio device.

Those functions enable the implementation PCIe endpoint function that comply with the virtio lecacy specification. Because modern virtio specifications require devices to implement custom PCIe capabilities, which are not currently supported by either PCIe controllers/drivers or the PCIe endpoint framework.

The helper functions work with the new struct epf_virtio, which is initialized and finalized using the following functions:

- int epf_virtio_init();
- void epf_virtio_final()

Once initialized, the PCIe configuration space can be read and written using the following functions:

- epf_virtio_cfg_{read,write}#size()
- epf_virtio_cfg_{set,clear}#size()
- epf_virtio_cfg_memcpy_toio()

The size is supported 8, 16 and 32bits. The content of configuration space varies depending on the type of virtio device.

After the setup, launch the kernel thread for negotiation with host virtio drivers and detection virtqueue notifications with the function:

- epf_virtio_launch_bgtask()

Also there are functions to shutdown and reset the kthread.

- epf_virtio_terminate_bgtask()
- epf_virtio_terminate_reset()

Data transfer function is also provide.

- epf_virtio_memcpy_kiov2kiov()

While this patch provides functions for negotiating with host drivers and copying data, each PCIe function driver must impl ement operations that depend on each specific device, such as network, block, etc.

---

Changes from v2:
- Improve the memcpy function between kiov and kiov on local ram and remote ram via pcie bus.

 drivers/pci/endpoint/functions/Kconfig        |   7 +
 drivers/pci/endpoint/functions/Makefile       |   1 +
 .../pci/endpoint/functions/pci-epf-virtio.c   | 458 ++++++++++++++++++
 .../pci/endpoint/functions/pci-epf-virtio.h   | 126 +++++
 4 files changed, 592 insertions(+)
 create mode 100644 drivers/pci/endpoint/functions/pci-epf-virtio.c
 create mode 100644 drivers/pci/endpoint/functions/pci-epf-virtio.h
drivers/pci/endpoint/functions/Kconfig +7 -0
index 9fd560886871..fa1a6a569a8f 100644
@@ -37,3 +37,10 @@ config PCI_EPF_VNTB
          between PCI Root Port and PCIe Endpoint.
 
          If in doubt, say "N" to disable Endpoint NTB driver.
+
+config PCI_EPF_VIRTIO
+       tristate
+       depends on PCI_ENDPOINT
+       select VHOST_RING_IOMEM
+       help
+         Helpers to implement PCI virtio Endpoint function
drivers/pci/endpoint/functions/Makefile +1 -0
index 5c13001deaba..a96f127ce900 100644
@@ -6,3 +6,4 @@
 obj-$(CONFIG_PCI_EPF_TEST)             += pci-epf-test.o
 obj-$(CONFIG_PCI_EPF_NTB)              += pci-epf-ntb.o
 obj-$(CONFIG_PCI_EPF_VNTB)             += pci-epf-vntb.o
+obj-$(CONFIG_PCI_EPF_VIRTIO)           += pci-epf-virtio.o
drivers/pci/endpoint/functions/pci-epf-virtio.c new file +98 -0
new file mode 100644
index 000000000000..f67610dd247d
@@ -0,0 +1,458 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Helpers to implement PCIe virtio EP function.
+ */
+#include <linux/virtio_pci.h>
+#include <linux/virtio_config.h>
+#include <linux/kthread.h>
+
+#include "pci-epf-virtio.h"
+
+static void epf_virtio_unmap_vq(struct pci_epf *epf, void __iomem *vq_virt,
+                               phys_addr_t vq_phys, unsigned int num)
+{
+       size_t vq_size = vring_size(num, VIRTIO_PCI_VRING_ALIGN);
+
+       pci_epc_unmap_addr(epf->epc, epf->func_no, epf->vfunc_no, vq_phys,
+                          vq_virt, vq_size);
+       pci_epc_mem_free_addr(epf->epc, vq_phys, vq_virt, vq_size);
+}
+
+static void __iomem *epf_virtio_map_vq(struct pci_epf *epf,
+                                      phys_addr_t vq_pci_addr,
+                                      unsigned int num, phys_addr_t *vq_phys)
+{
+       int err;
+       size_t vq_size;
+       void __iomem *vq_virt;
+
+       vq_size = vring_size(num, VIRTIO_PCI_VRING_ALIGN);
+
+       vq_virt = pci_epc_map_addr(epf->epc, epf->func_no, epf->vfunc_no,
+                                  vq_pci_addr, vq_phys, vq_size);
+       if (IS_ERR(vq_virt)) {
+               pr_err("Failed to map virtuqueue to local");
+               goto err_free;
+       }
+
+       return vq_virt;
+
+err_free:
+       pci_epc_mem_free_addr(epf->epc, *vq_phys, vq_virt, vq_size);
+
+       return ERR_PTR(err);
+}
+
+static void epf_virtio_free_vringh(struct pci_epf *epf, struct epf_vringh *evrh)
+{
+       epf_virtio_unmap_vq(epf, evrh->virt, evrh->phys, evrh->num);
+       kfree(evrh);
+}
+
+static struct epf_vringh *epf_virtio_alloc_vringh(struct pci_epf *epf,
+                                                 u64 features,
+                                                 phys_addr_t pci_addr,
+                                                 unsigned int num)
+{
+       int err;
+       struct vring vring;
+       struct epf_vringh *evrh;
+
+       evrh = kmalloc(sizeof(*evrh), GFP_KERNEL);
+       if (!evrh) {
+               err = -ENOMEM;
+               goto err_unmap_vq;
+       }
+
+       evrh->num = num;
+
+       evrh->virt = epf_virtio_map_vq(epf, pci_addr, num, &evrh->phys);
+       if (IS_ERR(evrh->virt))
+               return evrh->virt;
+
+       vring_init(&vring, num, evrh->virt, VIRTIO_PCI_VRING_ALIGN);
+
+       err = vringh_init_iomem(&evrh->vrh, features, num, false, vring.desc,
+                               vring.avail, vring.used);
+       if (err)
+               goto err_free_epf_vq;
+
+       return evrh;
+
+err_free_epf_vq:
+       kfree(evrh);
+
+err_unmap_vq:
+       epf_virtio_unmap_vq(epf, evrh->virt, evrh->phys, evrh->num);
+
+       return ERR_PTR(err);
+}
+
+#define VIRTIO_PCI_LEGACY_CFG_BAR 0
+
+static void __iomem *epf_virtio_alloc_bar(struct pci_epf *epf, size_t size)
+{
+       struct pci_epf_bar *config_bar = &epf->bar[VIRTIO_PCI_LEGACY_CFG_BAR];
+       const struct pci_epc_features *features;
+       void __iomem *bar;
+       int err;

drivers/pci/endpoint/functions/pci-epf-virtio.c line 108

Quoted 10 lines
drivers/pci/endpoint/functions/pci-epf-virtio.c new file +10 -0
+
+       features = pci_epc_get_features(epf->epc, epf->func_no, epf->vfunc_no);
+       if (!features) {
+               pr_debug("Failed to get PCI EPC features\n");
+               return ERR_PTR(-EOPNOTSUPP);
+       }
+
+       if (features->reserved_bar & BIT(VIRTIO_PCI_LEGACY_CFG_BAR)) {
+               pr_debug("Cannot use the PCI BAR for legacy virtio pci\n");
+               return ERR_PTR(-EOPNOTSUPP);

Since 1.0 has been used for years, I would suggest starting from a modern device other than a legacy one. Otherwise you may end up with some hacky stuffs more below.

Quoted 8 lines
drivers/pci/endpoint/functions/pci-epf-virtio.c new file +8 -0
+       }
+
+       if (features->bar_fixed_size[VIRTIO_PCI_LEGACY_CFG_BAR]) {
+               if (size >
+                   features->bar_fixed_size[VIRTIO_PCI_LEGACY_CFG_BAR]) {
+                       pr_debug("PCI BAR size is not enough\n");
+                       return ERR_PTR(-ENOMEM);
+               }

drivers/pci/endpoint/functions/pci-epf-virtio.c line 126

Quoted 10 lines
drivers/pci/endpoint/functions/pci-epf-virtio.c new file +10 -0
+       }
+
+       bar = pci_epf_alloc_space(epf, size, VIRTIO_PCI_LEGACY_CFG_BAR,
+                                 features->align, PRIMARY_INTERFACE);
+       if (!bar) {
+               pr_debug("Failed to allocate virtio-net config memory\n");
+               return ERR_PTR(-ENOMEM);
+       }
+
+       config_bar->flags |= PCI_BASE_ADDRESS_MEM_TYPE_64;

This is the tricky part:

1) spec said legacy bar is I/O not memory
2) this code may still work if the host is running with Linux since the virtio driver work for memory bar
3) but it may not work if the host is not running with Linux

Quoted 91 lines
drivers/pci/endpoint/functions/pci-epf-virtio.c new file +91 -0
+       err = pci_epc_set_bar(epf->epc, epf->func_no, epf->vfunc_no,
+                             config_bar);
+       if (err) {
+               pr_debug("Failed to set PCI BAR");
+               goto err_free_space;
+       }
+
+       return bar;
+
+err_free_space:
+
+       pci_epf_free_space(epf, bar, VIRTIO_PCI_LEGACY_CFG_BAR,
+                          PRIMARY_INTERFACE);
+
+       return ERR_PTR(err);
+}
+
+static void epf_virtio_free_bar(struct pci_epf *epf, void __iomem *bar)
+{
+       struct pci_epf_bar *config_bar = &epf->bar[VIRTIO_PCI_LEGACY_CFG_BAR];
+
+       pci_epc_clear_bar(epf->epc, epf->func_no, epf->vfunc_no, config_bar);
+       pci_epf_free_space(epf, bar, VIRTIO_PCI_LEGACY_CFG_BAR,
+                          PRIMARY_INTERFACE);
+}
+
+static void epf_virtio_init_bar(struct epf_virtio *evio, void __iomem *bar)
+{
+       evio->bar = bar;
+
+       epf_virtio_cfg_write32(evio, VIRTIO_PCI_HOST_FEATURES, evio->features);
+       epf_virtio_cfg_write16(evio, VIRTIO_PCI_ISR, VIRTIO_PCI_ISR_QUEUE);
+       epf_virtio_cfg_write16(evio, VIRTIO_PCI_QUEUE_NUM, evio->vqlen);
+       epf_virtio_cfg_write16(evio, VIRTIO_PCI_QUEUE_NOTIFY, evio->nvq);
+       epf_virtio_cfg_write8(evio, VIRTIO_PCI_STATUS, 0);
+}
+
+/**
+ * epf_virtio_init - initialize struct epf_virtio and setup BAR for virtio
+ * @evio: struct epf_virtio to initialize.
+ * @hdr: pci configuration space to show remote host.
+ * @bar_size: pci BAR size it depends on the virtio device type.
+ *
+ * Returns zero or a negative error.
+ */
+int epf_virtio_init(struct epf_virtio *evio, struct pci_epf_header *hdr,
+                   size_t bar_size)
+{
+       struct pci_epf *epf = evio->epf;
+       void __iomem *bar;
+       int err;
+
+       err = pci_epc_write_header(epf->epc, epf->func_no, epf->vfunc_no, hdr);
+       if (err)
+               return err;
+
+       bar = epf_virtio_alloc_bar(epf, bar_size);
+       if (IS_ERR(bar))
+               return PTR_ERR(bar);
+
+       epf_virtio_init_bar(evio, bar);
+
+       return 0;
+}
+EXPORT_SYMBOL_GPL(epf_virtio_init);
+
+/**
+ * epf_virtio_final - finalize struct epf_virtio. it frees bar and memories
+ * @evio: struct epf_virtio to finalize.
+ */
+void epf_virtio_final(struct epf_virtio *evio)
+{
+       epf_virtio_free_bar(evio->epf, evio->bar);
+
+       for (int i = 0; i < evio->nvq; i++)
+               epf_virtio_free_vringh(evio->epf, evio->vrhs[i]);
+
+       kfree(evio->vrhs);
+}
+EXPORT_SYMBOL_GPL(epf_virtio_final);
+
+static int epf_virtio_negotiate_vq(struct epf_virtio *evio)
+{
+       u32 pfn;
+       u16 sel;
+       int i = 0;
+       struct _pair {
+               u32 pfn;
+               u16 sel;
+       } *tmp;
+       int err = 0;

drivers/pci/endpoint/functions/pci-epf-virtio.c line 227

Quoted 10 lines
drivers/pci/endpoint/functions/pci-epf-virtio.c new file +10 -0
+       size_t nvq = evio->nvq;
+
+       tmp = kmalloc_array(nvq, sizeof(tmp[0]), GFP_KERNEL);
+       if (!tmp)
+               return -ENOMEM;
+
+       /*
+        * PCIe EP framework has no capability to hook access PCI BAR space from
+        * remote host driver, so poll the specific register, queue pfn to detect
+        * the writing from the driver.

There were discussions in the past which tried to have a new transport that works for the endpoint instead of doing tricks like this. Have you ever considered this?

drivers/pci/endpoint/functions/pci-epf-virtio.c line 236

Quoted 9 lines
drivers/pci/endpoint/functions/pci-epf-virtio.c new file +9 -0
+        *
+        * This implementation assumes that the address of each virtqueue is
+        * written only once.
+        */
+       for (i = 0; i < nvq; i++) {
+               while (!(pfn = epf_virtio_cfg_read32(evio,
+                                                    VIRTIO_PCI_QUEUE_PFN)) &&
+                      evio->running)
+                       ;

Should we do cond_resched() here?

Quoted 47 lines
drivers/pci/endpoint/functions/pci-epf-virtio.c new file +47 -0
+
+               sel = epf_virtio_cfg_read16(evio, VIRTIO_PCI_QUEUE_SEL);
+
+               epf_virtio_cfg_write32(evio, VIRTIO_PCI_QUEUE_PFN, 0);
+
+               tmp[i].pfn = pfn;
+               tmp[i].sel = sel;
+       }
+
+       if (!evio->running)
+               goto err_out;
+
+       evio->vrhs = kmalloc_array(nvq, sizeof(evio->vrhs[0]), GFP_KERNEL);
+       if (!evio->vrhs) {
+               err = -ENOMEM;
+               goto err_out;
+       }
+
+       for (i = 0; i < nvq; i++) {
+               phys_addr_t pci = tmp[i].pfn << VIRTIO_PCI_QUEUE_ADDR_SHIFT;
+
+               evio->vrhs[i] = epf_virtio_alloc_vringh(
+                       evio->epf, evio->features, pci, evio->vqlen);
+               if (IS_ERR(evio->vrhs[i])) {
+                       err = PTR_ERR(evio->vrhs[i]);
+                       goto err_free_evrhs;
+               }
+       }
+
+       kfree(tmp);
+
+       return 0;
+
+err_free_evrhs:
+       for (i -= 1; i > 0; i--)
+               epf_virtio_free_vringh(evio->epf, evio->vrhs[i]);
+
+       kfree(evio->vrhs);
+
+err_out:
+       kfree(tmp);
+
+       return err;
+}
+
+static void epf_virtio_monitor_qnotify(struct epf_virtio *evio)
+{

drivers/pci/endpoint/functions/pci-epf-virtio.c line 293

Quoted 10 lines
drivers/pci/endpoint/functions/pci-epf-virtio.c new file +10 -0
+       const u16 qn_default = evio->nvq;
+       u16 tmp;
+
+       /* Since there is no way to synchronize between the host and EP functions,
+        * it is possible to miss multiple notifications.
+        */
+       while (evio->running) {
+               tmp = epf_virtio_cfg_read16(evio, VIRTIO_PCI_QUEUE_NOTIFY);
+               if (tmp == qn_default)
+                       continue;

cond_resched()?

Quoted 126 lines
drivers/pci/endpoint/functions/pci-epf-virtio.c new file +126 -0
+
+               epf_virtio_cfg_write16(evio, VIRTIO_PCI_QUEUE_NOTIFY,
+                                      qn_default);
+
+               evio->qn_callback(evio->qn_param);
+       }
+}
+
+static int epf_virtio_bgtask(void *param)
+{
+       struct epf_virtio *evio = param;
+       int err;
+
+       err = epf_virtio_negotiate_vq(evio);
+       if (err < 0) {
+               pr_err("failed to negoticate configs with driver\n");
+               return err;
+       }
+
+       while (!(epf_virtio_cfg_read8(evio, VIRTIO_PCI_STATUS) &
+                VIRTIO_CONFIG_S_DRIVER_OK) &&
+              evio->running)
+               ;
+
+       if (evio->ic_callback && evio->running)
+               evio->ic_callback(evio->ic_param);
+
+       epf_virtio_monitor_qnotify(evio);
+
+       return 0;
+}
+
+/**
+ * epf_virtio_launch_bgtask - spawn a kthread that emulates virtio device
+ * operations.
+ * @evio: It should be initialized prior with epf_virtio_init().
+ *
+ * Returns zero or a negative error.
+ */
+int epf_virtio_launch_bgtask(struct epf_virtio *evio)
+{
+       evio->bgtask = kthread_create(epf_virtio_bgtask, evio,
+                                     "pci-epf-virtio/bgtask");
+       if (IS_ERR(evio->bgtask))
+               return PTR_ERR(evio->bgtask);
+
+       evio->running = true;
+
+       sched_set_fifo(evio->bgtask);
+       wake_up_process(evio->bgtask);
+
+       return 0;
+}
+EXPORT_SYMBOL_GPL(epf_virtio_launch_bgtask);
+
+/**
+ * epf_virtio_terminate_bgtask - shutdown a device emulation kthread.
+ * @evio: struct epf_virtio it already launched bgtask.
+ */
+void epf_virtio_terminate_bgtask(struct epf_virtio *evio)
+{
+       evio->running = false;
+
+       kthread_stop(evio->bgtask);
+}
+EXPORT_SYMBOL_GPL(epf_virtio_terminate_bgtask);
+
+/**
+ * epf_virtio_reset - reset virtio status
+ * @evio: struct epf_virtio to reset
+ *
+ * Returns zero or a negative error.
+ */
+int epf_virtio_reset(struct epf_virtio *evio)
+{
+       epf_virtio_terminate_bgtask(evio);
+       epf_virtio_init_bar(evio, evio->bar);
+
+       return epf_virtio_launch_bgtask(evio);
+}
+EXPORT_SYMBOL_GPL(epf_virtio_reset);
+
+int epf_virtio_getdesc(struct epf_virtio *evio, int index,
+                      struct vringh_kiov *riov, struct vringh_kiov *wiov,
+                      u16 *head)
+{
+       struct vringh *vrh = &evio->vrhs[index]->vrh;
+
+       return vringh_getdesc_iomem(vrh, riov, wiov, head, GFP_KERNEL);
+}
+
+void epf_virtio_abandon(struct epf_virtio *evio, int index, int num)
+{
+       struct vringh *vrh = &evio->vrhs[index]->vrh;
+
+       vringh_abandon_iomem(vrh, num);
+}
+
+void epf_virtio_iov_complete(struct epf_virtio *evio, int index, u16 head,
+                            size_t total_len)
+{
+       struct vringh *vrh = &evio->vrhs[index]->vrh;
+
+       vringh_complete_iomem(vrh, head, total_len);
+}
+
+int epf_virtio_memcpy_kiov2kiov(struct epf_virtio *evio,
+                               struct vringh_kiov *siov,
+                               struct vringh_kiov *diov,
+                               enum dma_transfer_direction dir)
+{
+       struct pci_epf *epf = evio->epf;
+       size_t slen, dlen;
+       u64 sbase, dbase;
+       phys_addr_t phys;
+       void *dst, *src;
+
+       for (; siov->i < siov->used; siov->i++, diov->i++) {
+               slen = siov->iov[siov->i].iov_len;
+               sbase = (u64)siov->iov[siov->i].iov_base;
+               dlen = diov->iov[diov->i].iov_len;
+               dbase = (u64)diov->iov[diov->i].iov_base;
+
+               if (dlen < slen) {
+                       pr_info("not enough buffer\n");
+                       return -EINVAL;

drivers/pci/endpoint/functions/pci-epf-virtio.c line 429

Quoted 10 lines
drivers/pci/endpoint/functions/pci-epf-virtio.c new file +10 -0
+               }
+
+               if (dir == DMA_MEM_TO_DEV) {
+                       src = phys_to_virt(sbase);
+
+                       dst = pci_epc_map_addr(epf->epc, epf->func_no,
+                                              epf->vfunc_no, dbase, &phys,
+                                              slen);
+                       if (IS_ERR(dst)) {
+                               pr_err("failed to map pci mmoery spact to local\n");

Typos.

drivers/pci/endpoint/functions/pci-epf-virtio.c line 437

Quoted 8 lines
drivers/pci/endpoint/functions/pci-epf-virtio.c new file +8 -0
+                               return PTR_ERR(dst);
+                       }
+               } else {
+                       src = pci_epc_map_addr(epf->epc, epf->func_no,
+                                              epf->vfunc_no, sbase, &phys,
+                                              slen);
+                       if (IS_ERR(src)) {
+                               pr_err("failed to map pci mmoery spact to local\n");

Typos.

Thanks

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#5 lore raw

RFC PATCH v2 2/3 virtio_pci: add a definition of queue flag in ISR

In reply to Shunsuke Mie

Already it has beed defined a config changed flag of ISR, but not the queue flag. Add a macro for it.

---
 include/uapi/linux/virtio_pci.h | 3 +++
 1 file changed, 3 insertions(+)
include/uapi/linux/virtio_pci.h +3 -0
index f703afc7ad31..d405bea27240 100644
@@ -94,6 +94,9 @@
 
 #endif /* VIRTIO_PCI_NO_LEGACY */
 
+/* Bits for ISR status field:only when if MSI-X is disabled */
+/* The bit of the ISR which indicates a queue entry update. */
+#define VIRTIO_PCI_ISR_QUEUE		0x1
 /* The bit of the ISR which indicates a device configuration change. */
 #define VIRTIO_PCI_ISR_CONFIG		0x2
 /* Vector value used to disable MSI for queue */
Signature · 1 line
2.25.1
Acked-by on v2 2/3
#6 lore raw

Re: [RFC PATCH v2 2/3] virtio_pci: add a definition of queue flag in ISR

Quoted 2 lines · Shunsuke Mie

Already it has beed defined a config changed flag of ISR, but not the queue

Typo should be "been".

Quoted 3 lines

flag. Add a macro for it.

Other than this,

Quoted 19 lines
---
 include/uapi/linux/virtio_pci.h | 3 +++
 1 file changed, 3 insertions(+)
include/uapi/linux/virtio_pci.h +3 -1
index f703afc7ad31..d405bea27240 100644
@@ -94,6 +94,9 @@
 
 #endif /* VIRTIO_PCI_NO_LEGACY */
 
+/* Bits for ISR status field:only when if MSI-X is disabled */
+/* The bit of the ISR which indicates a queue entry update. */
+#define VIRTIO_PCI_ISR_QUEUE           0x1
 /* The bit of the ISR which indicates a device configuration change. */
 #define VIRTIO_PCI_ISR_CONFIG          0x2
 /* Vector value used to disable MSI for queue */
--
Quoted 2 lines

2.25.1

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#7 lore raw

Re: [RFC PATCH v2 2/3] virtio_pci: add a definition of queue flag in ISR

2023年5月8日(月) 12:59 Jason Wang <[email protected]>:

Quoted 6 lines · Shunsuke Mie
Quoted 2 lines

Already it has beed defined a config changed flag of ISR, but not the queue

Typo should be "been".

Thanks for pointing that out.

Quoted 20 lines
Quoted 3 lines

flag. Add a macro for it.

Other than this,

Quoted 12 lines
---
 include/uapi/linux/virtio_pci.h | 3 +++
 1 file changed, 3 insertions(+)
include/uapi/linux/virtio_pci.h +0 -0
index f703afc7ad31..d405bea27240 100644
@@ -94,6 +94,9 @@
 
 #endif /* VIRTIO_PCI_NO_LEGACY */
 
Quoted 10 lines
Quoted 9 lines
include/uapi/linux/virtio_pci.h +3 -1
+/* Bits for ISR status field:only when if MSI-X is disabled */
+/* The bit of the ISR which indicates a queue entry update. */
+#define VIRTIO_PCI_ISR_QUEUE           0x1
 /* The bit of the ISR which indicates a device configuration change. */
 #define VIRTIO_PCI_ISR_CONFIG          0x2
 /* Vector value used to disable MSI for queue */
--

2.25.1

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#8 lore raw

RFC PATCH v2 3/3 PCI: endpoint: Add EP function driver to provide virtio-console functionality

In reply to Shunsuke Mie

Add a new PCIe endpoint function driver that works as a pci virtio-console device. The console connect to endpoint side console. It enables to communicate PCIe host and endpoint.

Architecture is following:

 ┌────────────┐         ┌──────────────────────┬────────────┐
 │virtioe     │         │                      │virtio      │
 │console drv │         ├───────────────┐      │console drv │
 ├────────────┤         │(virtio console│      ├────────────┤
 │ virtio bus │         │ device)       │◄────►│ virtio bus │
 ├────────────┤         ├---------------┤      └────────────┤
 │            │         │ pci ep virtio │                   │
 │  pci bus   │         │  console drv  │                   │
 │            │  pcie   ├───────────────┤                   │
 │            │ ◄─────► │  pci ep Bus   │                   │
 └────────────┘         └───────────────┴───────────────────┘
   PCIe Root              PCIe Endpoint

This driver has two roles. The first is as a PCIe endpoint virtio console function, which is implemented using the PCIe endpoint framework and PCIe EP virtio helpers. The second is as a virtual virtio console device connected to the virtio bus on PCIe endpoint Linux.

Communication between the two is achieved by copying the virtqueue data between PCIe root and endpoint, respectively.

This is a simple implementation and does not include features of virtio-console such as MULTIPORT, EMERG_WRITE, etc. As a result, each virtio console driver only displays /dev/hvc0.

As an example of usage, by setting getty to /dev/hvc0, it is possible to login to another host.

---
Changes from v2:
- Change to use copy functions between kiovs of pci-epf-virtio.

 drivers/pci/endpoint/functions/Kconfig        |  12 +
 drivers/pci/endpoint/functions/Makefile       |   1 +
 drivers/pci/endpoint/functions/pci-epf-vcon.c | 596 ++++++++++++++++++
 3 files changed, 609 insertions(+)
 create mode 100644 drivers/pci/endpoint/functions/pci-epf-vcon.c
drivers/pci/endpoint/functions/Kconfig +12 -0
index fa1a6a569a8f..9ce2698b67e1 100644
@@ -44,3 +44,15 @@ config PCI_EPF_VIRTIO
 	select VHOST_RING_IOMEM
 	help
 	  Helpers to implement PCI virtio Endpoint function
+
+config PCI_EPF_VCON
+	tristate "PCI Endpoint virito-console driver"
+	depends on PCI_ENDPOINT
+	select VHOST_RING
+	select PCI_EPF_VIRTIO
+	help
+	  PCIe Endpoint virtio-console function implementatino. This module
+	  enables to show the virtio-console as pci device to PCIe host side, and
+	  another virtual virtio-console device registers to endpoint system.
+	  Those devices are connected virtually and can communicate each other.
+
drivers/pci/endpoint/functions/Makefile +1 -0
index a96f127ce900..b4056689ce33 100644
@@ -7,3 +7,4 @@ obj-$(CONFIG_PCI_EPF_TEST)		+= pci-epf-test.o
 obj-$(CONFIG_PCI_EPF_NTB)		+= pci-epf-ntb.o
 obj-$(CONFIG_PCI_EPF_VNTB) 		+= pci-epf-vntb.o
 obj-$(CONFIG_PCI_EPF_VIRTIO)		+= pci-epf-virtio.o
+obj-$(CONFIG_PCI_EPF_VCON)		+= pci-epf-vcon.o
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +596 -0

596 lines not on this page · raw message

Signature · 6 lines
2.25.1

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
#9 lore raw

Re: [RFC PATCH v2 3/3] PCI: endpoint: Add EP function driver to provide virtio-console functionality

Random typos and trivial things. No need to repost until somebody does a more substantive review.

Quoted 3 lines · Shunsuke Mie

Add a new PCIe endpoint function driver that works as a pci virtio-console device. The console connect to endpoint side console. It enables to communicate PCIe host and endpoint.

s/pci/PCI/

Quoted 14 lines

Architecture is following:

 ┌────────────┐         ┌──────────────────────┬────────────┐
 │virtioe     │         │                      │virtio      │
 │console drv │         ├───────────────┐      │console drv │
 ├────────────┤         │(virtio console│      ├────────────┤
 │ virtio bus │         │ device)       │◄────►│ virtio bus │
 ├────────────┤         ├---------------┤      └────────────┤
 │            │         │ pci ep virtio │                   │
 │  pci bus   │         │  console drv  │                   │
 │            │  pcie   ├───────────────┤                   │
 │            │ ◄─────► │  pci ep Bus   │                   │
 └────────────┘         └───────────────┴───────────────────┘
   PCIe Root              PCIe Endpoint

s/virtioe/virtio/
s/pci/PCI/
s/pcie/PCIe/
s/ep/EP/

drivers/pci/endpoint/functions/Kconfig config PCI_EPF_VIRTIO · line 49

Quoted 2 lines
drivers/pci/endpoint/functions/Kconfig +2 -0
+config PCI_EPF_VCON
+	tristate "PCI Endpoint virito-console driver"

s/virito/virtio/

drivers/pci/endpoint/functions/Kconfig config PCI_EPF_VIRTIO · line 57

Quoted 8 lines
drivers/pci/endpoint/functions/Kconfig +8 -0
+	depends on PCI_ENDPOINT
+	select VHOST_RING
+	select PCI_EPF_VIRTIO
+	help
+	  PCIe Endpoint virtio-console function implementatino. This module
+	  enables to show the virtio-console as pci device to PCIe host side, and
+	  another virtual virtio-console device registers to endpoint system.
+	  Those devices are connected virtually and can communicate each other.

s/implementatino/implementation/
s/pci/PCI/
s/can communicate/can communicate with/

Quoted 2 lines

+ * PCI Endpoint function driver to impliment virtio-console device
+ * functionality.

s/impliment/implement/

drivers/pci/endpoint/functions/pci-epf-vcon.c line 16

Quoted 3 lines
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +3 -0
+static int virtio_queue_size = 0x100;
+module_param(virtio_queue_size, int, 0444);
+MODULE_PARM_DESC(virtio_queue_size, "A length of virtqueue");

When and why would users need this parameter? Where is it documented?

drivers/pci/endpoint/functions/pci-epf-vcon.c line 26

Quoted 1 line
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +1 -0
+	/* To access virtqueus of local host driver */

s/virtqueus/virtqueues/

drivers/pci/endpoint/functions/pci-epf-vcon.c line 38

Quoted 1 line
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +1 -0
+	/* To show a status whether this driver is ready and the remote is connected */

Make this fit in 80 columns.

drivers/pci/endpoint/functions/pci-epf-vcon.c line 59

Quoted 4 lines
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +4 -0
+	/* This is a minimum implementation. Doesn't support any features of
+	 * virtio console. It means driver and device use just 2 virtuque for tx
+	 * and rx.
+	 */

Use common multi-line comment style:

/*
* This is ...
*/

s/virtuque/virtqueues/

drivers/pci/endpoint/functions/pci-epf-vcon.c line 158

Quoted 4 lines
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +4 -0
+static void epf_vcon_raise_irq_handler(struct work_struct *work)
+{
+	struct epf_vcon *vcon =
+		container_of(work, struct epf_vcon, raise_irq_work);

Rewrap.

drivers/pci/endpoint/functions/pci-epf-vcon.c line 171

Quoted 7 lines
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +7 -0
+static int epf_vcon_setup_common(struct epf_vcon *vcon)
+{
+	vcon->features = 0;
+	vcon->connected = false;
+
+	vcon->task_wq =
+		alloc_workqueue("pci-epf-vcon/task-wq",

Looks like this would fit on the previous line?

drivers/pci/endpoint/functions/pci-epf-vcon.c line 206

Quoted 7 lines
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +6 -0
+				WQ_MEM_RECLAIM | WQ_HIGHPRI | WQ_UNBOUND, 0);
 
+static void epf_vcon_initialize_complete(void *param)
+{
+	struct epf_vcon *vcon = param;
+
+	pr_debug("Remote host has connected\n");

Is there any device info you could include here, e.g., with dev_dbg()? It's nice if users have a little context.

drivers/pci/endpoint/functions/pci-epf-vcon.c line 231

Quoted 8 lines
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +8 -0
+static int epf_vcon_setup_ep_func(struct epf_vcon *vcon, struct pci_epf *epf)
+{
+	int err;
+	struct epf_virtio *evio = &vcon->evio;
+	unsigned int nvq = epf_vcon_get_nvq(vcon);
+
+	vcon->rdev_iovs =
+		kmalloc_array(nvq, sizeof(vcon->rdev_iovs[0]), GFP_KERNEL);

Move the function name and as many parameters as fit in 80 columns to the previous line to match prevailing style.

drivers/pci/endpoint/functions/pci-epf-vcon.c line 303

Quoted 3 lines
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +3 -0
+	/* There is no config for virtio console because this console device
+	 * doesn't any support features
+	 */

Multi-line comment style.

s/doesn't any support/doesn't support any/? Is that what you mean?

drivers/pci/endpoint/functions/pci-epf-vcon.c line 311

Quoted 1 line
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +1 -0
+	/* Do nothing because this console device doesn't any support features */

Same.

drivers/pci/endpoint/functions/pci-epf-vcon.c line 322

Quoted 4 lines
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +4 -0
+static void epf_vcon_vdev_set_status(struct virtio_device *vdev, u8 status)
+{
+	if (status & VIRTIO_CONFIG_S_FAILED)
+		pr_debug("driver failed to setup this device\n");

dev_dbg() if possible.

drivers/pci/endpoint/functions/pci-epf-vcon.c line 388

Quoted 5 lines
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +5 -0
+		err = vringh_init_kern(&vcon->vdev_vrhs[i], vcon->features,
+				       virtio_queue_size, false, vring->desc,
+				       vring->avail, vring->used);
+		if (err) {
+			pr_err("failed to init vringh for vring %d\n", i);

dev_err() if possible.

drivers/pci/endpoint/functions/pci-epf-vcon.c line 444

Quoted 8 lines
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +8 -0
+static int epf_vcon_setup_vdev(struct epf_vcon *vcon, struct device *parent)
+{
+	int err;
+	struct virtio_device *vdev = &vcon->vdev;
+	const unsigned int nvq = epf_vcon_get_nvq(vcon);
+
+	vcon->vdev_vrhs =
+		kmalloc_array(nvq, sizeof(vcon->vdev_vrhs[0]), GFP_KERNEL);

Rewrap.

drivers/pci/endpoint/functions/pci-epf-vcon.c line 449

Quoted 2 lines
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +2 -0
+	vcon->vdev_iovs =
+		kmalloc_array(nvq, sizeof(vcon->vdev_iovs[0]), GFP_KERNEL);

Rewrap.

drivers/pci/endpoint/functions/pci-epf-vcon.c line 459

Quoted 2 lines
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +2 -0
+	vcon->vdev_vqs =
+		kmalloc_array(nvq, sizeof(vcon->vdev_vrhs[0]), GFP_KERNEL);

Rewrap.

drivers/pci/endpoint/functions/pci-epf-vcon.c line 497

Quoted 6 lines
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +6 -0
+static void epf_vcon_cleanup_vdev(struct epf_vcon *vcon)
+{
+	unregister_virtio_device(&vcon->vdev);
+	/* Cleanup struct virtio_device that has kobject, otherwise error occures when
+	 * reregister the virtio device.
+	 */

Multi-line style and rewrap to fit in 80 columns.

drivers/pci/endpoint/functions/pci-epf-vcon.c line 583

Quoted 7 lines
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +7 -0
+static int __init epf_vcon_init(void)
+{
+	int err;
+
+	err = pci_epf_register_driver(&epf_vcon_drv);
+	if (err)
+		pr_err("Failed to register PCI EP virtio-console function\n");

dev_err() if possible (doesn't look like it *is* possible).

Looks like this registers a *driver*, so maybe change the message from "function" to "driver"?

Bjorn

#10 lore raw

Re: [RFC PATCH v2 3/3] PCI: endpoint: Add EP function driver to provide virtio-console functionality

Hi Bjorn,
Thanks for the many comments. I will fix the mannerisms and typos as noted.

2023年4月28日(金) 3:09 Bjorn Helgaas <[email protected]>:

Quoted 73 lines

Random typos and trivial things. No need to repost until somebody does a more substantive review.

Quoted 3 lines · Shunsuke Mie

Add a new PCIe endpoint function driver that works as a pci virtio-console device. The console connect to endpoint side console. It enables to communicate PCIe host and endpoint.

s/pci/PCI/

Quoted 14 lines

Architecture is following:

 ┌────────────┐         ┌──────────────────────┬────────────┐
 │virtioe     │         │                      │virtio      │
 │console drv │         ├───────────────┐      │console drv │
 ├────────────┤         │(virtio console│      ├────────────┤
 │ virtio bus │         │ device)       │◄────►│ virtio bus │
 ├────────────┤         ├---------------┤      └────────────┤
 │            │         │ pci ep virtio │                   │
 │  pci bus   │         │  console drv  │                   │
 │            │  pcie   ├───────────────┤                   │
 │            │ ◄─────► │  pci ep Bus   │                   │
 └────────────┘         └───────────────┴───────────────────┘
   PCIe Root              PCIe Endpoint

s/virtioe/virtio/
s/pci/PCI/
s/pcie/PCIe/
s/ep/EP/

drivers/pci/endpoint/functions/Kconfig config PCI_EPF_VIRTIO · line 49

Quoted 2 lines
drivers/pci/endpoint/functions/Kconfig +2 -0
+config PCI_EPF_VCON
+     tristate "PCI Endpoint virito-console driver"

s/virito/virtio/

drivers/pci/endpoint/functions/Kconfig config PCI_EPF_VIRTIO · line 57

Quoted 8 lines
drivers/pci/endpoint/functions/Kconfig +8 -0
+     depends on PCI_ENDPOINT
+     select VHOST_RING
+     select PCI_EPF_VIRTIO
+     help
+       PCIe Endpoint virtio-console function implementatino. This module
+       enables to show the virtio-console as pci device to PCIe host side, and
+       another virtual virtio-console device registers to endpoint system.
+       Those devices are connected virtually and can communicate each other.

s/implementatino/implementation/
s/pci/PCI/
s/can communicate/can communicate with/

Quoted 2 lines

+ * PCI Endpoint function driver to impliment virtio-console device
+ * functionality.

s/impliment/implement/

drivers/pci/endpoint/functions/pci-epf-vcon.c line 16

Quoted 3 lines
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +3 -0
+static int virtio_queue_size = 0x100;
+module_param(virtio_queue_size, int, 0444);
+MODULE_PARM_DESC(virtio_queue_size, "A length of virtqueue");

When and why would users need this parameter? Where is it documented?

Quoted 1 line

+ /* To access virtqueus of local host driver */

s/virtqueus/virtqueues/

Quoted 1 line

+ /* To show a status whether this driver is ready and the remote is connected */

Make this fit in 80 columns.

drivers/pci/endpoint/functions/pci-epf-vcon.c line 59

Quoted 4 lines
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +4 -0
+     /* This is a minimum implementation. Doesn't support any features of
+      * virtio console. It means driver and device use just 2 virtuque for tx
+      * and rx.
+      */
Quoted 5 lines

Use common multi-line comment style:

/*
* This is ...
*/

I'll follow the style.

Quoted 21 lines

s/virtuque/virtqueues/

drivers/pci/endpoint/functions/pci-epf-vcon.c line 158

Quoted 4 lines
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +4 -0
+static void epf_vcon_raise_irq_handler(struct work_struct *work)
+{
+     struct epf_vcon *vcon =
+             container_of(work, struct epf_vcon, raise_irq_work);

Rewrap.

drivers/pci/endpoint/functions/pci-epf-vcon.c line 171

Quoted 7 lines
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +7 -0
+static int epf_vcon_setup_common(struct epf_vcon *vcon)
+{
+     vcon->features = 0;
+     vcon->connected = false;
+
+     vcon->task_wq =
+             alloc_workqueue("pci-epf-vcon/task-wq",

Looks like this would fit on the previous line?

Quoted 1 line

+ WQ_MEM_RECLAIM | WQ_HIGHPRI | WQ_UNBOUND, 0);

Quoted 8 lines

drivers/pci/endpoint/functions/pci-epf-vcon.c line 206

Quoted 5 lines
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +5 -0
+static void epf_vcon_initialize_complete(void *param)
+{
+     struct epf_vcon *vcon = param;
+
+     pr_debug("Remote host has connected\n");

Is there any device info you could include here, e.g., with dev_dbg()? It's nice if users have a little context.

I see. I'll use it.

Quoted 11 lines

drivers/pci/endpoint/functions/pci-epf-vcon.c line 231

Quoted 8 lines
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +8 -0
+static int epf_vcon_setup_ep_func(struct epf_vcon *vcon, struct pci_epf *epf)
+{
+     int err;
+     struct epf_virtio *evio = &vcon->evio;
+     unsigned int nvq = epf_vcon_get_nvq(vcon);
+
+     vcon->rdev_iovs =
+             kmalloc_array(nvq, sizeof(vcon->rdev_iovs[0]), GFP_KERNEL);

Move the function name and as many parameters as fit in 80 columns to the previous line to match prevailing style.

I've just applied clang-format... Ok, I'll fix it manually.

Quoted 67 lines

drivers/pci/endpoint/functions/pci-epf-vcon.c line 303

Quoted 3 lines
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +3 -0
+     /* There is no config for virtio console because this console device
+      * doesn't any support features
+      */

Multi-line comment style.

s/doesn't any support/doesn't support any/? Is that what you mean?

Quoted 1 line

+ /* Do nothing because this console device doesn't any support features */

Same.

drivers/pci/endpoint/functions/pci-epf-vcon.c line 322

Quoted 4 lines
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +4 -0
+static void epf_vcon_vdev_set_status(struct virtio_device *vdev, u8 status)
+{
+     if (status & VIRTIO_CONFIG_S_FAILED)
+             pr_debug("driver failed to setup this device\n");

dev_dbg() if possible.

drivers/pci/endpoint/functions/pci-epf-vcon.c line 388

Quoted 5 lines
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +5 -0
+             err = vringh_init_kern(&vcon->vdev_vrhs[i], vcon->features,
+                                    virtio_queue_size, false, vring->desc,
+                                    vring->avail, vring->used);
+             if (err) {
+                     pr_err("failed to init vringh for vring %d\n", i);

dev_err() if possible.

drivers/pci/endpoint/functions/pci-epf-vcon.c line 444

Quoted 8 lines
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +8 -0
+static int epf_vcon_setup_vdev(struct epf_vcon *vcon, struct device *parent)
+{
+     int err;
+     struct virtio_device *vdev = &vcon->vdev;
+     const unsigned int nvq = epf_vcon_get_nvq(vcon);
+
+     vcon->vdev_vrhs =
+             kmalloc_array(nvq, sizeof(vcon->vdev_vrhs[0]), GFP_KERNEL);

Rewrap.

drivers/pci/endpoint/functions/pci-epf-vcon.c line 449

Quoted 2 lines
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +2 -0
+     vcon->vdev_iovs =
+             kmalloc_array(nvq, sizeof(vcon->vdev_iovs[0]), GFP_KERNEL);

Rewrap.

drivers/pci/endpoint/functions/pci-epf-vcon.c line 459

Quoted 2 lines
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +2 -0
+     vcon->vdev_vqs =
+             kmalloc_array(nvq, sizeof(vcon->vdev_vrhs[0]), GFP_KERNEL);

Rewrap.

drivers/pci/endpoint/functions/pci-epf-vcon.c line 497

Quoted 6 lines
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +6 -0
+static void epf_vcon_cleanup_vdev(struct epf_vcon *vcon)
+{
+     unregister_virtio_device(&vcon->vdev);
+     /* Cleanup struct virtio_device that has kobject, otherwise error occures when
+      * reregister the virtio device.
+      */

Multi-line style and rewrap to fit in 80 columns.

drivers/pci/endpoint/functions/pci-epf-vcon.c line 583

Quoted 7 lines
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +7 -0
+static int __init epf_vcon_init(void)
+{
+     int err;
+
+     err = pci_epf_register_driver(&epf_vcon_drv);
+     if (err)
+             pr_err("Failed to register PCI EP virtio-console function\n");

dev_err() if possible (doesn't look like it *is* possible).

Quoted 2 lines

Looks like this registers a *driver*, so maybe change the message from "function" to "driver"?

It should be "driver".
https://docs.kernel.org/PCI/endpoint/pci-endpoint.html#epc-apis-for-the-pci-endpoint-function-driver

Quoted 1 line

Bjorn

Best regards,
Shunsuke
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#11 lore raw

Re: [RFC PATCH v2 3/3] PCI: endpoint: Add EP function driver to provide virtio-console functionality

In reply to Shunsuke Mie
Quoted 7 lines · Shunsuke Mie

Add a new PCIe endpoint function driver that works as a pci virtio-console device. The console connect to endpoint side console. It enables to communicate PCIe host and endpoint.

Architecture is following:

Quoted 13 lines
 ┌────────────┐         ┌──────────────────────┬────────────┐
 │virtioe     │         │                      │virtio      │
 │console drv │         ├───────────────┐      │console drv │
 ├────────────┤         │(virtio console│      ├────────────┤
 │ virtio bus │         │ device)       │◄────►│ virtio bus │
 ├────────────┤         ├---------------┤      └────────────┤
 │            │         │ pci ep virtio │                   │
 │  pci bus   │         │  console drv  │                   │
 │            │  pcie   ├───────────────┤                   │
 │            │ ◄─────► │  pci ep Bus   │                   │
 └────────────┘         └───────────────┴───────────────────┘
   PCIe Root              PCIe Endpoint

I think it might only works for peer devices like:

net, console or vsock.

So there're many choices here, I'd like to know what's the reason for you to implement a mediation.

An alternative is to implement a dedicated net, console and vsock driver for vringh (CAIF somehow works like this). This would have better performance.

Quoted 98 lines

This driver has two roles. The first is as a PCIe endpoint virtio console function, which is implemented using the PCIe endpoint framework and PCIe EP virtio helpers. The second is as a virtual virtio console device connected to the virtio bus on PCIe endpoint Linux.

Communication between the two is achieved by copying the virtqueue data between PCIe root and endpoint, respectively.

This is a simple implementation and does not include features of virtio-console such as MULTIPORT, EMERG_WRITE, etc. As a result, each virtio console driver only displays /dev/hvc0.

As an example of usage, by setting getty to /dev/hvc0, it is possible to login to another host.

---
Changes from v2:
- Change to use copy functions between kiovs of pci-epf-virtio.

 drivers/pci/endpoint/functions/Kconfig        |  12 +
 drivers/pci/endpoint/functions/Makefile       |   1 +
 drivers/pci/endpoint/functions/pci-epf-vcon.c | 596 ++++++++++++++++++
 3 files changed, 609 insertions(+)
 create mode 100644 drivers/pci/endpoint/functions/pci-epf-vcon.c
drivers/pci/endpoint/functions/Kconfig +12 -0
index fa1a6a569a8f..9ce2698b67e1 100644
@@ -44,3 +44,15 @@ config PCI_EPF_VIRTIO
        select VHOST_RING_IOMEM
        help
          Helpers to implement PCI virtio Endpoint function
+
+config PCI_EPF_VCON
+       tristate "PCI Endpoint virito-console driver"
+       depends on PCI_ENDPOINT
+       select VHOST_RING
+       select PCI_EPF_VIRTIO
+       help
+         PCIe Endpoint virtio-console function implementatino. This module
+         enables to show the virtio-console as pci device to PCIe host side, and
+         another virtual virtio-console device registers to endpoint system.
+         Those devices are connected virtually and can communicate each other.
+
drivers/pci/endpoint/functions/Makefile +1 -0
index a96f127ce900..b4056689ce33 100644
@@ -7,3 +7,4 @@ obj-$(CONFIG_PCI_EPF_TEST)              += pci-epf-test.o
 obj-$(CONFIG_PCI_EPF_NTB)              += pci-epf-ntb.o
 obj-$(CONFIG_PCI_EPF_VNTB)             += pci-epf-vntb.o
 obj-$(CONFIG_PCI_EPF_VIRTIO)           += pci-epf-virtio.o
+obj-$(CONFIG_PCI_EPF_VCON)             += pci-epf-vcon.o
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +37 -0
new file mode 100644
index 000000000000..31f4247cd10f
@@ -0,0 +1,596 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * PCI Endpoint function driver to impliment virtio-console device
+ * functionality.
+ */
+#include <linux/pci-epf.h>
+#include <linux/virtio_ids.h>
+#include <linux/virtio_pci.h>
+#include <linux/virtio_console.h>
+#include <linux/virtio_ring.h>
+
+#include "pci-epf-virtio.h"
+
+static int virtio_queue_size = 0x100;
+module_param(virtio_queue_size, int, 0444);
+MODULE_PARM_DESC(virtio_queue_size, "A length of virtqueue");
+
+struct epf_vcon {
+       /* To access virtqueues on remote host */
+       struct epf_virtio evio;
+       struct vringh_kiov *rdev_iovs;
+
+       /* To register a local virtio bus */
+       struct virtio_device vdev;
+
+       /* To access virtqueus of local host driver */
+       struct vringh *vdev_vrhs;
+       struct vringh_kiov *vdev_iovs;
+       struct virtqueue **vdev_vqs;
+
+       /* For transportation and notification */
+       struct workqueue_struct *task_wq;
+       struct work_struct raise_irq_work, rx_work, tx_work;
+
+       /* To retain virtio features. It is commonly used local and remote. */
+       u64 features;
+

drivers/pci/endpoint/functions/pci-epf-vcon.c line 47

Quoted 10 lines
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +10 -0
+       /* To show a status whether this driver is ready and the remote is connected */
+       bool connected;
+};
+
+enum {
+       VCON_VIRTQUEUE_RX,
+       VCON_VIRTQUEUE_TX,
+       // Should be end of enum
+       VCON_VIRTQUEUE_NUM
+};

It would be better if we can split the console specific thing out, then it allows us to do ethernet and vsock in the future.

Thanks

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#12 lore raw

Re: [RFC PATCH v2 3/3] PCI: endpoint: Add EP function driver to provide virtio-console functionality

Hi Json,
2023年5月8日(月) 13:03 Jason Wang <[email protected]>:

Quoted 23 lines · Shunsuke Mie
Quoted 7 lines

Add a new PCIe endpoint function driver that works as a pci virtio-console device. The console connect to endpoint side console. It enables to communicate PCIe host and endpoint.

Architecture is following:

Quoted 13 lines
 ┌────────────┐         ┌──────────────────────┬────────────┐
 │virtioe     │         │                      │virtio      │
 │console drv │         ├───────────────┐      │console drv │
 ├────────────┤         │(virtio console│      ├────────────┤
 │ virtio bus │         │ device)       │◄────►│ virtio bus │
 ├────────────┤         ├---------------┤      └────────────┤
 │            │         │ pci ep virtio │                   │
 │  pci bus   │         │  console drv  │                   │
 │            │  pcie   ├───────────────┤                   │
 │            │ ◄─────► │  pci ep Bus   │                   │
 └────────────┘         └───────────────┴───────────────────┘
   PCIe Root              PCIe Endpoint
Quoted 3 lines

I think it might only works for peer devices like:

net, console or vsock.

Could you tell me what "peer devices" means?

Quoted 6 lines

So there're many choices here, I'd like to know what's the reason for you to implement a mediation.

An alternative is to implement a dedicated net, console and vsock driver for vringh (CAIF somehow works like this). This would have better performance.

Does it mean that the driver also functions as a network driver directly?

Quoted 105 lines
Quoted 105 lines

This driver has two roles. The first is as a PCIe endpoint virtio console function, which is implemented using the PCIe endpoint framework and PCIe EP virtio helpers. The second is as a virtual virtio console device connected to the virtio bus on PCIe endpoint Linux.

Communication between the two is achieved by copying the virtqueue data between PCIe root and endpoint, respectively.

This is a simple implementation and does not include features of virtio-console such as MULTIPORT, EMERG_WRITE, etc. As a result, each virtio console driver only displays /dev/hvc0.

As an example of usage, by setting getty to /dev/hvc0, it is possible to login to another host.

---
Changes from v2:
- Change to use copy functions between kiovs of pci-epf-virtio.

 drivers/pci/endpoint/functions/Kconfig        |  12 +
 drivers/pci/endpoint/functions/Makefile       |   1 +
 drivers/pci/endpoint/functions/pci-epf-vcon.c | 596 ++++++++++++++++++
 3 files changed, 609 insertions(+)
 create mode 100644 drivers/pci/endpoint/functions/pci-epf-vcon.c
drivers/pci/endpoint/functions/Kconfig +12 -0
index fa1a6a569a8f..9ce2698b67e1 100644
@@ -44,3 +44,15 @@ config PCI_EPF_VIRTIO
        select VHOST_RING_IOMEM
        help
          Helpers to implement PCI virtio Endpoint function
+
+config PCI_EPF_VCON
+       tristate "PCI Endpoint virito-console driver"
+       depends on PCI_ENDPOINT
+       select VHOST_RING
+       select PCI_EPF_VIRTIO
+       help
+         PCIe Endpoint virtio-console function implementatino. This module
+         enables to show the virtio-console as pci device to PCIe host side, and
+         another virtual virtio-console device registers to endpoint system.
+         Those devices are connected virtually and can communicate each other.
+
drivers/pci/endpoint/functions/Makefile +1 -0
index a96f127ce900..b4056689ce33 100644
@@ -7,3 +7,4 @@ obj-$(CONFIG_PCI_EPF_TEST)              += pci-epf-test.o
 obj-$(CONFIG_PCI_EPF_NTB)              += pci-epf-ntb.o
 obj-$(CONFIG_PCI_EPF_VNTB)             += pci-epf-vntb.o
 obj-$(CONFIG_PCI_EPF_VIRTIO)           += pci-epf-virtio.o
+obj-$(CONFIG_PCI_EPF_VCON)             += pci-epf-vcon.o
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +42 -0
new file mode 100644
index 000000000000..31f4247cd10f
@@ -0,0 +1,596 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * PCI Endpoint function driver to impliment virtio-console device
+ * functionality.
+ */
+#include <linux/pci-epf.h>
+#include <linux/virtio_ids.h>
+#include <linux/virtio_pci.h>
+#include <linux/virtio_console.h>
+#include <linux/virtio_ring.h>
+
+#include "pci-epf-virtio.h"
+
+static int virtio_queue_size = 0x100;
+module_param(virtio_queue_size, int, 0444);
+MODULE_PARM_DESC(virtio_queue_size, "A length of virtqueue");
+
+struct epf_vcon {
+       /* To access virtqueues on remote host */
+       struct epf_virtio evio;
+       struct vringh_kiov *rdev_iovs;
+
+       /* To register a local virtio bus */
+       struct virtio_device vdev;
+
+       /* To access virtqueus of local host driver */
+       struct vringh *vdev_vrhs;
+       struct vringh_kiov *vdev_iovs;
+       struct virtqueue **vdev_vqs;
+
+       /* For transportation and notification */
+       struct workqueue_struct *task_wq;
+       struct work_struct raise_irq_work, rx_work, tx_work;
+
+       /* To retain virtio features. It is commonly used local and remote. */
+       u64 features;
+
+       /* To show a status whether this driver is ready and the remote is connected */
+       bool connected;
+};
+
+enum {
Quoted 8 lines

drivers/pci/endpoint/functions/pci-epf-vcon.c line 47

Quoted 5 lines
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +5 -0
+       VCON_VIRTQUEUE_RX,
+       VCON_VIRTQUEUE_TX,
+       // Should be end of enum
+       VCON_VIRTQUEUE_NUM
+};

It would be better if we can split the console specific thing out, then it allows us to do ethernet and vsock in the future.

I'm planning to implement each virtio device in a separate file.
https://lwn.net/Articles/922124/

Quoted 2 lines

Thanks

Best regards,
Shunsuke

#13 lore raw

Re: [RFC PATCH v2 3/3] PCI: endpoint: Add EP function driver to provide virtio-console functionality

Gentle ping ...

Thanks,

Shunsuke.

Quoted 154 lines · Shunsuke Mie

Hi Json,
2023年5月8日(月) 13:03 Jason Wang <[email protected]>:

Quoted 20 lines · Shunsuke Mie
Quoted 6 lines

Add a new PCIe endpoint function driver that works as a pci virtio-console device. The console connect to endpoint side console. It enables to communicate PCIe host and endpoint.

Architecture is following:

Quoted 13 lines
  ┌────────────┐         ┌──────────────────────┬────────────┐
  │virtioe     │         │                      │virtio      │
  │console drv │         ├───────────────┐      │console drv │
  ├────────────┤         │(virtio console│      ├────────────┤
  │ virtio bus │         │ device)       │◄────►│ virtio bus │
  ├────────────┤         ├---------------┤      └────────────┤
  │            │         │ pci ep virtio │                   │
  │  pci bus   │         │  console drv  │                   │
  │            │  pcie   ├───────────────┤                   │
  │            │ ◄─────► │  pci ep Bus   │                   │
  └────────────┘         └───────────────┴───────────────────┘
    PCIe Root              PCIe Endpoint
Quoted 3 lines

I think it might only works for peer devices like:

net, console or vsock.

Could you tell me what "peer devices" means?

Quoted 6 lines

So there're many choices here, I'd like to know what's the reason for you to implement a mediation.

An alternative is to implement a dedicated net, console and vsock driver for vringh (CAIF somehow works like this). This would have better performance.

Does it mean that the driver also functions as a network driver directly?

Quoted 103 lines
Quoted 103 lines

This driver has two roles. The first is as a PCIe endpoint virtio console function, which is implemented using the PCIe endpoint framework and PCIe EP virtio helpers. The second is as a virtual virtio console device connected to the virtio bus on PCIe endpoint Linux.

Communication between the two is achieved by copying the virtqueue data between PCIe root and endpoint, respectively.

This is a simple implementation and does not include features of virtio-console such as MULTIPORT, EMERG_WRITE, etc. As a result, each virtio console driver only displays /dev/hvc0.

As an example of usage, by setting getty to /dev/hvc0, it is possible to login to another host.

---
Changes from v2:
- Change to use copy functions between kiovs of pci-epf-virtio.

  drivers/pci/endpoint/functions/Kconfig        |  12 +
  drivers/pci/endpoint/functions/Makefile       |   1 +
  drivers/pci/endpoint/functions/pci-epf-vcon.c | 596 ++++++++++++++++++
  3 files changed, 609 insertions(+)

create mode 100644 drivers/pci/endpoint/functions/pci-epf-vcon.c

drivers/pci/endpoint/functions/Kconfig +12 -0
index fa1a6a569a8f..9ce2698b67e1 100644
@@ -44,3 +44,15 @@ config PCI_EPF_VIRTIO
         select VHOST_RING_IOMEM
         help
           Helpers to implement PCI virtio Endpoint function
+
+config PCI_EPF_VCON
+       tristate "PCI Endpoint virito-console driver"
+       depends on PCI_ENDPOINT
+       select VHOST_RING
+       select PCI_EPF_VIRTIO
+       help
+         PCIe Endpoint virtio-console function implementatino. This module
+         enables to show the virtio-console as pci device to PCIe host side, and
+         another virtual virtio-console device registers to endpoint system.
+         Those devices are connected virtually and can communicate each other.
+
drivers/pci/endpoint/functions/Makefile +1 -0
index a96f127ce900..b4056689ce33 100644
@@ -7,3 +7,4 @@ obj-$(CONFIG_PCI_EPF_TEST)              += pci-epf-test.o
  obj-$(CONFIG_PCI_EPF_NTB)              += pci-epf-ntb.o
  obj-$(CONFIG_PCI_EPF_VNTB)             += pci-epf-vntb.o
  obj-$(CONFIG_PCI_EPF_VIRTIO)           += pci-epf-virtio.o
+obj-$(CONFIG_PCI_EPF_VCON)             += pci-epf-vcon.o
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +41 -0
new file mode 100644
index 000000000000..31f4247cd10f
@@ -0,0 +1,596 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * PCI Endpoint function driver to impliment virtio-console device
+ * functionality.
+ */
+#include <linux/pci-epf.h>
+#include <linux/virtio_ids.h>
+#include <linux/virtio_pci.h>
+#include <linux/virtio_console.h>
+#include <linux/virtio_ring.h>
+
+#include "pci-epf-virtio.h"
+
+static int virtio_queue_size = 0x100;
+module_param(virtio_queue_size, int, 0444);
+MODULE_PARM_DESC(virtio_queue_size, "A length of virtqueue");
+
+struct epf_vcon {
+       /* To access virtqueues on remote host */
+       struct epf_virtio evio;
+       struct vringh_kiov *rdev_iovs;
+
+       /* To register a local virtio bus */
+       struct virtio_device vdev;
+
+       /* To access virtqueus of local host driver */
+       struct vringh *vdev_vrhs;
+       struct vringh_kiov *vdev_iovs;
+       struct virtqueue **vdev_vqs;
+
+       /* For transportation and notification */
+       struct workqueue_struct *task_wq;
+       struct work_struct raise_irq_work, rx_work, tx_work;
+
+       /* To retain virtio features. It is commonly used local and remote. */
+       u64 features;
+
+       /* To show a status whether this driver is ready and the remote is connected */
+       bool connected;
+};
+
Quoted 8 lines

drivers/pci/endpoint/functions/pci-epf-vcon.c line 47

Quoted 6 lines
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +6 -0
+enum {
+       VCON_VIRTQUEUE_RX,
+       VCON_VIRTQUEUE_TX,
+       // Should be end of enum
+       VCON_VIRTQUEUE_NUM
+};

It would be better if we can split the console specific thing out, then it allows us to do ethernet and vsock in the future.

I'm planning to implement each virtio device in a separate file.
https://lwn.net/Articles/922124/

Quoted 2 lines

Thanks

Best regards,
Shunsuke

#14 lore raw

Re: [RFC PATCH v2 3/3] PCI: endpoint: Add EP function driver to provide virtio-console functionality

Quoted 35 lines · Shunsuke Mie

Gentle ping ...

Thanks,

Shunsuke.

Quoted 26 lines · Shunsuke Mie

Hi Json,
2023年5月8日(月) 13:03 Jason Wang <[email protected]>:

Quoted 20 lines · Shunsuke Mie
Quoted 6 lines

Add a new PCIe endpoint function driver that works as a pci virtio-console device. The console connect to endpoint side console. It enables to communicate PCIe host and endpoint.

Architecture is following:

Quoted 13 lines
  ┌────────────┐         ┌──────────────────────┬────────────┐
  │virtioe     │         │                      │virtio      │
  │console drv │         ├───────────────┐      │console drv │
  ├────────────┤         │(virtio console│      ├────────────┤
  │ virtio bus │         │ device)       │◄────►│ virtio bus │
  ├────────────┤         ├---------------┤      └────────────┤
  │            │         │ pci ep virtio │                   │
  │  pci bus   │         │  console drv  │                   │
  │            │  pcie   ├───────────────┤                   │
  │            │ ◄─────► │  pci ep Bus   │                   │
  └────────────┘         └───────────────┴───────────────────┘
    PCIe Root              PCIe Endpoint
Quoted 3 lines

I think it might only works for peer devices like:

net, console or vsock.

Could you tell me what "peer devices" means?

I meant, for example we know in the case of virtio-net, TX can talk with RX belonging to another device directly.

But this is not the case for other devices like virito-blk.

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

So there're many choices here, I'd like to know what's the reason for you to implement a mediation.

An alternative is to implement a dedicated net, console and vsock driver for vringh (CAIF somehow works like this). This would have better performance.

Does it mean that the driver also functions as a network driver directly?

I meant, e.g in the case of networking, you can have a dedicated driver with two vringh in the endpoint side.

The benefit is the performance, no need for the (datapath) mediation.

But if we don't care about the performance, this proposal seems to be fine.

Thanks

Quoted 116 lines
Quoted 116 lines
Quoted 103 lines
Quoted 103 lines

This driver has two roles. The first is as a PCIe endpoint virtio console function, which is implemented using the PCIe endpoint framework and PCIe EP virtio helpers. The second is as a virtual virtio console device connected to the virtio bus on PCIe endpoint Linux.

Communication between the two is achieved by copying the virtqueue data between PCIe root and endpoint, respectively.

This is a simple implementation and does not include features of virtio-console such as MULTIPORT, EMERG_WRITE, etc. As a result, each virtio console driver only displays /dev/hvc0.

As an example of usage, by setting getty to /dev/hvc0, it is possible to login to another host.

---
Changes from v2:
- Change to use copy functions between kiovs of pci-epf-virtio.

  drivers/pci/endpoint/functions/Kconfig        |  12 +
  drivers/pci/endpoint/functions/Makefile       |   1 +
  drivers/pci/endpoint/functions/pci-epf-vcon.c | 596 ++++++++++++++++++
  3 files changed, 609 insertions(+)

create mode 100644 drivers/pci/endpoint/functions/pci-epf-vcon.c

drivers/pci/endpoint/functions/Kconfig +12 -0
index fa1a6a569a8f..9ce2698b67e1 100644
@@ -44,3 +44,15 @@ config PCI_EPF_VIRTIO
         select VHOST_RING_IOMEM
         help
           Helpers to implement PCI virtio Endpoint function
+
+config PCI_EPF_VCON
+       tristate "PCI Endpoint virito-console driver"
+       depends on PCI_ENDPOINT
+       select VHOST_RING
+       select PCI_EPF_VIRTIO
+       help
+         PCIe Endpoint virtio-console function implementatino. This module
+         enables to show the virtio-console as pci device to PCIe host side, and
+         another virtual virtio-console device registers to endpoint system.
+         Those devices are connected virtually and can communicate each other.
+
drivers/pci/endpoint/functions/Makefile +1 -0
index a96f127ce900..b4056689ce33 100644
@@ -7,3 +7,4 @@ obj-$(CONFIG_PCI_EPF_TEST)              += pci-epf-test.o
  obj-$(CONFIG_PCI_EPF_NTB)              += pci-epf-ntb.o
  obj-$(CONFIG_PCI_EPF_VNTB)             += pci-epf-vntb.o
  obj-$(CONFIG_PCI_EPF_VIRTIO)           += pci-epf-virtio.o
+obj-$(CONFIG_PCI_EPF_VCON)             += pci-epf-vcon.o
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +41 -0
new file mode 100644
index 000000000000..31f4247cd10f
@@ -0,0 +1,596 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * PCI Endpoint function driver to impliment virtio-console device
+ * functionality.
+ */
+#include <linux/pci-epf.h>
+#include <linux/virtio_ids.h>
+#include <linux/virtio_pci.h>
+#include <linux/virtio_console.h>
+#include <linux/virtio_ring.h>
+
+#include "pci-epf-virtio.h"
+
+static int virtio_queue_size = 0x100;
+module_param(virtio_queue_size, int, 0444);
+MODULE_PARM_DESC(virtio_queue_size, "A length of virtqueue");
+
+struct epf_vcon {
+       /* To access virtqueues on remote host */
+       struct epf_virtio evio;
+       struct vringh_kiov *rdev_iovs;
+
+       /* To register a local virtio bus */
+       struct virtio_device vdev;
+
+       /* To access virtqueus of local host driver */
+       struct vringh *vdev_vrhs;
+       struct vringh_kiov *vdev_iovs;
+       struct virtqueue **vdev_vqs;
+
+       /* For transportation and notification */
+       struct workqueue_struct *task_wq;
+       struct work_struct raise_irq_work, rx_work, tx_work;
+
+       /* To retain virtio features. It is commonly used local and remote. */
+       u64 features;
+
+       /* To show a status whether this driver is ready and the remote is connected */
+       bool connected;
+};
+
Quoted 8 lines

drivers/pci/endpoint/functions/pci-epf-vcon.c line 47

Quoted 6 lines
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +6 -0
+enum {
+       VCON_VIRTQUEUE_RX,
+       VCON_VIRTQUEUE_TX,
+       // Should be end of enum
+       VCON_VIRTQUEUE_NUM
+};

It would be better if we can split the console specific thing out, then it allows us to do ethernet and vsock in the future.

I'm planning to implement each virtio device in a separate file.
https://lwn.net/Articles/922124/

Quoted 5 lines
Quoted 4 lines
Quoted 2 lines

Thanks

Best regards,
Shunsuke

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#15 lore raw

Re: [RFC PATCH v2 3/3] PCI: endpoint: Add EP function driver to provide virtio-console functionality

I'm sorry for late response.

Quoted 31 lines · Jason Wang
Quoted 30 lines · Shunsuke Mie

Gentle ping ...

Thanks,

Shunsuke.

Quoted 22 lines · Shunsuke Mie

Hi Json,
2023年5月8日(月) 13:03 Jason Wang <[email protected]>:

Quoted 20 lines · Shunsuke Mie
Quoted 19 lines

Add a new PCIe endpoint function driver that works as a pci virtio-console device. The console connect to endpoint side console. It enables to communicate PCIe host and endpoint.

Architecture is following:

   ┌────────────┐         ┌──────────────────────┬────────────┐
   │virtioe     │         │                      │virtio      │
   │console drv │         ├───────────────┐      │console drv │
   ├────────────┤         │(virtio console│      ├────────────┤
   │ virtio bus │         │ device)       │◄────►│ virtio bus │
   ├────────────┤         ├---------------┤      └────────────┤
   │            │         │ pci ep virtio │                   │
   │  pci bus   │         │  console drv  │                   │
   │            │  pcie   ├───────────────┤                   │
   │            │ ◄─────► │  pci ep Bus   │                   │
   └────────────┘         └───────────────┴───────────────────┘
     PCIe Root              PCIe Endpoint
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

I think it might only works for peer devices like:

net, console or vsock.

Could you tell me what "peer devices" means?

I meant, for example we know in the case of virtio-net, TX can talk with RX belonging to another device directly.

But this is not the case for other devices like virito-blk.

Thank you. I comprehended it.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

So there're many choices here, I'd like to know what's the reason for you to implement a mediation.

An alternative is to implement a dedicated net, console and vsock driver for vringh (CAIF somehow works like this). This would have better performance.

Does it mean that the driver also functions as a network driver directly?

Quoted 8 lines

I meant, e.g in the case of networking, you can have a dedicated driver with two vringh in the endpoint side.

The benefit is the performance, no need for the (datapath) mediation.

But if we don't care about the performance, this proposal seems to be fine.

Thanks

I agree with you.  The design you suggested is better in terms of performance. However, the proposed design is not bad for the following the reasons I think.

The proposed design has one more operation in control plane because the data steps over the virtio-net driver, but the number of copies at the data plane remains the same. I think the operation added in control plane has small effects for performance.

Moreover, there are some advantages when the data step over the virtio-net driver. We can make use of the optimizations and some functions without modifications and implementations. e.g. ethtool and XDP(BPF) supports.

Any comments would be appreciated.

Quoted 115 lines
Quoted 115 lines
Quoted 115 lines
Quoted 102 lines
Quoted 102 lines

This driver has two roles. The first is as a PCIe endpoint virtio console function, which is implemented using the PCIe endpoint framework and PCIe EP virtio helpers. The second is as a virtual virtio console device connected to the virtio bus on PCIe endpoint Linux.

Communication between the two is achieved by copying the virtqueue data between PCIe root and endpoint, respectively.

This is a simple implementation and does not include features of virtio-console such as MULTIPORT, EMERG_WRITE, etc. As a result, each virtio console driver only displays /dev/hvc0.

As an example of usage, by setting getty to /dev/hvc0, it is possible to login to another host.

---
Changes from v2:
- Change to use copy functions between kiovs of pci-epf-virtio.

   drivers/pci/endpoint/functions/Kconfig        |  12 +
   drivers/pci/endpoint/functions/Makefile       |   1 +
   drivers/pci/endpoint/functions/pci-epf-vcon.c | 596 ++++++++++++++++++
   3 files changed, 609 insertions(+)

create mode 100644 drivers/pci/endpoint/functions/pci-epf-vcon.c

drivers/pci/endpoint/functions/Kconfig +12 -0
index fa1a6a569a8f..9ce2698b67e1 100644
@@ -44,3 +44,15 @@ config PCI_EPF_VIRTIO
          select VHOST_RING_IOMEM
          help
            Helpers to implement PCI virtio Endpoint function
+
+config PCI_EPF_VCON
+       tristate "PCI Endpoint virito-console driver"
+       depends on PCI_ENDPOINT
+       select VHOST_RING
+       select PCI_EPF_VIRTIO
+       help
+         PCIe Endpoint virtio-console function implementatino. This module
+         enables to show the virtio-console as pci device to PCIe host side, and
+         another virtual virtio-console device registers to endpoint system.
+         Those devices are connected virtually and can communicate each other.
+
drivers/pci/endpoint/functions/Makefile +1 -0
index a96f127ce900..b4056689ce33 100644
@@ -7,3 +7,4 @@ obj-$(CONFIG_PCI_EPF_TEST)              += pci-epf-test.o
   obj-$(CONFIG_PCI_EPF_NTB)              += pci-epf-ntb.o
   obj-$(CONFIG_PCI_EPF_VNTB)             += pci-epf-vntb.o
   obj-$(CONFIG_PCI_EPF_VIRTIO)           += pci-epf-virtio.o
+obj-$(CONFIG_PCI_EPF_VCON)             += pci-epf-vcon.o
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +41 -0
new file mode 100644
index 000000000000..31f4247cd10f
@@ -0,0 +1,596 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * PCI Endpoint function driver to impliment virtio-console device
+ * functionality.
+ */
+#include <linux/pci-epf.h>
+#include <linux/virtio_ids.h>
+#include <linux/virtio_pci.h>
+#include <linux/virtio_console.h>
+#include <linux/virtio_ring.h>
+
+#include "pci-epf-virtio.h"
+
+static int virtio_queue_size = 0x100;
+module_param(virtio_queue_size, int, 0444);
+MODULE_PARM_DESC(virtio_queue_size, "A length of virtqueue");
+
+struct epf_vcon {
+       /* To access virtqueues on remote host */
+       struct epf_virtio evio;
+       struct vringh_kiov *rdev_iovs;
+
+       /* To register a local virtio bus */
+       struct virtio_device vdev;
+
+       /* To access virtqueus of local host driver */
+       struct vringh *vdev_vrhs;
+       struct vringh_kiov *vdev_iovs;
+       struct virtqueue **vdev_vqs;
+
+       /* For transportation and notification */
+       struct workqueue_struct *task_wq;
+       struct work_struct raise_irq_work, rx_work, tx_work;
+
+       /* To retain virtio features. It is commonly used local and remote. */
+       u64 features;
+
+       /* To show a status whether this driver is ready and the remote is connected */
+       bool connected;
+};
+
Quoted 8 lines

drivers/pci/endpoint/functions/pci-epf-vcon.c line 47

Quoted 6 lines
drivers/pci/endpoint/functions/pci-epf-vcon.c new file +6 -0
+enum {
+       VCON_VIRTQUEUE_RX,
+       VCON_VIRTQUEUE_TX,
+       // Should be end of enum
+       VCON_VIRTQUEUE_NUM
+};

It would be better if we can split the console specific thing out, then it allows us to do ethernet and vsock in the future.

I'm planning to implement each virtio device in a separate file.
https://lwn.net/Articles/922124/

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

Thanks

Best regards,
Shunsuke

Best regards,

Shunsuke

#16 lore raw

Re: [RFC PATCH v2 3/3] PCI: endpoint: Add EP function driver to provide virtio-console functionality

In reply to Shunsuke Mie
kernel test robot noticed the following build errors:

Hi Shunsuke,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:

[auto build test ERROR on pci/next]
[also build test ERROR on pci/for-linus linus/master v6.4-rc1 next-20230509] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Shunsuke-Mie/PCI-endpoint-introduce-a-helper-to-implement-pci-ep-virtio-function/20230427-184534
base:   https://git.kernel.org/pub/scm/linux/kernel/git/pci/pci.git next
patch link:    https://lore.kernel.org/r/20230427104428.862643-4-mie%40igel.co.jp
patch subject: [RFC PATCH v2 3/3] PCI: endpoint: Add EP function driver to provide virtio-console functionality
config: loongarch-allmodconfig (https://download.01.org/0day-ci/archive/20230509/[email protected]/config)
compiler: loongarch64-linux-gcc (GCC) 12.1.0

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# https://github.com/intel-lab-lkp/linux/commit/988a24cb673bb7a63b7f92f04eff6393be2e70b3
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Shunsuke-Mie/PCI-endpoint-introduce-a-helper-to-implement-pci-ep-virtio-function/20230427-184534 git checkout 988a24cb673bb7a63b7f92f04eff6393be2e70b3
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=loongarch olddefconfig COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-12.1.0 make.cross W=1 O=build_dir ARCH=loongarch SHELL=/bin/bash drivers/pci/endpoint/functions/

If you fix the issue, kindly add following tag where applicable
| Reported-by: kernel test robot <[email protected]>
| Link: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

drivers/pci/endpoint/functions/pci-epf-virtio.c: In function 'epf_virtio_alloc_vringh':

>> drivers/pci/endpoint/functions/pci-epf-virtio.c:75:15: error: implicit declaration of function 'vringh_init_iomem'; did you mean 'vringh_init_iotlb'? [-Werror=implicit-function-declaration]
      75 |         err = vringh_init_iomem(&evrh->vrh, features, num, false, vring.desc,
         |               ^~~~~~~~~~~~~~~~~
         |               vringh_init_iotlb
   drivers/pci/endpoint/functions/pci-epf-virtio.c: In function 'epf_virtio_getdesc':
>> drivers/pci/endpoint/functions/pci-epf-virtio.c:382:16: error: implicit declaration of function 'vringh_getdesc_iomem'; did you mean 'vringh_getdesc_iotlb'? [-Werror=implicit-function-declaration]
     382 |         return vringh_getdesc_iomem(vrh, riov, wiov, head, GFP_KERNEL);
         |                ^~~~~~~~~~~~~~~~~~~~
         |                vringh_getdesc_iotlb
   drivers/pci/endpoint/functions/pci-epf-virtio.c: In function 'epf_virtio_abandon':
>> drivers/pci/endpoint/functions/pci-epf-virtio.c:389:9: error: implicit declaration of function 'vringh_abandon_iomem'; did you mean 'vringh_abandon_iotlb'? [-Werror=implicit-function-declaration]
     389 |         vringh_abandon_iomem(vrh, num);
         |         ^~~~~~~~~~~~~~~~~~~~
         |         vringh_abandon_iotlb
   drivers/pci/endpoint/functions/pci-epf-virtio.c: In function 'epf_virtio_iov_complete':
>> drivers/pci/endpoint/functions/pci-epf-virtio.c:397:9: error: implicit declaration of function 'vringh_complete_iomem'; did you mean 'vringh_complete_iotlb'? [-Werror=implicit-function-declaration]
     397 |         vringh_complete_iomem(vrh, head, total_len);
         |         ^~~~~~~~~~~~~~~~~~~~~
         |         vringh_complete_iotlb
   cc1: some warnings being treated as errors
--
   drivers/pci/endpoint/functions/pci-epf-vcon.c: In function 'epf_vcon_raise_irq_handler':
>> drivers/pci/endpoint/functions/pci-epf-vcon.c:162:27: error: 'PCI_EPC_IRQ_INTX' undeclared (first use in this function); did you mean 'PCI_EPC_IRQ_MSIX'?
     162 |                           PCI_EPC_IRQ_INTX, 0);
         |                           ^~~~~~~~~~~~~~~~
         |                           PCI_EPC_IRQ_MSIX

drivers/pci/endpoint/functions/pci-epf-vcon.c:162:27: note: each undeclared identifier is reported only once for each function it appears in

vim +162 drivers/pci/endpoint/functions/pci-epf-vcon.c

154
155 static void epf_vcon_raise_irq_handler(struct work_struct *work)
156 {
157 struct epf_vcon *vcon =
158 container_of(work, struct epf_vcon, raise_irq_work);
159 struct pci_epf *epf = vcon->evio.epf;
160
161 pci_epc_raise_irq(epf->epc, epf->func_no, epf->vfunc_no,

 > 162				  PCI_EPC_IRQ_INTX, 0);

163 }
164

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests
#17 lore raw
In reply to Shunsuke Mie
Quoted 5 lines · Shunsuke Mie

...
PCI: endpoint: introduce a helper to implement pci ep virtio function
virtio_pci: add a definition of queue flag in ISR
PCI: endpoint: Add EP function driver to provide virtio-console
functionality

Capitalize the first word consistently to match history ("Introduce", "Add").

Capitalize "PCI" in English text.

Capitalize "EP" since it's sort of an acronym (you did once, but do it both places :))

Bjorn