Skip to contentSkip to conversation

Enhance virtio rpmsg bus driver buffer allocation

PATCH In review Xiang Xiao 3 participants 17 messages Jan 31, 2019 – Jul 1, 2019

Subsystems (5)
lore
Share & tools
lore

Xiang Xiao

author
#0 lore raw

cover Enhance virtio rpmsg bus driver buffer allocation

Hi,
This series enhance the buffer allocation by:
1.Support the different buffer number in rx/tx direction 2.Get the individual rx/tx buffer size from config space

Here is the related OpenAMP change:
https://github.com/OpenAMP/open-amp/pull/155

Xiang Xiao (3):
rpmsg: virtio_rpmsg_bus: allow the different vring size for send/recv
rpmsg: virtio_rpmsg_bus: allocate rx/tx buffer separately
rpmsg: virtio_rpmsg_bus: get buffer size from config space

 drivers/rpmsg/virtio_rpmsg_bus.c  | 127 +++++++++++++++++++++++---------------
 include/uapi/linux/virtio_rpmsg.h |  24 +++++++
 2 files changed, 100 insertions(+), 51 deletions(-)
 create mode 100644 include/uapi/linux/virtio_rpmsg.h
Signature · 1 line
2.7.4
author
#1 lore raw

PATCH 1/3 rpmsg: virtio_rpmsg_bus: allow the different vring size for send/recv

it's useful if the communication throughput is different from each side

---
 drivers/rpmsg/virtio_rpmsg_bus.c | 47 ++++++++++++++++++++--------------------
 1 file changed, 24 insertions(+), 23 deletions(-)
drivers/rpmsg/virtio_rpmsg_bus.c +24 -23
index 664f957..fb0d2eb 100644
@@ -36,8 +36,9 @@
  * @svq:	tx virtqueue
  * @rbufs:	kernel address of rx buffers
  * @sbufs:	kernel address of tx buffers
- * @num_bufs:	total number of buffers for rx and tx
- * @buf_size:   size of one rx or tx buffer
+ * @num_rbufs:	total number of buffers for rx
+ * @num_sbufs:	total number of buffers for tx
+ * @buf_size:	size of one rx or tx buffer
  * @last_sbuf:	index of last tx buffer used
  * @bufs_dma:	dma base addr of the buffers
  * @tx_lock:	protects svq, sbufs and sleepers, to allow concurrent senders.
@@ -57,7 +58,8 @@ struct virtproc_info {
 	struct virtio_device *vdev;
 	struct virtqueue *rvq, *svq;
 	void *rbufs, *sbufs;
-	unsigned int num_bufs;
+	unsigned int num_rbufs;
+	unsigned int num_sbufs;
 	unsigned int buf_size;
 	int last_sbuf;
 	dma_addr_t bufs_dma;
@@ -136,7 +138,7 @@ struct virtio_rpmsg_channel {
 /*
  * We're allocating buffers of 512 bytes each for communications. The
  * number of buffers will be computed from the number of buffers supported
- * by the vring, upto a maximum of 512 buffers (256 in each direction).
+ * by the vring, up to a maximum of 256 in each direction.
  *
  * Each buffer will have 16 bytes for the msg header and 496 bytes for
  * the payload.
@@ -151,7 +153,7 @@ struct virtio_rpmsg_channel {
  * can change this without changing anything in the firmware of the remote
  * processor.
  */
-#define MAX_RPMSG_NUM_BUFS	(512)
+#define MAX_RPMSG_NUM_BUFS	(256)
 #define MAX_RPMSG_BUF_SIZE	(512)
 
 /*
@@ -446,11 +448,8 @@ static void *get_a_tx_buf(struct virtproc_info *vrp)
 	/* support multiple concurrent senders */
 	mutex_lock(&vrp->tx_lock);
 
-	/*
-	 * either pick the next unused tx buffer
-	 * (half of our buffers are used for sending messages)
-	 */
-	if (vrp->last_sbuf < vrp->num_bufs / 2)
+	/* either pick the next unused tx buffer */
+	if (vrp->last_sbuf < vrp->num_sbufs)
 		ret = vrp->sbufs + vrp->buf_size * vrp->last_sbuf++;
 	/* or recycle a used one */
 	else
@@ -897,19 +896,20 @@ static int rpmsg_probe(struct virtio_device *vdev)
 	vrp->rvq = vqs[0];
 	vrp->svq = vqs[1];
 
-	/* we expect symmetric tx/rx vrings */
-	WARN_ON(virtqueue_get_vring_size(vrp->rvq) !=
-		virtqueue_get_vring_size(vrp->svq));
-
 	/* we need less buffers if vrings are small */
-	if (virtqueue_get_vring_size(vrp->rvq) < MAX_RPMSG_NUM_BUFS / 2)
-		vrp->num_bufs = virtqueue_get_vring_size(vrp->rvq) * 2;
+	if (virtqueue_get_vring_size(vrp->rvq) < MAX_RPMSG_NUM_BUFS)
+		vrp->num_rbufs = virtqueue_get_vring_size(vrp->rvq);
+	else
+		vrp->num_rbufs = MAX_RPMSG_NUM_BUFS;
+
+	if (virtqueue_get_vring_size(vrp->svq) < MAX_RPMSG_NUM_BUFS)
+		vrp->num_sbufs = virtqueue_get_vring_size(vrp->svq);
 	else
-		vrp->num_bufs = MAX_RPMSG_NUM_BUFS;
+		vrp->num_sbufs = MAX_RPMSG_NUM_BUFS;
 
 	vrp->buf_size = MAX_RPMSG_BUF_SIZE;
 
-	total_buf_space = vrp->num_bufs * vrp->buf_size;
+	total_buf_space = (vrp->num_rbufs + vrp->num_sbufs) * vrp->buf_size;
 
 	/* allocate coherent memory for the buffers */
 	bufs_va = dma_alloc_coherent(vdev->dev.parent->parent,
@@ -923,14 +923,14 @@ static int rpmsg_probe(struct virtio_device *vdev)
 	dev_dbg(&vdev->dev, "buffers: va %p, dma %pad\n",
 		bufs_va, &vrp->bufs_dma);
 
-	/* half of the buffers is dedicated for RX */
+	/* first part of the buffers is dedicated for RX */
 	vrp->rbufs = bufs_va;
 
-	/* and half is dedicated for TX */
-	vrp->sbufs = bufs_va + total_buf_space / 2;
+	/* and second part is dedicated for TX */
+	vrp->sbufs = bufs_va + vrp->num_rbufs * vrp->buf_size;
 
 	/* set up the receive buffers */
-	for (i = 0; i < vrp->num_bufs / 2; i++) {
+	for (i = 0; i < vrp->num_rbufs; i++) {
 		struct scatterlist sg;
 		void *cpu_addr = vrp->rbufs + i * vrp->buf_size;
 
@@ -999,7 +999,8 @@ static int rpmsg_remove_device(struct device *dev, void *data)
 static void rpmsg_remove(struct virtio_device *vdev)
 {
 	struct virtproc_info *vrp = vdev->priv;
-	size_t total_buf_space = vrp->num_bufs * vrp->buf_size;
+	unsigned int num_bufs = vrp->num_rbufs + vrp->num_sbufs;
+	size_t total_buf_space = num_bufs * vrp->buf_size;
 	int ret;
 
 	vdev->config->reset(vdev);
Signature · 1 line
2.7.4
Acked-by on v1 1/3
#2 lore raw

Re: [PATCH 1/3] rpmsg: virtio_rpmsg_bus: allow the different vring size for send/recv

Hello Xiang,

I tested your patch on stm32 platform, no issue.
No remark on you code.

Thanks,
Arnaud

Quoted 124 lines · Xiang Xiao

it's useful if the communication throughput is different from each side

---
 drivers/rpmsg/virtio_rpmsg_bus.c | 47 ++++++++++++++++++++--------------------
 1 file changed, 24 insertions(+), 23 deletions(-)
drivers/rpmsg/virtio_rpmsg_bus.c +24 -23
index 664f957..fb0d2eb 100644
@@ -36,8 +36,9 @@
  * @svq:	tx virtqueue
  * @rbufs:	kernel address of rx buffers
  * @sbufs:	kernel address of tx buffers
- * @num_bufs:	total number of buffers for rx and tx
- * @buf_size:   size of one rx or tx buffer
+ * @num_rbufs:	total number of buffers for rx
+ * @num_sbufs:	total number of buffers for tx
+ * @buf_size:	size of one rx or tx buffer
  * @last_sbuf:	index of last tx buffer used
  * @bufs_dma:	dma base addr of the buffers
  * @tx_lock:	protects svq, sbufs and sleepers, to allow concurrent senders.
@@ -57,7 +58,8 @@ struct virtproc_info {
 	struct virtio_device *vdev;
 	struct virtqueue *rvq, *svq;
 	void *rbufs, *sbufs;
-	unsigned int num_bufs;
+	unsigned int num_rbufs;
+	unsigned int num_sbufs;
 	unsigned int buf_size;
 	int last_sbuf;
 	dma_addr_t bufs_dma;
@@ -136,7 +138,7 @@ struct virtio_rpmsg_channel {
 /*
  * We're allocating buffers of 512 bytes each for communications. The
  * number of buffers will be computed from the number of buffers supported
- * by the vring, upto a maximum of 512 buffers (256 in each direction).
+ * by the vring, up to a maximum of 256 in each direction.
  *
  * Each buffer will have 16 bytes for the msg header and 496 bytes for
  * the payload.
@@ -151,7 +153,7 @@ struct virtio_rpmsg_channel {
  * can change this without changing anything in the firmware of the remote
  * processor.
  */
-#define MAX_RPMSG_NUM_BUFS	(512)
+#define MAX_RPMSG_NUM_BUFS	(256)
 #define MAX_RPMSG_BUF_SIZE	(512)
 
 /*
@@ -446,11 +448,8 @@ static void *get_a_tx_buf(struct virtproc_info *vrp)
 	/* support multiple concurrent senders */
 	mutex_lock(&vrp->tx_lock);
 
-	/*
-	 * either pick the next unused tx buffer
-	 * (half of our buffers are used for sending messages)
-	 */
-	if (vrp->last_sbuf < vrp->num_bufs / 2)
+	/* either pick the next unused tx buffer */
+	if (vrp->last_sbuf < vrp->num_sbufs)
 		ret = vrp->sbufs + vrp->buf_size * vrp->last_sbuf++;
 	/* or recycle a used one */
 	else
@@ -897,19 +896,20 @@ static int rpmsg_probe(struct virtio_device *vdev)
 	vrp->rvq = vqs[0];
 	vrp->svq = vqs[1];
 
-	/* we expect symmetric tx/rx vrings */
-	WARN_ON(virtqueue_get_vring_size(vrp->rvq) !=
-		virtqueue_get_vring_size(vrp->svq));
-
 	/* we need less buffers if vrings are small */
-	if (virtqueue_get_vring_size(vrp->rvq) < MAX_RPMSG_NUM_BUFS / 2)
-		vrp->num_bufs = virtqueue_get_vring_size(vrp->rvq) * 2;
+	if (virtqueue_get_vring_size(vrp->rvq) < MAX_RPMSG_NUM_BUFS)
+		vrp->num_rbufs = virtqueue_get_vring_size(vrp->rvq);
+	else
+		vrp->num_rbufs = MAX_RPMSG_NUM_BUFS;
+
+	if (virtqueue_get_vring_size(vrp->svq) < MAX_RPMSG_NUM_BUFS)
+		vrp->num_sbufs = virtqueue_get_vring_size(vrp->svq);
 	else
-		vrp->num_bufs = MAX_RPMSG_NUM_BUFS;
+		vrp->num_sbufs = MAX_RPMSG_NUM_BUFS;
 
 	vrp->buf_size = MAX_RPMSG_BUF_SIZE;
 
-	total_buf_space = vrp->num_bufs * vrp->buf_size;
+	total_buf_space = (vrp->num_rbufs + vrp->num_sbufs) * vrp->buf_size;
 
 	/* allocate coherent memory for the buffers */
 	bufs_va = dma_alloc_coherent(vdev->dev.parent->parent,
@@ -923,14 +923,14 @@ static int rpmsg_probe(struct virtio_device *vdev)
 	dev_dbg(&vdev->dev, "buffers: va %p, dma %pad\n",
 		bufs_va, &vrp->bufs_dma);
 
-	/* half of the buffers is dedicated for RX */
+	/* first part of the buffers is dedicated for RX */
 	vrp->rbufs = bufs_va;
 
-	/* and half is dedicated for TX */
-	vrp->sbufs = bufs_va + total_buf_space / 2;
+	/* and second part is dedicated for TX */
+	vrp->sbufs = bufs_va + vrp->num_rbufs * vrp->buf_size;
 
 	/* set up the receive buffers */
-	for (i = 0; i < vrp->num_bufs / 2; i++) {
+	for (i = 0; i < vrp->num_rbufs; i++) {
 		struct scatterlist sg;
 		void *cpu_addr = vrp->rbufs + i * vrp->buf_size;
 
@@ -999,7 +999,8 @@ static int rpmsg_remove_device(struct device *dev, void *data)
 static void rpmsg_remove(struct virtio_device *vdev)
 {
 	struct virtproc_info *vrp = vdev->priv;
-	size_t total_buf_space = vrp->num_bufs * vrp->buf_size;
+	unsigned int num_bufs = vrp->num_rbufs + vrp->num_sbufs;
+	size_t total_buf_space = num_bufs * vrp->buf_size;
 	int ret;
 
 	vdev->config->reset(vdev);
author
#3 lore raw

PATCH 2/3 rpmsg: virtio_rpmsg_bus: allocate rx/tx buffer separately

In reply to Xiang Xiao

many dma allocator align the returned address with buffer size, so two small allocation could reduce the alignment requirement and save the the memory space wasted by the potential alignment.

---
 drivers/rpmsg/virtio_rpmsg_bus.c | 58 +++++++++++++++++++++++-----------------
 1 file changed, 34 insertions(+), 24 deletions(-)
drivers/rpmsg/virtio_rpmsg_bus.c +34 -24
index fb0d2eb..59c4554 100644
@@ -40,7 +40,8 @@
  * @num_sbufs:	total number of buffers for tx
  * @buf_size:	size of one rx or tx buffer
  * @last_sbuf:	index of last tx buffer used
- * @bufs_dma:	dma base addr of the buffers
+ * @rbufs_dma:	dma base addr of rx buffers
+ * @sbufs_dma:	dma base addr of tx buffers
  * @tx_lock:	protects svq, sbufs and sleepers, to allow concurrent senders.
  *		sending a message might require waking up a dozing remote
  *		processor, which involves sleeping, hence the mutex.
@@ -62,7 +63,8 @@ struct virtproc_info {
 	unsigned int num_sbufs;
 	unsigned int buf_size;
 	int last_sbuf;
-	dma_addr_t bufs_dma;
+	dma_addr_t rbufs_dma;
+	dma_addr_t sbufs_dma;
 	struct mutex tx_lock;
 	struct idr endpoints;
 	struct mutex endpoints_lock;
@@ -872,9 +874,7 @@ static int rpmsg_probe(struct virtio_device *vdev)
 	static const char * const names[] = { "input", "output" };
 	struct virtqueue *vqs[2];
 	struct virtproc_info *vrp;
-	void *bufs_va;
 	int err = 0, i;
-	size_t total_buf_space;
 	bool notify;
 
 	vrp = kzalloc(sizeof(*vrp), GFP_KERNEL);
@@ -909,25 +909,28 @@ static int rpmsg_probe(struct virtio_device *vdev)
 
 	vrp->buf_size = MAX_RPMSG_BUF_SIZE;
 
-	total_buf_space = (vrp->num_rbufs + vrp->num_sbufs) * vrp->buf_size;
-
 	/* allocate coherent memory for the buffers */
-	bufs_va = dma_alloc_coherent(vdev->dev.parent->parent,
-				     total_buf_space, &vrp->bufs_dma,
-				     GFP_KERNEL);
-	if (!bufs_va) {
+	vrp->rbufs = dma_alloc_coherent(vdev->dev.parent->parent,
+					vrp->num_rbufs * vrp->buf_size,
+					&vrp->rbufs_dma, GFP_KERNEL);
+	if (!vrp->rbufs) {
 		err = -ENOMEM;
 		goto vqs_del;
 	}
 
-	dev_dbg(&vdev->dev, "buffers: va %p, dma %pad\n",
-		bufs_va, &vrp->bufs_dma);
+	dev_dbg(&vdev->dev, "rx buffers: va %p, dma 0x%pad\n",
+		vrp->rbufs, &vrp->rbufs_dma);
 
-	/* first part of the buffers is dedicated for RX */
-	vrp->rbufs = bufs_va;
+	vrp->sbufs = dma_alloc_coherent(vdev->dev.parent->parent,
+					vrp->num_sbufs * vrp->buf_size,
+					&vrp->sbufs_dma, GFP_KERNEL);
+	if (!vrp->sbufs) {
+		err = -ENOMEM;
+		goto free_rbufs;
+	}
 
-	/* and second part is dedicated for TX */
-	vrp->sbufs = bufs_va + vrp->num_rbufs * vrp->buf_size;
+	dev_dbg(&vdev->dev, "tx buffers: va %p, dma 0x%pad\n",
+		vrp->sbufs, &vrp->sbufs_dma);
 
 	/* set up the receive buffers */
 	for (i = 0; i < vrp->num_rbufs; i++) {
@@ -954,7 +957,7 @@ static int rpmsg_probe(struct virtio_device *vdev)
 		if (!vrp->ns_ept) {
 			dev_err(&vdev->dev, "failed to create the ns ept\n");
 			err = -ENOMEM;
-			goto free_coherent;
+			goto free_sbufs;
 		}
 	}
 
@@ -979,9 +982,14 @@ static int rpmsg_probe(struct virtio_device *vdev)
 
 	return 0;
 
-free_coherent:
-	dma_free_coherent(vdev->dev.parent->parent, total_buf_space,
-			  bufs_va, vrp->bufs_dma);
+free_sbufs:
+	dma_free_coherent(vdev->dev.parent->parent,
+			  vrp->num_sbufs * vrp->buf_size,
+			  vrp->sbufs, vrp->sbufs_dma);
+free_rbufs:
+	dma_free_coherent(vdev->dev.parent->parent,
+			  vrp->num_rbufs * vrp->buf_size,
+			  vrp->rbufs, vrp->rbufs_dma);
 vqs_del:
 	vdev->config->del_vqs(vrp->vdev);
 free_vrp:
@@ -999,8 +1007,6 @@ static int rpmsg_remove_device(struct device *dev, void *data)
 static void rpmsg_remove(struct virtio_device *vdev)
 {
 	struct virtproc_info *vrp = vdev->priv;
-	unsigned int num_bufs = vrp->num_rbufs + vrp->num_sbufs;
-	size_t total_buf_space = num_bufs * vrp->buf_size;
 	int ret;
 
 	vdev->config->reset(vdev);
@@ -1016,8 +1022,12 @@ static void rpmsg_remove(struct virtio_device *vdev)
 
 	vdev->config->del_vqs(vrp->vdev);
 
-	dma_free_coherent(vdev->dev.parent->parent, total_buf_space,
-			  vrp->rbufs, vrp->bufs_dma);
+	dma_free_coherent(vdev->dev.parent->parent,
+			  vrp->num_sbufs * vrp->buf_size,
+			  vrp->sbufs, vrp->sbufs_dma);
+	dma_free_coherent(vdev->dev.parent->parent,
+			  vrp->num_rbufs * vrp->buf_size,
+			  vrp->rbufs, vrp->rbufs_dma);
 
 	kfree(vrp);
 }
Signature · 1 line
2.7.4
#4 lore raw

Re: [PATCH 2/3] rpmsg: virtio_rpmsg_bus: allocate rx/tx buffer separately

Hello Xiang,

This patch has the opposite effect on my platform as DMA allocation is aligned on 4k page.
For instance i declared:
- in RX 6 buffers (of 512 bytes)
- in TX 4 buffers ( of 512 bytes)

The result is (kernel trace)

[   41.915896] virtio_rpmsg_bus virtio0: rx buffers: va ebb5f5ca, dma
0x0x10042000
[   41.915922] virtio_rpmsg_bus virtio0: tx buffers: va a7865153, dma

0x0x10043000

The TX buffer memory is allocated on next 4k page...

Anyway separate the RX and TX allocation makes sense. This could also allow to allocate buffers in 2 different memories.
For time being, issue is that only one memory area can be attached to the virtio device for DMA allocation... and PA/DA translations are missing. This means that we probably need (in a first step) a new remoteproc API for memory allocation.
These memories should be declared and mmaped in rproc platform drivers (memory region) or in resource table (carveout).
This is partially done in the API for the platform driver (rproc_mem_entry_init) but not available for rproc clients.

Regards
Arnaud

Quoted 136 lines · Xiang Xiao

many dma allocator align the returned address with buffer size, so two small allocation could reduce the alignment requirement and save the the memory space wasted by the potential alignment.

---
 drivers/rpmsg/virtio_rpmsg_bus.c | 58 +++++++++++++++++++++++-----------------
 1 file changed, 34 insertions(+), 24 deletions(-)
drivers/rpmsg/virtio_rpmsg_bus.c +34 -24
index fb0d2eb..59c4554 100644
@@ -40,7 +40,8 @@
  * @num_sbufs:	total number of buffers for tx
  * @buf_size:	size of one rx or tx buffer
  * @last_sbuf:	index of last tx buffer used
- * @bufs_dma:	dma base addr of the buffers
+ * @rbufs_dma:	dma base addr of rx buffers
+ * @sbufs_dma:	dma base addr of tx buffers
  * @tx_lock:	protects svq, sbufs and sleepers, to allow concurrent senders.
  *		sending a message might require waking up a dozing remote
  *		processor, which involves sleeping, hence the mutex.
@@ -62,7 +63,8 @@ struct virtproc_info {
 	unsigned int num_sbufs;
 	unsigned int buf_size;
 	int last_sbuf;
-	dma_addr_t bufs_dma;
+	dma_addr_t rbufs_dma;
+	dma_addr_t sbufs_dma;
 	struct mutex tx_lock;
 	struct idr endpoints;
 	struct mutex endpoints_lock;
@@ -872,9 +874,7 @@ static int rpmsg_probe(struct virtio_device *vdev)
 	static const char * const names[] = { "input", "output" };
 	struct virtqueue *vqs[2];
 	struct virtproc_info *vrp;
-	void *bufs_va;
 	int err = 0, i;
-	size_t total_buf_space;
 	bool notify;
 
 	vrp = kzalloc(sizeof(*vrp), GFP_KERNEL);
@@ -909,25 +909,28 @@ static int rpmsg_probe(struct virtio_device *vdev)
 
 	vrp->buf_size = MAX_RPMSG_BUF_SIZE;
 
-	total_buf_space = (vrp->num_rbufs + vrp->num_sbufs) * vrp->buf_size;
-
 	/* allocate coherent memory for the buffers */
-	bufs_va = dma_alloc_coherent(vdev->dev.parent->parent,
-				     total_buf_space, &vrp->bufs_dma,
-				     GFP_KERNEL);
-	if (!bufs_va) {
+	vrp->rbufs = dma_alloc_coherent(vdev->dev.parent->parent,
+					vrp->num_rbufs * vrp->buf_size,
+					&vrp->rbufs_dma, GFP_KERNEL);
+	if (!vrp->rbufs) {
 		err = -ENOMEM;
 		goto vqs_del;
 	}
 
-	dev_dbg(&vdev->dev, "buffers: va %p, dma %pad\n",
-		bufs_va, &vrp->bufs_dma);
+	dev_dbg(&vdev->dev, "rx buffers: va %p, dma 0x%pad\n",
+		vrp->rbufs, &vrp->rbufs_dma);
 
-	/* first part of the buffers is dedicated for RX */
-	vrp->rbufs = bufs_va;
+	vrp->sbufs = dma_alloc_coherent(vdev->dev.parent->parent,
+					vrp->num_sbufs * vrp->buf_size,
+					&vrp->sbufs_dma, GFP_KERNEL);
+	if (!vrp->sbufs) {
+		err = -ENOMEM;
+		goto free_rbufs;
+	}
 
-	/* and second part is dedicated for TX */
-	vrp->sbufs = bufs_va + vrp->num_rbufs * vrp->buf_size;
+	dev_dbg(&vdev->dev, "tx buffers: va %p, dma 0x%pad\n",
+		vrp->sbufs, &vrp->sbufs_dma);
 
 	/* set up the receive buffers */
 	for (i = 0; i < vrp->num_rbufs; i++) {
@@ -954,7 +957,7 @@ static int rpmsg_probe(struct virtio_device *vdev)
 		if (!vrp->ns_ept) {
 			dev_err(&vdev->dev, "failed to create the ns ept\n");
 			err = -ENOMEM;
-			goto free_coherent;
+			goto free_sbufs;
 		}
 	}
 
@@ -979,9 +982,14 @@ static int rpmsg_probe(struct virtio_device *vdev)
 
 	return 0;
 
-free_coherent:
-	dma_free_coherent(vdev->dev.parent->parent, total_buf_space,
-			  bufs_va, vrp->bufs_dma);
+free_sbufs:
+	dma_free_coherent(vdev->dev.parent->parent,
+			  vrp->num_sbufs * vrp->buf_size,
+			  vrp->sbufs, vrp->sbufs_dma);
+free_rbufs:
+	dma_free_coherent(vdev->dev.parent->parent,
+			  vrp->num_rbufs * vrp->buf_size,
+			  vrp->rbufs, vrp->rbufs_dma);
 vqs_del:
 	vdev->config->del_vqs(vrp->vdev);
 free_vrp:
@@ -999,8 +1007,6 @@ static int rpmsg_remove_device(struct device *dev, void *data)
 static void rpmsg_remove(struct virtio_device *vdev)
 {
 	struct virtproc_info *vrp = vdev->priv;
-	unsigned int num_bufs = vrp->num_rbufs + vrp->num_sbufs;
-	size_t total_buf_space = num_bufs * vrp->buf_size;
 	int ret;
 
 	vdev->config->reset(vdev);
@@ -1016,8 +1022,12 @@ static void rpmsg_remove(struct virtio_device *vdev)
 
 	vdev->config->del_vqs(vrp->vdev);
 
-	dma_free_coherent(vdev->dev.parent->parent, total_buf_space,
-			  vrp->rbufs, vrp->bufs_dma);
+	dma_free_coherent(vdev->dev.parent->parent,
+			  vrp->num_sbufs * vrp->buf_size,
+			  vrp->sbufs, vrp->sbufs_dma);
+	dma_free_coherent(vdev->dev.parent->parent,
+			  vrp->num_rbufs * vrp->buf_size,
+			  vrp->rbufs, vrp->rbufs_dma);
 
 	kfree(vrp);
 }
author
#5 lore raw

Re: [PATCH 2/3] rpmsg: virtio_rpmsg_bus: allocate rx/tx buffer separately

Quoted 9 lines · Arnaud Pouliquen

Hello Xiang,

This patch has the opposite effect on my platform as DMA allocation is aligned on 4k page.
For instance i declared:
- in RX 6 buffers (of 512 bytes)
- in TX 4 buffers ( of 512 bytes)

Yes, dma_init_coherent_memory always allocate memory by 4KB unit, but this limitation is too waste memory for remoteproc/rpmsg. The attached patch fix this problem by adding a new device tree option to customize the unit size.

Quoted 157 lines

The result is (kernel trace)

[   41.915896] virtio_rpmsg_bus virtio0: rx buffers: va ebb5f5ca, dma
0x0x10042000
[   41.915922] virtio_rpmsg_bus virtio0: tx buffers: va a7865153, dma

0x0x10043000

The TX buffer memory is allocated on next 4k page...

Anyway separate the RX and TX allocation makes sense. This could also allow to allocate buffers in 2 different memories.
For time being, issue is that only one memory area can be attached to the virtio device for DMA allocation... and PA/DA translations are missing. This means that we probably need (in a first step) a new remoteproc API for memory allocation.
These memories should be declared and mmaped in rproc platform drivers (memory region) or in resource table (carveout).
This is partially done in the API for the platform driver (rproc_mem_entry_init) but not available for rproc clients.

Regards
Arnaud

Quoted 133 lines · Xiang Xiao

many dma allocator align the returned address with buffer size, so two small allocation could reduce the alignment requirement and save the the memory space wasted by the potential alignment.

---
 drivers/rpmsg/virtio_rpmsg_bus.c | 58 +++++++++++++++++++++++-----------------
 1 file changed, 34 insertions(+), 24 deletions(-)
drivers/rpmsg/virtio_rpmsg_bus.c +34 -24
index fb0d2eb..59c4554 100644
@@ -40,7 +40,8 @@
  * @num_sbufs:       total number of buffers for tx
  * @buf_size:        size of one rx or tx buffer
  * @last_sbuf:       index of last tx buffer used
- * @bufs_dma:        dma base addr of the buffers
+ * @rbufs_dma:       dma base addr of rx buffers
+ * @sbufs_dma:       dma base addr of tx buffers
  * @tx_lock: protects svq, sbufs and sleepers, to allow concurrent senders.
  *           sending a message might require waking up a dozing remote
  *           processor, which involves sleeping, hence the mutex.
@@ -62,7 +63,8 @@ struct virtproc_info {
      unsigned int num_sbufs;
      unsigned int buf_size;
      int last_sbuf;
-     dma_addr_t bufs_dma;
+     dma_addr_t rbufs_dma;
+     dma_addr_t sbufs_dma;
      struct mutex tx_lock;
      struct idr endpoints;
      struct mutex endpoints_lock;
@@ -872,9 +874,7 @@ static int rpmsg_probe(struct virtio_device *vdev)
      static const char * const names[] = { "input", "output" };
      struct virtqueue *vqs[2];
      struct virtproc_info *vrp;
-     void *bufs_va;
      int err = 0, i;
-     size_t total_buf_space;
      bool notify;
 
      vrp = kzalloc(sizeof(*vrp), GFP_KERNEL);
@@ -909,25 +909,28 @@ static int rpmsg_probe(struct virtio_device *vdev)
 
      vrp->buf_size = MAX_RPMSG_BUF_SIZE;
 
-     total_buf_space = (vrp->num_rbufs + vrp->num_sbufs) * vrp->buf_size;
-
      /* allocate coherent memory for the buffers */
-     bufs_va = dma_alloc_coherent(vdev->dev.parent->parent,
-                                  total_buf_space, &vrp->bufs_dma,
-                                  GFP_KERNEL);
-     if (!bufs_va) {
+     vrp->rbufs = dma_alloc_coherent(vdev->dev.parent->parent,
+                                     vrp->num_rbufs * vrp->buf_size,
+                                     &vrp->rbufs_dma, GFP_KERNEL);
+     if (!vrp->rbufs) {
              err = -ENOMEM;
              goto vqs_del;
      }
 
-     dev_dbg(&vdev->dev, "buffers: va %p, dma %pad\n",
-             bufs_va, &vrp->bufs_dma);
+     dev_dbg(&vdev->dev, "rx buffers: va %p, dma 0x%pad\n",
+             vrp->rbufs, &vrp->rbufs_dma);
 
-     /* first part of the buffers is dedicated for RX */
-     vrp->rbufs = bufs_va;
+     vrp->sbufs = dma_alloc_coherent(vdev->dev.parent->parent,
+                                     vrp->num_sbufs * vrp->buf_size,
+                                     &vrp->sbufs_dma, GFP_KERNEL);
+     if (!vrp->sbufs) {
+             err = -ENOMEM;
+             goto free_rbufs;
+     }
 
-     /* and second part is dedicated for TX */
-     vrp->sbufs = bufs_va + vrp->num_rbufs * vrp->buf_size;
+     dev_dbg(&vdev->dev, "tx buffers: va %p, dma 0x%pad\n",
+             vrp->sbufs, &vrp->sbufs_dma);
 
      /* set up the receive buffers */
      for (i = 0; i < vrp->num_rbufs; i++) {
@@ -954,7 +957,7 @@ static int rpmsg_probe(struct virtio_device *vdev)
              if (!vrp->ns_ept) {
                      dev_err(&vdev->dev, "failed to create the ns ept\n");
                      err = -ENOMEM;
-                     goto free_coherent;
+                     goto free_sbufs;
              }
      }
 
@@ -979,9 +982,14 @@ static int rpmsg_probe(struct virtio_device *vdev)
 
      return 0;
 
-free_coherent:
-     dma_free_coherent(vdev->dev.parent->parent, total_buf_space,
-                       bufs_va, vrp->bufs_dma);
+free_sbufs:
+     dma_free_coherent(vdev->dev.parent->parent,
+                       vrp->num_sbufs * vrp->buf_size,
+                       vrp->sbufs, vrp->sbufs_dma);
+free_rbufs:
+     dma_free_coherent(vdev->dev.parent->parent,
+                       vrp->num_rbufs * vrp->buf_size,
+                       vrp->rbufs, vrp->rbufs_dma);
 vqs_del:
      vdev->config->del_vqs(vrp->vdev);
 free_vrp:
@@ -999,8 +1007,6 @@ static int rpmsg_remove_device(struct device *dev, void *data)
 static void rpmsg_remove(struct virtio_device *vdev)
 {
      struct virtproc_info *vrp = vdev->priv;
-     unsigned int num_bufs = vrp->num_rbufs + vrp->num_sbufs;
-     size_t total_buf_space = num_bufs * vrp->buf_size;
      int ret;
 
      vdev->config->reset(vdev);
@@ -1016,8 +1022,12 @@ static void rpmsg_remove(struct virtio_device *vdev)
 
      vdev->config->del_vqs(vrp->vdev);
 
-     dma_free_coherent(vdev->dev.parent->parent, total_buf_space,
-                       vrp->rbufs, vrp->bufs_dma);
+     dma_free_coherent(vdev->dev.parent->parent,
+                       vrp->num_sbufs * vrp->buf_size,
+                       vrp->sbufs, vrp->sbufs_dma);
+     dma_free_coherent(vdev->dev.parent->parent,
+                       vrp->num_rbufs * vrp->buf_size,
+                       vrp->rbufs, vrp->rbufs_dma);
 
Quoted 3 lines

drivers/rpmsg/virtio_rpmsg_bus.c static void rpmsg_remove(struct virtio_device *vdev) · line 1033

Quoted 3 lines
drivers/rpmsg/virtio_rpmsg_bus.c +0 -0
      kfree(vrp);
 }

From 62672e1df79c9ffccb062f5aa0dcb18e938ced0c Mon Sep 17 00:00:00 2001
From: Xiang Xiao <[email protected]>
Date: Thu, 22 Jun 2017 14:30:50 +0800
Subject: [PATCH] dma-coherent: support the alignment smaller than PAGE_SIZE

and read the alignment(default PAGE_SIZE) from device tree

Change-Id I512128859b0aac2e25be09ca8e0968190057e160
---
 .../bindings/reserved-memory/reserved-memory.txt   |  2 +
 drivers/base/dma-coherent.c                        | 74 ++++++++++++++--------
 include/linux/dma-mapping.h                        |  6 +-
 3 files changed, 51 insertions(+), 31 deletions(-)
Documentation/devicetree/bindings/reserved-memory/reserved-memory.txt +2 -0
index 3da0ebdba8d9..383ada09213b 100644
@@ -63,6 +63,8 @@ reusable (optional) - empty property
       able to reclaim it back. Typically that means that the operating
       system can use that region to store volatile or cached data that
       can be otherwise regenerated or migrated elsewhere.
+align-shift (optional) - the allocation alignment (1 << align-shift)
+    - The default value is PAGE_SHIFT.
 
 Linux implementation note:
 - If a "linux,cma-default" property is present, then Linux will use the
drivers/base/dma-coherent.c +46 -28

179 lines not on this page · raw message

include/linux/dma-mapping.h +3 -3
index 08528afdf58b..8d9fd7b01dbb 100644
@@ -152,13 +152,13 @@ static inline int is_device_dma_capable(struct device *dev)
  */
 int dma_alloc_from_coherent(struct device *dev, ssize_t size,
 				       dma_addr_t *dma_handle, void **ret);
-int dma_release_from_coherent(struct device *dev, int order, void *vaddr);
+int dma_release_from_coherent(struct device *dev, int size, void *vaddr);
 
 int dma_mmap_from_coherent(struct device *dev, struct vm_area_struct *vma,
 			    void *cpu_addr, size_t size, int *ret);
 #else
 #define dma_alloc_from_coherent(dev, size, handle, ret) (0)
-#define dma_release_from_coherent(dev, order, vaddr) (0)
+#define dma_release_from_coherent(dev, size, vaddr) (0)
 #define dma_mmap_from_coherent(dev, vma, vaddr, order, ret) (0)
 #endif /* CONFIG_HAVE_GENERIC_DMA_COHERENT */
 
@@ -478,7 +478,7 @@ static inline void dma_free_attrs(struct device *dev, size_t size,
 	BUG_ON(!ops);
 	WARN_ON(irqs_disabled());
 
-	if (dma_release_from_coherent(dev, get_order(size), cpu_addr))
+	if (dma_release_from_coherent(dev, size, cpu_addr))
 		return;
 
 	if (!ops->free || !cpu_addr)
Signature · 1 line
2.16.2
author
#6 lore raw

PATCH 3/3 rpmsg: virtio_rpmsg_bus: get buffer size from config space

In reply to Xiang Xiao

512 bytes isn't always suitable for all case, let firmware maker decide the best value from resource table. enable by VIRTIO_RPMSG_F_BUFSZ feature bit.

---
 drivers/rpmsg/virtio_rpmsg_bus.c  | 50 +++++++++++++++++++++++++--------------
 include/uapi/linux/virtio_rpmsg.h | 24 +++++++++++++++++++
 2 files changed, 56 insertions(+), 18 deletions(-)
 create mode 100644 include/uapi/linux/virtio_rpmsg.h
drivers/rpmsg/virtio_rpmsg_bus.c +32 -18
index 59c4554..049dd97 100644
@@ -16,6 +16,7 @@
 #include <linux/virtio.h>
 #include <linux/virtio_ids.h>
 #include <linux/virtio_config.h>
+#include <linux/virtio_rpmsg.h>
 #include <linux/scatterlist.h>
 #include <linux/dma-mapping.h>
 #include <linux/slab.h>
@@ -38,7 +39,8 @@
  * @sbufs:	kernel address of tx buffers
  * @num_rbufs:	total number of buffers for rx
  * @num_sbufs:	total number of buffers for tx
- * @buf_size:	size of one rx or tx buffer
+ * @rbuf_size:	size of one rx buffer
+ * @sbuf_size:	size of one tx buffer
  * @last_sbuf:	index of last tx buffer used
  * @rbufs_dma:	dma base addr of rx buffers
  * @sbufs_dma:	dma base addr of tx buffers
@@ -61,7 +63,8 @@ struct virtproc_info {
 	void *rbufs, *sbufs;
 	unsigned int num_rbufs;
 	unsigned int num_sbufs;
-	unsigned int buf_size;
+	unsigned int rbuf_size;
+	unsigned int sbuf_size;
 	int last_sbuf;
 	dma_addr_t rbufs_dma;
 	dma_addr_t sbufs_dma;
@@ -73,9 +76,6 @@ struct virtproc_info {
 	struct rpmsg_endpoint *ns_ept;
 };
 
-/* The feature bitmap for virtio rpmsg */
-#define VIRTIO_RPMSG_F_NS	0 /* RP supports name service notifications */
-
 /**
  * struct rpmsg_hdr - common header for all rpmsg messages
  * @src: source address
@@ -452,7 +452,7 @@ static void *get_a_tx_buf(struct virtproc_info *vrp)
 
 	/* either pick the next unused tx buffer */
 	if (vrp->last_sbuf < vrp->num_sbufs)
-		ret = vrp->sbufs + vrp->buf_size * vrp->last_sbuf++;
+		ret = vrp->sbufs + vrp->sbuf_size * vrp->last_sbuf++;
 	/* or recycle a used one */
 	else
 		ret = virtqueue_get_buf(vrp->svq, &len);
@@ -578,7 +578,7 @@ static int rpmsg_send_offchannel_raw(struct rpmsg_device *rpdev,
 	 * messaging), or to improve the buffer allocator, to support
 	 * variable-length buffer sizes.
 	 */
-	if (len > vrp->buf_size - sizeof(struct rpmsg_hdr)) {
+	if (len > vrp->sbuf_size - sizeof(struct rpmsg_hdr)) {
 		dev_err(dev, "message is too big (%d)\n", len);
 		return -EMSGSIZE;
 	}
@@ -718,7 +718,7 @@ static int rpmsg_recv_single(struct virtproc_info *vrp, struct device *dev,
 	 * We currently use fixed-sized buffers, so trivially sanitize
 	 * the reported payload length.
 	 */
-	if (len > vrp->buf_size ||
+	if (len > vrp->rbuf_size ||
 	    msg->len > (len - sizeof(struct rpmsg_hdr))) {
 		dev_warn(dev, "inbound msg too big: (%d, %d)\n", len, msg->len);
 		return -EINVAL;
@@ -751,7 +751,7 @@ static int rpmsg_recv_single(struct virtproc_info *vrp, struct device *dev,
 		dev_warn(dev, "msg received with no recipient\n");
 
 	/* publish the real size of the buffer */
-	rpmsg_sg_init(&sg, msg, vrp->buf_size);
+	rpmsg_sg_init(&sg, msg, vrp->rbuf_size);
 
 	/* add the buffer back to the remote processor's virtqueue */
 	err = virtqueue_add_inbuf(vrp->rvq, &sg, 1, msg, GFP_KERNEL);
@@ -907,11 +907,24 @@ static int rpmsg_probe(struct virtio_device *vdev)
 	else
 		vrp->num_sbufs = MAX_RPMSG_NUM_BUFS;
 
-	vrp->buf_size = MAX_RPMSG_BUF_SIZE;
+	/* try to get buffer size from config space */
+	if (virtio_has_feature(vdev, VIRTIO_RPMSG_F_BUFSZ)) {
+		/* note: virtio_rpmsg_config is defined from remote view */
+		virtio_cread(vdev, struct virtio_rpmsg_config,
+			     txbuf_size, &vrp->rbuf_size);
+		virtio_cread(vdev, struct virtio_rpmsg_config,
+			     rxbuf_size, &vrp->sbuf_size);
+	}
+
+	/* use the default if resource table doesn't provide one */
+	if (!vrp->rbuf_size)
+		vrp->rbuf_size = MAX_RPMSG_BUF_SIZE;
+	if (!vrp->sbuf_size)
+		vrp->sbuf_size = MAX_RPMSG_BUF_SIZE;
 
 	/* allocate coherent memory for the buffers */
 	vrp->rbufs = dma_alloc_coherent(vdev->dev.parent->parent,
-					vrp->num_rbufs * vrp->buf_size,
+					vrp->num_rbufs * vrp->rbuf_size,
 					&vrp->rbufs_dma, GFP_KERNEL);
 	if (!vrp->rbufs) {
 		err = -ENOMEM;
@@ -922,7 +935,7 @@ static int rpmsg_probe(struct virtio_device *vdev)
 		vrp->rbufs, &vrp->rbufs_dma);
 
 	vrp->sbufs = dma_alloc_coherent(vdev->dev.parent->parent,
-					vrp->num_sbufs * vrp->buf_size,
+					vrp->num_sbufs * vrp->sbuf_size,
 					&vrp->sbufs_dma, GFP_KERNEL);
 	if (!vrp->sbufs) {
 		err = -ENOMEM;
@@ -935,9 +948,9 @@ static int rpmsg_probe(struct virtio_device *vdev)
 	/* set up the receive buffers */
 	for (i = 0; i < vrp->num_rbufs; i++) {
 		struct scatterlist sg;
-		void *cpu_addr = vrp->rbufs + i * vrp->buf_size;
+		void *cpu_addr = vrp->rbufs + i * vrp->rbuf_size;
 
-		rpmsg_sg_init(&sg, cpu_addr, vrp->buf_size);
+		rpmsg_sg_init(&sg, cpu_addr, vrp->rbuf_size);
 
 		err = virtqueue_add_inbuf(vrp->rvq, &sg, 1, cpu_addr,
 					  GFP_KERNEL);
@@ -984,11 +997,11 @@ static int rpmsg_probe(struct virtio_device *vdev)
 
 free_sbufs:
 	dma_free_coherent(vdev->dev.parent->parent,
-			  vrp->num_sbufs * vrp->buf_size,
+			  vrp->num_sbufs * vrp->sbuf_size,
 			  vrp->sbufs, vrp->sbufs_dma);
 free_rbufs:
 	dma_free_coherent(vdev->dev.parent->parent,
-			  vrp->num_rbufs * vrp->buf_size,
+			  vrp->num_rbufs * vrp->rbuf_size,
 			  vrp->rbufs, vrp->rbufs_dma);
 vqs_del:
 	vdev->config->del_vqs(vrp->vdev);
@@ -1023,10 +1036,10 @@ static void rpmsg_remove(struct virtio_device *vdev)
 	vdev->config->del_vqs(vrp->vdev);
 
 	dma_free_coherent(vdev->dev.parent->parent,
-			  vrp->num_sbufs * vrp->buf_size,
+			  vrp->num_sbufs * vrp->sbuf_size,
 			  vrp->sbufs, vrp->sbufs_dma);
 	dma_free_coherent(vdev->dev.parent->parent,
-			  vrp->num_rbufs * vrp->buf_size,
+			  vrp->num_rbufs * vrp->rbuf_size,
 			  vrp->rbufs, vrp->rbufs_dma);
 
 	kfree(vrp);
@@ -1039,6 +1052,7 @@ static struct virtio_device_id id_table[] = {
 
 static unsigned int features[] = {
 	VIRTIO_RPMSG_F_NS,
+	VIRTIO_RPMSG_F_BUFSZ,
 };
 
 static struct virtio_driver virtio_ipc_driver = {
include/uapi/linux/virtio_rpmsg.h new file +24 -0
new file mode 100644
index 0000000..24fa0dd
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Copyright (C) Pinecone Inc. 2019
+ * Copyright (C) Xiang Xiao <[email protected]>
+ */
+
+#ifndef _UAPI_LINUX_VIRTIO_RPMSG_H
+#define _UAPI_LINUX_VIRTIO_RPMSG_H
+
+#include <linux/types.h>
+
+/* The feature bitmap for virtio rpmsg */
+#define VIRTIO_RPMSG_F_NS	0 /* RP supports name service notifications */
+#define VIRTIO_RPMSG_F_BUFSZ	2 /* RP get buffer size from config space */
+
+struct virtio_rpmsg_config {
+	/* The tx/rx individual buffer size(if VIRTIO_RPMSG_F_BUFSZ) */
+	__u32 txbuf_size;
+	__u32 rxbuf_size;
+	__u32 reserved[14]; /* Reserve for the future use */
+	/* Put the customize config here */
+} __attribute__((packed));
+
+#endif /* _UAPI_LINUX_VIRTIO_RPMSG_H */
Signature · 1 line
2.7.4
#7 lore raw

Re: [PATCH 3/3] rpmsg: virtio_rpmsg_bus: get buffer size from config space

Hello Xiang,

Similar mechanism has been proposed by Loic 2 years ago (link to the series here https://lkml.org/lkml/2017/3/28/349).

Did you see them? Regarding history, patches seem just on hold...

Main differences (except interesting RX/TX size split) seems that you
- don't use the virtio_config_ops->get
- define a new feature VIRTIO_RPMSG_F_NS.

Regards
Arnaud

Quoted 203 lines · Xiang Xiao

512 bytes isn't always suitable for all case, let firmware maker decide the best value from resource table. enable by VIRTIO_RPMSG_F_BUFSZ feature bit.

---
 drivers/rpmsg/virtio_rpmsg_bus.c  | 50 +++++++++++++++++++++++++--------------
 include/uapi/linux/virtio_rpmsg.h | 24 +++++++++++++++++++
 2 files changed, 56 insertions(+), 18 deletions(-)
 create mode 100644 include/uapi/linux/virtio_rpmsg.h
drivers/rpmsg/virtio_rpmsg_bus.c +32 -18
index 59c4554..049dd97 100644
@@ -16,6 +16,7 @@
 #include <linux/virtio.h>
 #include <linux/virtio_ids.h>
 #include <linux/virtio_config.h>
+#include <linux/virtio_rpmsg.h>
 #include <linux/scatterlist.h>
 #include <linux/dma-mapping.h>
 #include <linux/slab.h>
@@ -38,7 +39,8 @@
  * @sbufs:	kernel address of tx buffers
  * @num_rbufs:	total number of buffers for rx
  * @num_sbufs:	total number of buffers for tx
- * @buf_size:	size of one rx or tx buffer
+ * @rbuf_size:	size of one rx buffer
+ * @sbuf_size:	size of one tx buffer
  * @last_sbuf:	index of last tx buffer used
  * @rbufs_dma:	dma base addr of rx buffers
  * @sbufs_dma:	dma base addr of tx buffers
@@ -61,7 +63,8 @@ struct virtproc_info {
 	void *rbufs, *sbufs;
 	unsigned int num_rbufs;
 	unsigned int num_sbufs;
-	unsigned int buf_size;
+	unsigned int rbuf_size;
+	unsigned int sbuf_size;
 	int last_sbuf;
 	dma_addr_t rbufs_dma;
 	dma_addr_t sbufs_dma;
@@ -73,9 +76,6 @@ struct virtproc_info {
 	struct rpmsg_endpoint *ns_ept;
 };
 
-/* The feature bitmap for virtio rpmsg */
-#define VIRTIO_RPMSG_F_NS	0 /* RP supports name service notifications */
-
 /**
  * struct rpmsg_hdr - common header for all rpmsg messages
  * @src: source address
@@ -452,7 +452,7 @@ static void *get_a_tx_buf(struct virtproc_info *vrp)
 
 	/* either pick the next unused tx buffer */
 	if (vrp->last_sbuf < vrp->num_sbufs)
-		ret = vrp->sbufs + vrp->buf_size * vrp->last_sbuf++;
+		ret = vrp->sbufs + vrp->sbuf_size * vrp->last_sbuf++;
 	/* or recycle a used one */
 	else
 		ret = virtqueue_get_buf(vrp->svq, &len);
@@ -578,7 +578,7 @@ static int rpmsg_send_offchannel_raw(struct rpmsg_device *rpdev,
 	 * messaging), or to improve the buffer allocator, to support
 	 * variable-length buffer sizes.
 	 */
-	if (len > vrp->buf_size - sizeof(struct rpmsg_hdr)) {
+	if (len > vrp->sbuf_size - sizeof(struct rpmsg_hdr)) {
 		dev_err(dev, "message is too big (%d)\n", len);
 		return -EMSGSIZE;
 	}
@@ -718,7 +718,7 @@ static int rpmsg_recv_single(struct virtproc_info *vrp, struct device *dev,
 	 * We currently use fixed-sized buffers, so trivially sanitize
 	 * the reported payload length.
 	 */
-	if (len > vrp->buf_size ||
+	if (len > vrp->rbuf_size ||
 	    msg->len > (len - sizeof(struct rpmsg_hdr))) {
 		dev_warn(dev, "inbound msg too big: (%d, %d)\n", len, msg->len);
 		return -EINVAL;
@@ -751,7 +751,7 @@ static int rpmsg_recv_single(struct virtproc_info *vrp, struct device *dev,
 		dev_warn(dev, "msg received with no recipient\n");
 
 	/* publish the real size of the buffer */
-	rpmsg_sg_init(&sg, msg, vrp->buf_size);
+	rpmsg_sg_init(&sg, msg, vrp->rbuf_size);
 
 	/* add the buffer back to the remote processor's virtqueue */
 	err = virtqueue_add_inbuf(vrp->rvq, &sg, 1, msg, GFP_KERNEL);
@@ -907,11 +907,24 @@ static int rpmsg_probe(struct virtio_device *vdev)
 	else
 		vrp->num_sbufs = MAX_RPMSG_NUM_BUFS;
 
-	vrp->buf_size = MAX_RPMSG_BUF_SIZE;
+	/* try to get buffer size from config space */
+	if (virtio_has_feature(vdev, VIRTIO_RPMSG_F_BUFSZ)) {
+		/* note: virtio_rpmsg_config is defined from remote view */
+		virtio_cread(vdev, struct virtio_rpmsg_config,
+			     txbuf_size, &vrp->rbuf_size);
+		virtio_cread(vdev, struct virtio_rpmsg_config,
+			     rxbuf_size, &vrp->sbuf_size);
+	}
+
+	/* use the default if resource table doesn't provide one */
+	if (!vrp->rbuf_size)
+		vrp->rbuf_size = MAX_RPMSG_BUF_SIZE;
+	if (!vrp->sbuf_size)
+		vrp->sbuf_size = MAX_RPMSG_BUF_SIZE;
 
 	/* allocate coherent memory for the buffers */
 	vrp->rbufs = dma_alloc_coherent(vdev->dev.parent->parent,
-					vrp->num_rbufs * vrp->buf_size,
+					vrp->num_rbufs * vrp->rbuf_size,
 					&vrp->rbufs_dma, GFP_KERNEL);
 	if (!vrp->rbufs) {
 		err = -ENOMEM;
@@ -922,7 +935,7 @@ static int rpmsg_probe(struct virtio_device *vdev)
 		vrp->rbufs, &vrp->rbufs_dma);
 
 	vrp->sbufs = dma_alloc_coherent(vdev->dev.parent->parent,
-					vrp->num_sbufs * vrp->buf_size,
+					vrp->num_sbufs * vrp->sbuf_size,
 					&vrp->sbufs_dma, GFP_KERNEL);
 	if (!vrp->sbufs) {
 		err = -ENOMEM;
@@ -935,9 +948,9 @@ static int rpmsg_probe(struct virtio_device *vdev)
 	/* set up the receive buffers */
 	for (i = 0; i < vrp->num_rbufs; i++) {
 		struct scatterlist sg;
-		void *cpu_addr = vrp->rbufs + i * vrp->buf_size;
+		void *cpu_addr = vrp->rbufs + i * vrp->rbuf_size;
 
-		rpmsg_sg_init(&sg, cpu_addr, vrp->buf_size);
+		rpmsg_sg_init(&sg, cpu_addr, vrp->rbuf_size);
 
 		err = virtqueue_add_inbuf(vrp->rvq, &sg, 1, cpu_addr,
 					  GFP_KERNEL);
@@ -984,11 +997,11 @@ static int rpmsg_probe(struct virtio_device *vdev)
 
 free_sbufs:
 	dma_free_coherent(vdev->dev.parent->parent,
-			  vrp->num_sbufs * vrp->buf_size,
+			  vrp->num_sbufs * vrp->sbuf_size,
 			  vrp->sbufs, vrp->sbufs_dma);
 free_rbufs:
 	dma_free_coherent(vdev->dev.parent->parent,
-			  vrp->num_rbufs * vrp->buf_size,
+			  vrp->num_rbufs * vrp->rbuf_size,
 			  vrp->rbufs, vrp->rbufs_dma);
 vqs_del:
 	vdev->config->del_vqs(vrp->vdev);
@@ -1023,10 +1036,10 @@ static void rpmsg_remove(struct virtio_device *vdev)
 	vdev->config->del_vqs(vrp->vdev);
 
 	dma_free_coherent(vdev->dev.parent->parent,
-			  vrp->num_sbufs * vrp->buf_size,
+			  vrp->num_sbufs * vrp->sbuf_size,
 			  vrp->sbufs, vrp->sbufs_dma);
 	dma_free_coherent(vdev->dev.parent->parent,
-			  vrp->num_rbufs * vrp->buf_size,
+			  vrp->num_rbufs * vrp->rbuf_size,
 			  vrp->rbufs, vrp->rbufs_dma);
 
 	kfree(vrp);
@@ -1039,6 +1052,7 @@ static struct virtio_device_id id_table[] = {
 
 static unsigned int features[] = {
 	VIRTIO_RPMSG_F_NS,
+	VIRTIO_RPMSG_F_BUFSZ,
 };
 
 static struct virtio_driver virtio_ipc_driver = {
include/uapi/linux/virtio_rpmsg.h new file +24 -0
new file mode 100644
index 0000000..24fa0dd
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Copyright (C) Pinecone Inc. 2019
+ * Copyright (C) Xiang Xiao <[email protected]>
+ */
+
+#ifndef _UAPI_LINUX_VIRTIO_RPMSG_H
+#define _UAPI_LINUX_VIRTIO_RPMSG_H
+
+#include <linux/types.h>
+
+/* The feature bitmap for virtio rpmsg */
+#define VIRTIO_RPMSG_F_NS	0 /* RP supports name service notifications */
+#define VIRTIO_RPMSG_F_BUFSZ	2 /* RP get buffer size from config space */
+
+struct virtio_rpmsg_config {
+	/* The tx/rx individual buffer size(if VIRTIO_RPMSG_F_BUFSZ) */
+	__u32 txbuf_size;
+	__u32 rxbuf_size;
+	__u32 reserved[14]; /* Reserve for the future use */
+	/* Put the customize config here */
+} __attribute__((packed));
+
+#endif /* _UAPI_LINUX_VIRTIO_RPMSG_H */
author
#8 lore raw

Re: [PATCH 3/3] rpmsg: virtio_rpmsg_bus: get buffer size from config space

Quoted 8 lines · Arnaud Pouliquen

Hello Xiang,

Similar mechanism has been proposed by Loic 2 years ago (link to the series here https://lkml.org/lkml/2017/3/28/349).

Did you see them? Regarding history, patches seem just on hold...

Just saw this patchset, so it's common problem hit by many vendor, rpmsg framework need to address it.:)

Quoted 2 lines

Main differences (except interesting RX/TX size split) seems that you
- don't use the virtio_config_ops->get

virtio_cread call virtio_config_ops->get internally, the ideal is same for both patch, just the implementation detail is different.

Quoted 1 line

- define a new feature VIRTIO_RPMSG_F_NS.

I add this flag to keep the compatibility with old remote peer, and also follow the common virito driver practice.

Quoted 209 lines

Regards
Arnaud

Quoted 203 lines · Xiang Xiao

512 bytes isn't always suitable for all case, let firmware maker decide the best value from resource table. enable by VIRTIO_RPMSG_F_BUFSZ feature bit.

---
 drivers/rpmsg/virtio_rpmsg_bus.c  | 50 +++++++++++++++++++++++++--------------
 include/uapi/linux/virtio_rpmsg.h | 24 +++++++++++++++++++
 2 files changed, 56 insertions(+), 18 deletions(-)
 create mode 100644 include/uapi/linux/virtio_rpmsg.h
drivers/rpmsg/virtio_rpmsg_bus.c +32 -18
index 59c4554..049dd97 100644
@@ -16,6 +16,7 @@
 #include <linux/virtio.h>
 #include <linux/virtio_ids.h>
 #include <linux/virtio_config.h>
+#include <linux/virtio_rpmsg.h>
 #include <linux/scatterlist.h>
 #include <linux/dma-mapping.h>
 #include <linux/slab.h>
@@ -38,7 +39,8 @@
  * @sbufs:   kernel address of tx buffers
  * @num_rbufs:       total number of buffers for rx
  * @num_sbufs:       total number of buffers for tx
- * @buf_size:        size of one rx or tx buffer
+ * @rbuf_size:       size of one rx buffer
+ * @sbuf_size:       size of one tx buffer
  * @last_sbuf:       index of last tx buffer used
  * @rbufs_dma:       dma base addr of rx buffers
  * @sbufs_dma:       dma base addr of tx buffers
@@ -61,7 +63,8 @@ struct virtproc_info {
      void *rbufs, *sbufs;
      unsigned int num_rbufs;
      unsigned int num_sbufs;
-     unsigned int buf_size;
+     unsigned int rbuf_size;
+     unsigned int sbuf_size;
      int last_sbuf;
      dma_addr_t rbufs_dma;
      dma_addr_t sbufs_dma;
@@ -73,9 +76,6 @@ struct virtproc_info {
      struct rpmsg_endpoint *ns_ept;
 };
 
-/* The feature bitmap for virtio rpmsg */
-#define VIRTIO_RPMSG_F_NS    0 /* RP supports name service notifications */
-
 /**
  * struct rpmsg_hdr - common header for all rpmsg messages
  * @src: source address
@@ -452,7 +452,7 @@ static void *get_a_tx_buf(struct virtproc_info *vrp)
 
      /* either pick the next unused tx buffer */
      if (vrp->last_sbuf < vrp->num_sbufs)
-             ret = vrp->sbufs + vrp->buf_size * vrp->last_sbuf++;
+             ret = vrp->sbufs + vrp->sbuf_size * vrp->last_sbuf++;
      /* or recycle a used one */
      else
              ret = virtqueue_get_buf(vrp->svq, &len);
@@ -578,7 +578,7 @@ static int rpmsg_send_offchannel_raw(struct rpmsg_device *rpdev,
       * messaging), or to improve the buffer allocator, to support
       * variable-length buffer sizes.
       */
-     if (len > vrp->buf_size - sizeof(struct rpmsg_hdr)) {
+     if (len > vrp->sbuf_size - sizeof(struct rpmsg_hdr)) {
              dev_err(dev, "message is too big (%d)\n", len);
              return -EMSGSIZE;
      }
@@ -718,7 +718,7 @@ static int rpmsg_recv_single(struct virtproc_info *vrp, struct device *dev,
       * We currently use fixed-sized buffers, so trivially sanitize
       * the reported payload length.
       */
-     if (len > vrp->buf_size ||
+     if (len > vrp->rbuf_size ||
          msg->len > (len - sizeof(struct rpmsg_hdr))) {
              dev_warn(dev, "inbound msg too big: (%d, %d)\n", len, msg->len);
              return -EINVAL;
@@ -751,7 +751,7 @@ static int rpmsg_recv_single(struct virtproc_info *vrp, struct device *dev,
              dev_warn(dev, "msg received with no recipient\n");
 
      /* publish the real size of the buffer */
-     rpmsg_sg_init(&sg, msg, vrp->buf_size);
+     rpmsg_sg_init(&sg, msg, vrp->rbuf_size);
 
      /* add the buffer back to the remote processor's virtqueue */
      err = virtqueue_add_inbuf(vrp->rvq, &sg, 1, msg, GFP_KERNEL);
@@ -907,11 +907,24 @@ static int rpmsg_probe(struct virtio_device *vdev)
      else
              vrp->num_sbufs = MAX_RPMSG_NUM_BUFS;
 
-     vrp->buf_size = MAX_RPMSG_BUF_SIZE;
+     /* try to get buffer size from config space */
+     if (virtio_has_feature(vdev, VIRTIO_RPMSG_F_BUFSZ)) {
+             /* note: virtio_rpmsg_config is defined from remote view */
+             virtio_cread(vdev, struct virtio_rpmsg_config,
+                          txbuf_size, &vrp->rbuf_size);
+             virtio_cread(vdev, struct virtio_rpmsg_config,
+                          rxbuf_size, &vrp->sbuf_size);
+     }
+
+     /* use the default if resource table doesn't provide one */
+     if (!vrp->rbuf_size)
+             vrp->rbuf_size = MAX_RPMSG_BUF_SIZE;
+     if (!vrp->sbuf_size)
+             vrp->sbuf_size = MAX_RPMSG_BUF_SIZE;
 
      /* allocate coherent memory for the buffers */
      vrp->rbufs = dma_alloc_coherent(vdev->dev.parent->parent,
-                                     vrp->num_rbufs * vrp->buf_size,
+                                     vrp->num_rbufs * vrp->rbuf_size,
                                      &vrp->rbufs_dma, GFP_KERNEL);
      if (!vrp->rbufs) {
              err = -ENOMEM;
@@ -922,7 +935,7 @@ static int rpmsg_probe(struct virtio_device *vdev)
              vrp->rbufs, &vrp->rbufs_dma);
 
      vrp->sbufs = dma_alloc_coherent(vdev->dev.parent->parent,
-                                     vrp->num_sbufs * vrp->buf_size,
+                                     vrp->num_sbufs * vrp->sbuf_size,
                                      &vrp->sbufs_dma, GFP_KERNEL);
      if (!vrp->sbufs) {
              err = -ENOMEM;
@@ -935,9 +948,9 @@ static int rpmsg_probe(struct virtio_device *vdev)
      /* set up the receive buffers */
      for (i = 0; i < vrp->num_rbufs; i++) {
              struct scatterlist sg;
-             void *cpu_addr = vrp->rbufs + i * vrp->buf_size;
+             void *cpu_addr = vrp->rbufs + i * vrp->rbuf_size;
 
-             rpmsg_sg_init(&sg, cpu_addr, vrp->buf_size);
+             rpmsg_sg_init(&sg, cpu_addr, vrp->rbuf_size);
 
              err = virtqueue_add_inbuf(vrp->rvq, &sg, 1, cpu_addr,
                                        GFP_KERNEL);
@@ -984,11 +997,11 @@ static int rpmsg_probe(struct virtio_device *vdev)
 
 free_sbufs:
      dma_free_coherent(vdev->dev.parent->parent,
-                       vrp->num_sbufs * vrp->buf_size,
+                       vrp->num_sbufs * vrp->sbuf_size,
                        vrp->sbufs, vrp->sbufs_dma);
 free_rbufs:
      dma_free_coherent(vdev->dev.parent->parent,
-                       vrp->num_rbufs * vrp->buf_size,
+                       vrp->num_rbufs * vrp->rbuf_size,
                        vrp->rbufs, vrp->rbufs_dma);
 vqs_del:
      vdev->config->del_vqs(vrp->vdev);
@@ -1023,10 +1036,10 @@ static void rpmsg_remove(struct virtio_device *vdev)
      vdev->config->del_vqs(vrp->vdev);
 
      dma_free_coherent(vdev->dev.parent->parent,
-                       vrp->num_sbufs * vrp->buf_size,
+                       vrp->num_sbufs * vrp->sbuf_size,
                        vrp->sbufs, vrp->sbufs_dma);
      dma_free_coherent(vdev->dev.parent->parent,
-                       vrp->num_rbufs * vrp->buf_size,
+                       vrp->num_rbufs * vrp->rbuf_size,
                        vrp->rbufs, vrp->rbufs_dma);
 
      kfree(vrp);
@@ -1039,6 +1052,7 @@ static struct virtio_device_id id_table[] = {
 
 static unsigned int features[] = {
      VIRTIO_RPMSG_F_NS,
+     VIRTIO_RPMSG_F_BUFSZ,
 };
 
 static struct virtio_driver virtio_ipc_driver = {
include/uapi/linux/virtio_rpmsg.h new file +24 -0
new file mode 100644
index 0000000..24fa0dd
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Copyright (C) Pinecone Inc. 2019
+ * Copyright (C) Xiang Xiao <[email protected]>
+ */
+
+#ifndef _UAPI_LINUX_VIRTIO_RPMSG_H
+#define _UAPI_LINUX_VIRTIO_RPMSG_H
+
+#include <linux/types.h>
+
+/* The feature bitmap for virtio rpmsg */
+#define VIRTIO_RPMSG_F_NS    0 /* RP supports name service notifications */
+#define VIRTIO_RPMSG_F_BUFSZ 2 /* RP get buffer size from config space */
+
+struct virtio_rpmsg_config {
+     /* The tx/rx individual buffer size(if VIRTIO_RPMSG_F_BUFSZ) */
+     __u32 txbuf_size;
+     __u32 rxbuf_size;
+     __u32 reserved[14]; /* Reserve for the future use */
+     /* Put the customize config here */
+} __attribute__((packed));
+
+#endif /* _UAPI_LINUX_VIRTIO_RPMSG_H */
#9 lore raw

Re: [PATCH 3/3] rpmsg: virtio_rpmsg_bus: get buffer size from config space

Hello Xiang,

Quoted 16 lines · xiang xiao
Quoted 8 lines · Arnaud Pouliquen

Hello Xiang,

Similar mechanism has been proposed by Loic 2 years ago (link to the series here https://lkml.org/lkml/2017/3/28/349).

Did you see them? Regarding history, patches seem just on hold...

Just saw this patchset, so it's common problem hit by many vendor, rpmsg framework need to address it.:)

Quoted 2 lines

Main differences (except interesting RX/TX size split) seems that you
- don't use the virtio_config_ops->get

Quoted 7 lines

virtio_cread call virtio_config_ops->get internally, the ideal is same for both patch, just the implementation detail is different.

Quoted 1 line

- define a new feature VIRTIO_RPMSG_F_NS.

I add this flag to keep the compatibility with old remote peer, and also follow the common virito driver practice.

I discussed with Loic, he is ok to go further with your patch and abandon his one. Please find some remarks below in-line

Quoted 100 lines
Quoted 100 lines

Regards
Arnaud

Quoted 93 lines · Xiang Xiao

512 bytes isn't always suitable for all case, let firmware maker decide the best value from resource table. enable by VIRTIO_RPMSG_F_BUFSZ feature bit.

---
 drivers/rpmsg/virtio_rpmsg_bus.c  | 50 +++++++++++++++++++++++++--------------
 include/uapi/linux/virtio_rpmsg.h | 24 +++++++++++++++++++
 2 files changed, 56 insertions(+), 18 deletions(-)
 create mode 100644 include/uapi/linux/virtio_rpmsg.h
drivers/rpmsg/virtio_rpmsg_bus.c +9 -9
index 59c4554..049dd97 100644
@@ -16,6 +16,7 @@
 #include <linux/virtio.h>
 #include <linux/virtio_ids.h>
 #include <linux/virtio_config.h>
+#include <linux/virtio_rpmsg.h>
 #include <linux/scatterlist.h>
 #include <linux/dma-mapping.h>
 #include <linux/slab.h>
@@ -38,7 +39,8 @@
  * @sbufs:   kernel address of tx buffers
  * @num_rbufs:       total number of buffers for rx
  * @num_sbufs:       total number of buffers for tx
- * @buf_size:        size of one rx or tx buffer
+ * @rbuf_size:       size of one rx buffer
+ * @sbuf_size:       size of one tx buffer
  * @last_sbuf:       index of last tx buffer used
  * @rbufs_dma:       dma base addr of rx buffers
  * @sbufs_dma:       dma base addr of tx buffers
@@ -61,7 +63,8 @@ struct virtproc_info {
      void *rbufs, *sbufs;
      unsigned int num_rbufs;
      unsigned int num_sbufs;
-     unsigned int buf_size;
+     unsigned int rbuf_size;
+     unsigned int sbuf_size;
      int last_sbuf;
      dma_addr_t rbufs_dma;
      dma_addr_t sbufs_dma;
@@ -73,9 +76,6 @@ struct virtproc_info {
      struct rpmsg_endpoint *ns_ept;
 };
 
-/* The feature bitmap for virtio rpmsg */
-#define VIRTIO_RPMSG_F_NS    0 /* RP supports name service notifications */
-
 /**
  * struct rpmsg_hdr - common header for all rpmsg messages
  * @src: source address
@@ -452,7 +452,7 @@ static void *get_a_tx_buf(struct virtproc_info *vrp)
 
      /* either pick the next unused tx buffer */
      if (vrp->last_sbuf < vrp->num_sbufs)
-             ret = vrp->sbufs + vrp->buf_size * vrp->last_sbuf++;
+             ret = vrp->sbufs + vrp->sbuf_size * vrp->last_sbuf++;
      /* or recycle a used one */
      else
              ret = virtqueue_get_buf(vrp->svq, &len);
@@ -578,7 +578,7 @@ static int rpmsg_send_offchannel_raw(struct rpmsg_device *rpdev,
       * messaging), or to improve the buffer allocator, to support
       * variable-length buffer sizes.
       */
-     if (len > vrp->buf_size - sizeof(struct rpmsg_hdr)) {
+     if (len > vrp->sbuf_size - sizeof(struct rpmsg_hdr)) {
              dev_err(dev, "message is too big (%d)\n", len);
              return -EMSGSIZE;
      }
@@ -718,7 +718,7 @@ static int rpmsg_recv_single(struct virtproc_info *vrp, struct device *dev,
       * We currently use fixed-sized buffers, so trivially sanitize
       * the reported payload length.
       */
-     if (len > vrp->buf_size ||
+     if (len > vrp->rbuf_size ||
          msg->len > (len - sizeof(struct rpmsg_hdr))) {
              dev_warn(dev, "inbound msg too big: (%d, %d)\n", len, msg->len);
              return -EINVAL;
@@ -751,7 +751,7 @@ static int rpmsg_recv_single(struct virtproc_info *vrp, struct device *dev,
              dev_warn(dev, "msg received with no recipient\n");
 
      /* publish the real size of the buffer */
-     rpmsg_sg_init(&sg, msg, vrp->buf_size);
+     rpmsg_sg_init(&sg, msg, vrp->rbuf_size);
 
      /* add the buffer back to the remote processor's virtqueue */
      err = virtqueue_add_inbuf(vrp->rvq, &sg, 1, msg, GFP_KERNEL);
@@ -907,11 +907,24 @@ static int rpmsg_probe(struct virtio_device *vdev)
      else
              vrp->num_sbufs = MAX_RPMSG_NUM_BUFS;
 
Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
drivers/rpmsg/virtio_rpmsg_bus.c +4 -1
-     vrp->buf_size = MAX_RPMSG_BUF_SIZE;
+     /* try to get buffer size from config space */
+     if (virtio_has_feature(vdev, VIRTIO_RPMSG_F_BUFSZ)) {
+             /* note: virtio_rpmsg_config is defined from remote view */
+             virtio_cread(vdev, struct virtio_rpmsg_config,
Quoted 8 lines

drivers/rpmsg/virtio_rpmsg_bus.c static int rpmsg_probe(struct virtio_device *vdev) · line 921

Quoted 8 lines
drivers/rpmsg/virtio_rpmsg_bus.c +8 -0
+                          txbuf_size, &vrp->rbuf_size);
+             virtio_cread(vdev, struct virtio_rpmsg_config,
+                          rxbuf_size, &vrp->sbuf_size);
+     }
+
+     /* use the default if resource table doesn't provide one */
+     if (!vrp->rbuf_size)
+             vrp->rbuf_size = MAX_RPMSG_BUF_SIZE;

In this case constant should be renamed DEFAULT_RPMSG_BUF_SIZE as it is no more a max value

Quoted 2 lines
Quoted 2 lines

drivers/rpmsg/virtio_rpmsg_bus.c static int rpmsg_probe(struct virtio_device *vdev) · line 923

Quoted 2 lines
drivers/rpmsg/virtio_rpmsg_bus.c +2 -0
+     if (!vrp->sbuf_size)
+             vrp->sbuf_size = MAX_RPMSG_BUF_SIZE;

Here, if the config space exists you need to update it in consequence to ensure coherency with the remote processor config.

Quoted 63 lines
Quoted 63 lines
Quoted 63 lines
drivers/rpmsg/virtio_rpmsg_bus.c +9 -8
      /* allocate coherent memory for the buffers */
      vrp->rbufs = dma_alloc_coherent(vdev->dev.parent->parent,
-                                     vrp->num_rbufs * vrp->buf_size,
+                                     vrp->num_rbufs * vrp->rbuf_size,
                                      &vrp->rbufs_dma, GFP_KERNEL);
      if (!vrp->rbufs) {
              err = -ENOMEM;
@@ -922,7 +935,7 @@ static int rpmsg_probe(struct virtio_device *vdev)
              vrp->rbufs, &vrp->rbufs_dma);
 
      vrp->sbufs = dma_alloc_coherent(vdev->dev.parent->parent,
-                                     vrp->num_sbufs * vrp->buf_size,
+                                     vrp->num_sbufs * vrp->sbuf_size,
                                      &vrp->sbufs_dma, GFP_KERNEL);
      if (!vrp->sbufs) {
              err = -ENOMEM;
@@ -935,9 +948,9 @@ static int rpmsg_probe(struct virtio_device *vdev)
      /* set up the receive buffers */
      for (i = 0; i < vrp->num_rbufs; i++) {
              struct scatterlist sg;
-             void *cpu_addr = vrp->rbufs + i * vrp->buf_size;
+             void *cpu_addr = vrp->rbufs + i * vrp->rbuf_size;
 
-             rpmsg_sg_init(&sg, cpu_addr, vrp->buf_size);
+             rpmsg_sg_init(&sg, cpu_addr, vrp->rbuf_size);
 
              err = virtqueue_add_inbuf(vrp->rvq, &sg, 1, cpu_addr,
                                        GFP_KERNEL);
@@ -984,11 +997,11 @@ static int rpmsg_probe(struct virtio_device *vdev)
 
 free_sbufs:
      dma_free_coherent(vdev->dev.parent->parent,
-                       vrp->num_sbufs * vrp->buf_size,
+                       vrp->num_sbufs * vrp->sbuf_size,
                        vrp->sbufs, vrp->sbufs_dma);
 free_rbufs:
      dma_free_coherent(vdev->dev.parent->parent,
-                       vrp->num_rbufs * vrp->buf_size,
+                       vrp->num_rbufs * vrp->rbuf_size,
                        vrp->rbufs, vrp->rbufs_dma);
 vqs_del:
      vdev->config->del_vqs(vrp->vdev);
@@ -1023,10 +1036,10 @@ static void rpmsg_remove(struct virtio_device *vdev)
      vdev->config->del_vqs(vrp->vdev);
 
      dma_free_coherent(vdev->dev.parent->parent,
-                       vrp->num_sbufs * vrp->buf_size,
+                       vrp->num_sbufs * vrp->sbuf_size,
                        vrp->sbufs, vrp->sbufs_dma);
      dma_free_coherent(vdev->dev.parent->parent,
-                       vrp->num_rbufs * vrp->buf_size,
+                       vrp->num_rbufs * vrp->rbuf_size,
                        vrp->rbufs, vrp->rbufs_dma);
 
      kfree(vrp);
@@ -1039,6 +1052,7 @@ static struct virtio_device_id id_table[] = {
 
 static unsigned int features[] = {
      VIRTIO_RPMSG_F_NS,
+     VIRTIO_RPMSG_F_BUFSZ,
 };
 
Quoted 6 lines
Quoted 6 lines

include/uapi/linux/virtio_rpmsg.h

Quoted 6 lines
drivers/rpmsg/virtio_rpmsg_bus.c +0 -0
 static struct virtio_driver virtio_ipc_driver = {
include/uapi/linux/virtio_rpmsg.h new file +0 -0
new file mode 100644
index 0000000..24fa0dd

Strange to define a user space API for kernel usage need. Could you elaborate?

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
include/uapi/linux/virtio_rpmsg.h new file +6 -0
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Copyright (C) Pinecone Inc. 2019
+ * Copyright (C) Xiang Xiao <[email protected]>
+ */
+
Quoted 8 lines
Quoted 8 lines

include/uapi/linux/virtio_rpmsg.h line 14

Quoted 8 lines
include/uapi/linux/virtio_rpmsg.h new file +8 -0
+#ifndef _UAPI_LINUX_VIRTIO_RPMSG_H
+#define _UAPI_LINUX_VIRTIO_RPMSG_H
+
+#include <linux/types.h>
+
+/* The feature bitmap for virtio rpmsg */
+#define VIRTIO_RPMSG_F_NS    0 /* RP supports name service notifications */
+#define VIRTIO_RPMSG_F_BUFSZ 2 /* RP get buffer size from config space */

Would be useful to document it in rpmsg.txt

Quoted 9 lines
Quoted 9 lines

include/uapi/linux/virtio_rpmsg.h line 23

Quoted 9 lines
include/uapi/linux/virtio_rpmsg.h new file +9 -0
+
+struct virtio_rpmsg_config {
+     /* The tx/rx individual buffer size(if VIRTIO_RPMSG_F_BUFSZ) */
+     __u32 txbuf_size;
+     __u32 rxbuf_size;
+     __u32 reserved[14]; /* Reserve for the future use */
+     /* Put the customize config here */
+} __attribute__((packed));
+

Wouldn't it be better to add an identifier and a version fields at the beginning of the structure? Idea would be to simplify a future extension In this case is VIRTIO_RPMSG_F_BUFSZ still useful?

Quoted 2 lines
Quoted 2 lines

include/uapi/linux/virtio_rpmsg.h line 24

Quoted 2 lines
include/uapi/linux/virtio_rpmsg.h new file +1 -0
+#endif /* _UAPI_LINUX_VIRTIO_RPMSG_H */

--
Thanks
Arnaud

author
#10 lore raw

Re: [PATCH 3/3] rpmsg: virtio_rpmsg_bus: get buffer size from config space

Quoted 129 lines · Arnaud Pouliquen

Hello Xiang,

Quoted 16 lines · xiang xiao
Quoted 8 lines · Arnaud Pouliquen

Hello Xiang,

Similar mechanism has been proposed by Loic 2 years ago (link to the series here https://lkml.org/lkml/2017/3/28/349).

Did you see them? Regarding history, patches seem just on hold...

Just saw this patchset, so it's common problem hit by many vendor, rpmsg framework need to address it.:)

Quoted 2 lines

Main differences (except interesting RX/TX size split) seems that you
- don't use the virtio_config_ops->get

Quoted 7 lines

virtio_cread call virtio_config_ops->get internally, the ideal is same for both patch, just the implementation detail is different.

Quoted 1 line

- define a new feature VIRTIO_RPMSG_F_NS.

I add this flag to keep the compatibility with old remote peer, and also follow the common virito driver practice.

I discussed with Loic, he is ok to go further with your patch and abandon his one. Please find some remarks below in-line

Quoted 100 lines
Quoted 100 lines

Regards
Arnaud

Quoted 93 lines · Xiang Xiao

512 bytes isn't always suitable for all case, let firmware maker decide the best value from resource table. enable by VIRTIO_RPMSG_F_BUFSZ feature bit.

---
 drivers/rpmsg/virtio_rpmsg_bus.c  | 50 +++++++++++++++++++++++++--------------
 include/uapi/linux/virtio_rpmsg.h | 24 +++++++++++++++++++
 2 files changed, 56 insertions(+), 18 deletions(-)
 create mode 100644 include/uapi/linux/virtio_rpmsg.h
drivers/rpmsg/virtio_rpmsg_bus.c +9 -9
index 59c4554..049dd97 100644
@@ -16,6 +16,7 @@
 #include <linux/virtio.h>
 #include <linux/virtio_ids.h>
 #include <linux/virtio_config.h>
+#include <linux/virtio_rpmsg.h>
 #include <linux/scatterlist.h>
 #include <linux/dma-mapping.h>
 #include <linux/slab.h>
@@ -38,7 +39,8 @@
  * @sbufs:   kernel address of tx buffers
  * @num_rbufs:       total number of buffers for rx
  * @num_sbufs:       total number of buffers for tx
- * @buf_size:        size of one rx or tx buffer
+ * @rbuf_size:       size of one rx buffer
+ * @sbuf_size:       size of one tx buffer
  * @last_sbuf:       index of last tx buffer used
  * @rbufs_dma:       dma base addr of rx buffers
  * @sbufs_dma:       dma base addr of tx buffers
@@ -61,7 +63,8 @@ struct virtproc_info {
      void *rbufs, *sbufs;
      unsigned int num_rbufs;
      unsigned int num_sbufs;
-     unsigned int buf_size;
+     unsigned int rbuf_size;
+     unsigned int sbuf_size;
      int last_sbuf;
      dma_addr_t rbufs_dma;
      dma_addr_t sbufs_dma;
@@ -73,9 +76,6 @@ struct virtproc_info {
      struct rpmsg_endpoint *ns_ept;
 };
 
-/* The feature bitmap for virtio rpmsg */
-#define VIRTIO_RPMSG_F_NS    0 /* RP supports name service notifications */
-
 /**
  * struct rpmsg_hdr - common header for all rpmsg messages
  * @src: source address
@@ -452,7 +452,7 @@ static void *get_a_tx_buf(struct virtproc_info *vrp)
 
      /* either pick the next unused tx buffer */
      if (vrp->last_sbuf < vrp->num_sbufs)
-             ret = vrp->sbufs + vrp->buf_size * vrp->last_sbuf++;
+             ret = vrp->sbufs + vrp->sbuf_size * vrp->last_sbuf++;
      /* or recycle a used one */
      else
              ret = virtqueue_get_buf(vrp->svq, &len);
@@ -578,7 +578,7 @@ static int rpmsg_send_offchannel_raw(struct rpmsg_device *rpdev,
       * messaging), or to improve the buffer allocator, to support
       * variable-length buffer sizes.
       */
-     if (len > vrp->buf_size - sizeof(struct rpmsg_hdr)) {
+     if (len > vrp->sbuf_size - sizeof(struct rpmsg_hdr)) {
              dev_err(dev, "message is too big (%d)\n", len);
              return -EMSGSIZE;
      }
@@ -718,7 +718,7 @@ static int rpmsg_recv_single(struct virtproc_info *vrp, struct device *dev,
       * We currently use fixed-sized buffers, so trivially sanitize
       * the reported payload length.
       */
-     if (len > vrp->buf_size ||
+     if (len > vrp->rbuf_size ||
          msg->len > (len - sizeof(struct rpmsg_hdr))) {
              dev_warn(dev, "inbound msg too big: (%d, %d)\n", len, msg->len);
              return -EINVAL;
@@ -751,7 +751,7 @@ static int rpmsg_recv_single(struct virtproc_info *vrp, struct device *dev,
              dev_warn(dev, "msg received with no recipient\n");
 
      /* publish the real size of the buffer */
-     rpmsg_sg_init(&sg, msg, vrp->buf_size);
+     rpmsg_sg_init(&sg, msg, vrp->rbuf_size);
 
      /* add the buffer back to the remote processor's virtqueue */
      err = virtqueue_add_inbuf(vrp->rvq, &sg, 1, msg, GFP_KERNEL);
@@ -907,11 +907,24 @@ static int rpmsg_probe(struct virtio_device *vdev)
      else
              vrp->num_sbufs = MAX_RPMSG_NUM_BUFS;
 
Quoted 15 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
drivers/rpmsg/virtio_rpmsg_bus.c +4 -1
-     vrp->buf_size = MAX_RPMSG_BUF_SIZE;
+     /* try to get buffer size from config space */
+     if (virtio_has_feature(vdev, VIRTIO_RPMSG_F_BUFSZ)) {
+             /* note: virtio_rpmsg_config is defined from remote view */
+             virtio_cread(vdev, struct virtio_rpmsg_config,
Quoted 8 lines
Quoted 8 lines

drivers/rpmsg/virtio_rpmsg_bus.c static int rpmsg_probe(struct virtio_device *vdev) · line 921

Quoted 8 lines
drivers/rpmsg/virtio_rpmsg_bus.c +8 -0
+                          txbuf_size, &vrp->rbuf_size);
+             virtio_cread(vdev, struct virtio_rpmsg_config,
+                          rxbuf_size, &vrp->sbuf_size);
+     }
+
+     /* use the default if resource table doesn't provide one */
+     if (!vrp->rbuf_size)
+             vrp->rbuf_size = MAX_RPMSG_BUF_SIZE;

In this case constant should be renamed DEFAULT_RPMSG_BUF_SIZE as it is no more a max value

Yes, DEFAULT_RPMSG_BUF_SIZE is more reasonable now.

Quoted 4 lines
Quoted 2 lines
Quoted 2 lines

drivers/rpmsg/virtio_rpmsg_bus.c static int rpmsg_probe(struct virtio_device *vdev) · line 923

Quoted 2 lines
drivers/rpmsg/virtio_rpmsg_bus.c +2 -0
+     if (!vrp->sbuf_size)
+             vrp->sbuf_size = MAX_RPMSG_BUF_SIZE;

Here, if the config space exists you need to update it in consequence to ensure coherency with the remote processor config.

The update is already done in if (virtio_has_feature(vdev, VIRTIO_RPMSG_F_BUFSZ)), here just handle the zero value in config space which mean the remote side want to use the default value even VIRTIO_RPMSG_F_BUFSZ set.
For example:
1.remote side want to change one direction buffer size, but keep another direction as default
2.or remote side want to change other config options(define in the furture) not the buffer size

Quoted 64 lines
Quoted 64 lines
Quoted 64 lines
Quoted 64 lines
drivers/rpmsg/virtio_rpmsg_bus.c +9 -8
      /* allocate coherent memory for the buffers */
      vrp->rbufs = dma_alloc_coherent(vdev->dev.parent->parent,
-                                     vrp->num_rbufs * vrp->buf_size,
+                                     vrp->num_rbufs * vrp->rbuf_size,
                                      &vrp->rbufs_dma, GFP_KERNEL);
      if (!vrp->rbufs) {
              err = -ENOMEM;
@@ -922,7 +935,7 @@ static int rpmsg_probe(struct virtio_device *vdev)
              vrp->rbufs, &vrp->rbufs_dma);
 
      vrp->sbufs = dma_alloc_coherent(vdev->dev.parent->parent,
-                                     vrp->num_sbufs * vrp->buf_size,
+                                     vrp->num_sbufs * vrp->sbuf_size,
                                      &vrp->sbufs_dma, GFP_KERNEL);
      if (!vrp->sbufs) {
              err = -ENOMEM;
@@ -935,9 +948,9 @@ static int rpmsg_probe(struct virtio_device *vdev)
      /* set up the receive buffers */
      for (i = 0; i < vrp->num_rbufs; i++) {
              struct scatterlist sg;
-             void *cpu_addr = vrp->rbufs + i * vrp->buf_size;
+             void *cpu_addr = vrp->rbufs + i * vrp->rbuf_size;
 
-             rpmsg_sg_init(&sg, cpu_addr, vrp->buf_size);
+             rpmsg_sg_init(&sg, cpu_addr, vrp->rbuf_size);
 
              err = virtqueue_add_inbuf(vrp->rvq, &sg, 1, cpu_addr,
                                        GFP_KERNEL);
@@ -984,11 +997,11 @@ static int rpmsg_probe(struct virtio_device *vdev)
 
 free_sbufs:
      dma_free_coherent(vdev->dev.parent->parent,
-                       vrp->num_sbufs * vrp->buf_size,
+                       vrp->num_sbufs * vrp->sbuf_size,
                        vrp->sbufs, vrp->sbufs_dma);
 free_rbufs:
      dma_free_coherent(vdev->dev.parent->parent,
-                       vrp->num_rbufs * vrp->buf_size,
+                       vrp->num_rbufs * vrp->rbuf_size,
                        vrp->rbufs, vrp->rbufs_dma);
 vqs_del:
      vdev->config->del_vqs(vrp->vdev);
@@ -1023,10 +1036,10 @@ static void rpmsg_remove(struct virtio_device *vdev)
      vdev->config->del_vqs(vrp->vdev);
 
      dma_free_coherent(vdev->dev.parent->parent,
-                       vrp->num_sbufs * vrp->buf_size,
+                       vrp->num_sbufs * vrp->sbuf_size,
                        vrp->sbufs, vrp->sbufs_dma);
      dma_free_coherent(vdev->dev.parent->parent,
-                       vrp->num_rbufs * vrp->buf_size,
+                       vrp->num_rbufs * vrp->rbuf_size,
                        vrp->rbufs, vrp->rbufs_dma);
 
      kfree(vrp);
@@ -1039,6 +1052,7 @@ static struct virtio_device_id id_table[] = {
 
 static unsigned int features[] = {
      VIRTIO_RPMSG_F_NS,
+     VIRTIO_RPMSG_F_BUFSZ,
 };
 
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines

include/uapi/linux/virtio_rpmsg.h

Quoted 6 lines
drivers/rpmsg/virtio_rpmsg_bus.c +0 -0
 static struct virtio_driver virtio_ipc_driver = {
include/uapi/linux/virtio_rpmsg.h new file +0 -0
new file mode 100644
index 0000000..24fa0dd

Strange to define a user space API for kernel usage need. Could you elaborate?

I just follow the practice other virtio drivers(e.g. include/uapi/virtio_net.h) applied, but rpmsg driver don't need to talk with the host VM software like other virtio driver, yes this header file isn't really needed.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
include/uapi/linux/virtio_rpmsg.h new file +7 -0
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Copyright (C) Pinecone Inc. 2019
+ * Copyright (C) Xiang Xiao <[email protected]>
+ */
+
+#ifndef _UAPI_LINUX_VIRTIO_RPMSG_H
Quoted 8 lines
Quoted 7 lines
Quoted 7 lines

include/uapi/linux/virtio_rpmsg.h line 14

Quoted 7 lines
include/uapi/linux/virtio_rpmsg.h new file +7 -0
+#define _UAPI_LINUX_VIRTIO_RPMSG_H
+
+#include <linux/types.h>
+
+/* The feature bitmap for virtio rpmsg */
+#define VIRTIO_RPMSG_F_NS    0 /* RP supports name service notifications */
+#define VIRTIO_RPMSG_F_BUFSZ 2 /* RP get buffer size from config space */

Would be useful to document it in rpmsg.txt

Good point, but it is better to put them into this document:
https://docs.oasis-open.org/virtio/virtio/v1.1/csprd01/virtio-v1.1-csprd01.html
like other vritio driver spec.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
include/uapi/linux/virtio_rpmsg.h new file +4 -0
+
+struct virtio_rpmsg_config {
+     /* The tx/rx individual buffer size(if VIRTIO_RPMSG_F_BUFSZ) */
+     __u32 txbuf_size;
Quoted 9 lines
Quoted 5 lines
Quoted 5 lines

include/uapi/linux/virtio_rpmsg.h line 23

Quoted 5 lines
include/uapi/linux/virtio_rpmsg.h new file +5 -0
+     __u32 rxbuf_size;
+     __u32 reserved[14]; /* Reserve for the future use */
+     /* Put the customize config here */
+} __attribute__((packed));
+

Wouldn't it be better to add an identifier and a version fields at the beginning of the structure? Idea would be to simplify a future extension In this case is VIRTIO_RPMSG_F_BUFSZ still useful?

Yes, I consider this option before, but after review all include/uapi/virtio_*.h, I found that virito driver prefer feature bits than version number to handle the compability issue. For example, if we need introduce more options in the furture, we need:
1.Add new feature bit to notice the option exist
2.Allocate the field from reserved space

Quoted 5 lines
Quoted 2 lines
Quoted 2 lines

include/uapi/linux/virtio_rpmsg.h line 24

Quoted 2 lines
include/uapi/linux/virtio_rpmsg.h new file +1 -0
+#endif /* _UAPI_LINUX_VIRTIO_RPMSG_H */

--
Thanks
Arnaud

#11 lore raw

Re: [PATCH 3/3] rpmsg: virtio_rpmsg_bus: get buffer size from config space

Quoted 154 lines · xiang xiao
Quoted 129 lines · Arnaud Pouliquen

Hello Xiang,

Quoted 16 lines · xiang xiao
Quoted 8 lines · Arnaud Pouliquen

Hello Xiang,

Similar mechanism has been proposed by Loic 2 years ago (link to the series here https://lkml.org/lkml/2017/3/28/349).

Did you see them? Regarding history, patches seem just on hold...

Just saw this patchset, so it's common problem hit by many vendor, rpmsg framework need to address it.:)

Quoted 2 lines

Main differences (except interesting RX/TX size split) seems that you
- don't use the virtio_config_ops->get

Quoted 7 lines

virtio_cread call virtio_config_ops->get internally, the ideal is same for both patch, just the implementation detail is different.

Quoted 1 line

- define a new feature VIRTIO_RPMSG_F_NS.

I add this flag to keep the compatibility with old remote peer, and also follow the common virito driver practice.

I discussed with Loic, he is ok to go further with your patch and abandon his one. Please find some remarks below in-line

Quoted 100 lines
Quoted 100 lines

Regards
Arnaud

Quoted 93 lines · Xiang Xiao

512 bytes isn't always suitable for all case, let firmware maker decide the best value from resource table. enable by VIRTIO_RPMSG_F_BUFSZ feature bit.

---
 drivers/rpmsg/virtio_rpmsg_bus.c  | 50 +++++++++++++++++++++++++--------------
 include/uapi/linux/virtio_rpmsg.h | 24 +++++++++++++++++++
 2 files changed, 56 insertions(+), 18 deletions(-)
 create mode 100644 include/uapi/linux/virtio_rpmsg.h
drivers/rpmsg/virtio_rpmsg_bus.c +9 -9
index 59c4554..049dd97 100644
@@ -16,6 +16,7 @@
 #include <linux/virtio.h>
 #include <linux/virtio_ids.h>
 #include <linux/virtio_config.h>
+#include <linux/virtio_rpmsg.h>
 #include <linux/scatterlist.h>
 #include <linux/dma-mapping.h>
 #include <linux/slab.h>
@@ -38,7 +39,8 @@
  * @sbufs:   kernel address of tx buffers
  * @num_rbufs:       total number of buffers for rx
  * @num_sbufs:       total number of buffers for tx
- * @buf_size:        size of one rx or tx buffer
+ * @rbuf_size:       size of one rx buffer
+ * @sbuf_size:       size of one tx buffer
  * @last_sbuf:       index of last tx buffer used
  * @rbufs_dma:       dma base addr of rx buffers
  * @sbufs_dma:       dma base addr of tx buffers
@@ -61,7 +63,8 @@ struct virtproc_info {
      void *rbufs, *sbufs;
      unsigned int num_rbufs;
      unsigned int num_sbufs;
-     unsigned int buf_size;
+     unsigned int rbuf_size;
+     unsigned int sbuf_size;
      int last_sbuf;
      dma_addr_t rbufs_dma;
      dma_addr_t sbufs_dma;
@@ -73,9 +76,6 @@ struct virtproc_info {
      struct rpmsg_endpoint *ns_ept;
 };
 
-/* The feature bitmap for virtio rpmsg */
-#define VIRTIO_RPMSG_F_NS    0 /* RP supports name service notifications */
-
 /**
  * struct rpmsg_hdr - common header for all rpmsg messages
  * @src: source address
@@ -452,7 +452,7 @@ static void *get_a_tx_buf(struct virtproc_info *vrp)
 
      /* either pick the next unused tx buffer */
      if (vrp->last_sbuf < vrp->num_sbufs)
-             ret = vrp->sbufs + vrp->buf_size * vrp->last_sbuf++;
+             ret = vrp->sbufs + vrp->sbuf_size * vrp->last_sbuf++;
      /* or recycle a used one */
      else
              ret = virtqueue_get_buf(vrp->svq, &len);
@@ -578,7 +578,7 @@ static int rpmsg_send_offchannel_raw(struct rpmsg_device *rpdev,
       * messaging), or to improve the buffer allocator, to support
       * variable-length buffer sizes.
       */
-     if (len > vrp->buf_size - sizeof(struct rpmsg_hdr)) {
+     if (len > vrp->sbuf_size - sizeof(struct rpmsg_hdr)) {
              dev_err(dev, "message is too big (%d)\n", len);
              return -EMSGSIZE;
      }
@@ -718,7 +718,7 @@ static int rpmsg_recv_single(struct virtproc_info *vrp, struct device *dev,
       * We currently use fixed-sized buffers, so trivially sanitize
       * the reported payload length.
       */
-     if (len > vrp->buf_size ||
+     if (len > vrp->rbuf_size ||
          msg->len > (len - sizeof(struct rpmsg_hdr))) {
              dev_warn(dev, "inbound msg too big: (%d, %d)\n", len, msg->len);
              return -EINVAL;
@@ -751,7 +751,7 @@ static int rpmsg_recv_single(struct virtproc_info *vrp, struct device *dev,
              dev_warn(dev, "msg received with no recipient\n");
 
      /* publish the real size of the buffer */
-     rpmsg_sg_init(&sg, msg, vrp->buf_size);
+     rpmsg_sg_init(&sg, msg, vrp->rbuf_size);
 
      /* add the buffer back to the remote processor's virtqueue */
      err = virtqueue_add_inbuf(vrp->rvq, &sg, 1, msg, GFP_KERNEL);
@@ -907,11 +907,24 @@ static int rpmsg_probe(struct virtio_device *vdev)
      else
              vrp->num_sbufs = MAX_RPMSG_NUM_BUFS;
 
Quoted 15 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
drivers/rpmsg/virtio_rpmsg_bus.c +4 -1
-     vrp->buf_size = MAX_RPMSG_BUF_SIZE;
+     /* try to get buffer size from config space */
+     if (virtio_has_feature(vdev, VIRTIO_RPMSG_F_BUFSZ)) {
+             /* note: virtio_rpmsg_config is defined from remote view */
+             virtio_cread(vdev, struct virtio_rpmsg_config,
Quoted 8 lines
Quoted 8 lines

drivers/rpmsg/virtio_rpmsg_bus.c static int rpmsg_probe(struct virtio_device *vdev) · line 921

Quoted 8 lines
drivers/rpmsg/virtio_rpmsg_bus.c +8 -0
+                          txbuf_size, &vrp->rbuf_size);
+             virtio_cread(vdev, struct virtio_rpmsg_config,
+                          rxbuf_size, &vrp->sbuf_size);
+     }
+
+     /* use the default if resource table doesn't provide one */
+     if (!vrp->rbuf_size)
+             vrp->rbuf_size = MAX_RPMSG_BUF_SIZE;

In this case constant should be renamed DEFAULT_RPMSG_BUF_SIZE as it is no more a max value

Yes, DEFAULT_RPMSG_BUF_SIZE is more reasonable now.

Quoted 4 lines
Quoted 2 lines
Quoted 2 lines

drivers/rpmsg/virtio_rpmsg_bus.c static int rpmsg_probe(struct virtio_device *vdev) · line 923

Quoted 2 lines
drivers/rpmsg/virtio_rpmsg_bus.c +2 -0
+     if (!vrp->sbuf_size)
+             vrp->sbuf_size = MAX_RPMSG_BUF_SIZE;

Here, if the config space exists you need to update it in consequence to ensure coherency with the remote processor config.

Quoted 9 lines

The update is already done in if (virtio_has_feature(vdev, VIRTIO_RPMSG_F_BUFSZ)), here just handle the zero value in config space which mean the remote side want to use the default value even VIRTIO_RPMSG_F_BUFSZ set.
For example:
1.remote side want to change one direction buffer size, but keep another direction as default
2.or remote side want to change other config options(define in the furture) not the buffer size

In code above i can see a virtio_cread of the config structure, but no writing of it...
I mentioned the configs space in the resource table itself. Without an update, you must ensure that both have the same default value... In addition, it makes sense that the master can update the buffer size according to some other constraints.

Quoted 126 lines
Quoted 65 lines
Quoted 65 lines
Quoted 65 lines
Quoted 65 lines
drivers/rpmsg/virtio_rpmsg_bus.c +9 -8
      /* allocate coherent memory for the buffers */
      vrp->rbufs = dma_alloc_coherent(vdev->dev.parent->parent,
-                                     vrp->num_rbufs * vrp->buf_size,
+                                     vrp->num_rbufs * vrp->rbuf_size,
                                      &vrp->rbufs_dma, GFP_KERNEL);
      if (!vrp->rbufs) {
              err = -ENOMEM;
@@ -922,7 +935,7 @@ static int rpmsg_probe(struct virtio_device *vdev)
              vrp->rbufs, &vrp->rbufs_dma);
 
      vrp->sbufs = dma_alloc_coherent(vdev->dev.parent->parent,
-                                     vrp->num_sbufs * vrp->buf_size,
+                                     vrp->num_sbufs * vrp->sbuf_size,
                                      &vrp->sbufs_dma, GFP_KERNEL);
      if (!vrp->sbufs) {
              err = -ENOMEM;
@@ -935,9 +948,9 @@ static int rpmsg_probe(struct virtio_device *vdev)
      /* set up the receive buffers */
      for (i = 0; i < vrp->num_rbufs; i++) {
              struct scatterlist sg;
-             void *cpu_addr = vrp->rbufs + i * vrp->buf_size;
+             void *cpu_addr = vrp->rbufs + i * vrp->rbuf_size;
 
-             rpmsg_sg_init(&sg, cpu_addr, vrp->buf_size);
+             rpmsg_sg_init(&sg, cpu_addr, vrp->rbuf_size);
 
              err = virtqueue_add_inbuf(vrp->rvq, &sg, 1, cpu_addr,
                                        GFP_KERNEL);
@@ -984,11 +997,11 @@ static int rpmsg_probe(struct virtio_device *vdev)
 
 free_sbufs:
      dma_free_coherent(vdev->dev.parent->parent,
-                       vrp->num_sbufs * vrp->buf_size,
+                       vrp->num_sbufs * vrp->sbuf_size,
                        vrp->sbufs, vrp->sbufs_dma);
 free_rbufs:
      dma_free_coherent(vdev->dev.parent->parent,
-                       vrp->num_rbufs * vrp->buf_size,
+                       vrp->num_rbufs * vrp->rbuf_size,
                        vrp->rbufs, vrp->rbufs_dma);
 vqs_del:
      vdev->config->del_vqs(vrp->vdev);
@@ -1023,10 +1036,10 @@ static void rpmsg_remove(struct virtio_device *vdev)
      vdev->config->del_vqs(vrp->vdev);
 
      dma_free_coherent(vdev->dev.parent->parent,
-                       vrp->num_sbufs * vrp->buf_size,
+                       vrp->num_sbufs * vrp->sbuf_size,
                        vrp->sbufs, vrp->sbufs_dma);
      dma_free_coherent(vdev->dev.parent->parent,
-                       vrp->num_rbufs * vrp->buf_size,
+                       vrp->num_rbufs * vrp->rbuf_size,
                        vrp->rbufs, vrp->rbufs_dma);
 
      kfree(vrp);
@@ -1039,6 +1052,7 @@ static struct virtio_device_id id_table[] = {
 
 static unsigned int features[] = {
      VIRTIO_RPMSG_F_NS,
+     VIRTIO_RPMSG_F_BUFSZ,
 };
 
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines

include/uapi/linux/virtio_rpmsg.h

Quoted 6 lines
drivers/rpmsg/virtio_rpmsg_bus.c +0 -0
 static struct virtio_driver virtio_ipc_driver = {
include/uapi/linux/virtio_rpmsg.h new file +0 -0
new file mode 100644
index 0000000..24fa0dd

Strange to define a user space API for kernel usage need. Could you elaborate?

I just follow the practice other virtio drivers(e.g. include/uapi/virtio_net.h) applied, but rpmsg driver don't need to talk with the host VM software like other virtio driver, yes this header file isn't really needed.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
include/uapi/linux/virtio_rpmsg.h new file +7 -0
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Copyright (C) Pinecone Inc. 2019
+ * Copyright (C) Xiang Xiao <[email protected]>
+ */
+
+#ifndef _UAPI_LINUX_VIRTIO_RPMSG_H
Quoted 8 lines
Quoted 7 lines
Quoted 7 lines

include/uapi/linux/virtio_rpmsg.h line 14

Quoted 7 lines
include/uapi/linux/virtio_rpmsg.h new file +7 -0
+#define _UAPI_LINUX_VIRTIO_RPMSG_H
+
+#include <linux/types.h>
+
+/* The feature bitmap for virtio rpmsg */
+#define VIRTIO_RPMSG_F_NS    0 /* RP supports name service notifications */
+#define VIRTIO_RPMSG_F_BUFSZ 2 /* RP get buffer size from config space */

Would be useful to document it in rpmsg.txt

Good point, but it is better to put them into this document:
https://docs.oasis-open.org/virtio/virtio/v1.1/csprd01/virtio-v1.1-csprd01.html
like other vritio driver spec.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
include/uapi/linux/virtio_rpmsg.h new file +4 -0
+
+struct virtio_rpmsg_config {
+     /* The tx/rx individual buffer size(if VIRTIO_RPMSG_F_BUFSZ) */
+     __u32 txbuf_size;
Quoted 9 lines
Quoted 5 lines
Quoted 5 lines

include/uapi/linux/virtio_rpmsg.h line 23

Quoted 5 lines
include/uapi/linux/virtio_rpmsg.h new file +5 -0
+     __u32 rxbuf_size;
+     __u32 reserved[14]; /* Reserve for the future use */
+     /* Put the customize config here */
+} __attribute__((packed));
+

Wouldn't it be better to add an identifier and a version fields at the beginning of the structure? Idea would be to simplify a future extension In this case is VIRTIO_RPMSG_F_BUFSZ still useful?

Yes, I consider this option before, but after review all include/uapi/virtio_*.h, I found that virito driver prefer feature bits than version number to handle the compability issue. For example, if we need introduce more options in the furture, we need:
1.Add new feature bit to notice the option exist
2.Allocate the field from reserved space

Quoted 5 lines
Quoted 2 lines
Quoted 2 lines

include/uapi/linux/virtio_rpmsg.h line 24

Quoted 2 lines
include/uapi/linux/virtio_rpmsg.h new file +1 -0
+#endif /* _UAPI_LINUX_VIRTIO_RPMSG_H */

--
Thanks
Arnaud

author
#12 lore raw

Re: [PATCH 3/3] rpmsg: virtio_rpmsg_bus: get buffer size from config space

Quoted 168 lines · Arnaud Pouliquen
Quoted 154 lines · xiang xiao
Quoted 129 lines · Arnaud Pouliquen

Hello Xiang,

Quoted 16 lines · xiang xiao
Quoted 8 lines · Arnaud Pouliquen

Hello Xiang,

Similar mechanism has been proposed by Loic 2 years ago (link to the series here https://lkml.org/lkml/2017/3/28/349).

Did you see them? Regarding history, patches seem just on hold...

Just saw this patchset, so it's common problem hit by many vendor, rpmsg framework need to address it.:)

Quoted 2 lines

Main differences (except interesting RX/TX size split) seems that you
- don't use the virtio_config_ops->get

Quoted 7 lines

virtio_cread call virtio_config_ops->get internally, the ideal is same for both patch, just the implementation detail is different.

Quoted 1 line

- define a new feature VIRTIO_RPMSG_F_NS.

I add this flag to keep the compatibility with old remote peer, and also follow the common virito driver practice.

I discussed with Loic, he is ok to go further with your patch and abandon his one. Please find some remarks below in-line

Quoted 100 lines
Quoted 100 lines

Regards
Arnaud

Quoted 93 lines · Xiang Xiao

512 bytes isn't always suitable for all case, let firmware maker decide the best value from resource table. enable by VIRTIO_RPMSG_F_BUFSZ feature bit.

---
 drivers/rpmsg/virtio_rpmsg_bus.c  | 50 +++++++++++++++++++++++++--------------
 include/uapi/linux/virtio_rpmsg.h | 24 +++++++++++++++++++
 2 files changed, 56 insertions(+), 18 deletions(-)
 create mode 100644 include/uapi/linux/virtio_rpmsg.h
drivers/rpmsg/virtio_rpmsg_bus.c +9 -9
index 59c4554..049dd97 100644
@@ -16,6 +16,7 @@
 #include <linux/virtio.h>
 #include <linux/virtio_ids.h>
 #include <linux/virtio_config.h>
+#include <linux/virtio_rpmsg.h>
 #include <linux/scatterlist.h>
 #include <linux/dma-mapping.h>
 #include <linux/slab.h>
@@ -38,7 +39,8 @@
  * @sbufs:   kernel address of tx buffers
  * @num_rbufs:       total number of buffers for rx
  * @num_sbufs:       total number of buffers for tx
- * @buf_size:        size of one rx or tx buffer
+ * @rbuf_size:       size of one rx buffer
+ * @sbuf_size:       size of one tx buffer
  * @last_sbuf:       index of last tx buffer used
  * @rbufs_dma:       dma base addr of rx buffers
  * @sbufs_dma:       dma base addr of tx buffers
@@ -61,7 +63,8 @@ struct virtproc_info {
      void *rbufs, *sbufs;
      unsigned int num_rbufs;
      unsigned int num_sbufs;
-     unsigned int buf_size;
+     unsigned int rbuf_size;
+     unsigned int sbuf_size;
      int last_sbuf;
      dma_addr_t rbufs_dma;
      dma_addr_t sbufs_dma;
@@ -73,9 +76,6 @@ struct virtproc_info {
      struct rpmsg_endpoint *ns_ept;
 };
 
-/* The feature bitmap for virtio rpmsg */
-#define VIRTIO_RPMSG_F_NS    0 /* RP supports name service notifications */
-
 /**
  * struct rpmsg_hdr - common header for all rpmsg messages
  * @src: source address
@@ -452,7 +452,7 @@ static void *get_a_tx_buf(struct virtproc_info *vrp)
 
      /* either pick the next unused tx buffer */
      if (vrp->last_sbuf < vrp->num_sbufs)
-             ret = vrp->sbufs + vrp->buf_size * vrp->last_sbuf++;
+             ret = vrp->sbufs + vrp->sbuf_size * vrp->last_sbuf++;
      /* or recycle a used one */
      else
              ret = virtqueue_get_buf(vrp->svq, &len);
@@ -578,7 +578,7 @@ static int rpmsg_send_offchannel_raw(struct rpmsg_device *rpdev,
       * messaging), or to improve the buffer allocator, to support
       * variable-length buffer sizes.
       */
-     if (len > vrp->buf_size - sizeof(struct rpmsg_hdr)) {
+     if (len > vrp->sbuf_size - sizeof(struct rpmsg_hdr)) {
              dev_err(dev, "message is too big (%d)\n", len);
              return -EMSGSIZE;
      }
@@ -718,7 +718,7 @@ static int rpmsg_recv_single(struct virtproc_info *vrp, struct device *dev,
       * We currently use fixed-sized buffers, so trivially sanitize
       * the reported payload length.
       */
-     if (len > vrp->buf_size ||
+     if (len > vrp->rbuf_size ||
          msg->len > (len - sizeof(struct rpmsg_hdr))) {
              dev_warn(dev, "inbound msg too big: (%d, %d)\n", len, msg->len);
              return -EINVAL;
@@ -751,7 +751,7 @@ static int rpmsg_recv_single(struct virtproc_info *vrp, struct device *dev,
              dev_warn(dev, "msg received with no recipient\n");
 
      /* publish the real size of the buffer */
-     rpmsg_sg_init(&sg, msg, vrp->buf_size);
+     rpmsg_sg_init(&sg, msg, vrp->rbuf_size);
 
      /* add the buffer back to the remote processor's virtqueue */
      err = virtqueue_add_inbuf(vrp->rvq, &sg, 1, msg, GFP_KERNEL);
@@ -907,11 +907,24 @@ static int rpmsg_probe(struct virtio_device *vdev)
      else
              vrp->num_sbufs = MAX_RPMSG_NUM_BUFS;
 
Quoted 15 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
drivers/rpmsg/virtio_rpmsg_bus.c +4 -1
-     vrp->buf_size = MAX_RPMSG_BUF_SIZE;
+     /* try to get buffer size from config space */
+     if (virtio_has_feature(vdev, VIRTIO_RPMSG_F_BUFSZ)) {
+             /* note: virtio_rpmsg_config is defined from remote view */
+             virtio_cread(vdev, struct virtio_rpmsg_config,
Quoted 8 lines
Quoted 8 lines

drivers/rpmsg/virtio_rpmsg_bus.c static int rpmsg_probe(struct virtio_device *vdev) · line 921

Quoted 8 lines
drivers/rpmsg/virtio_rpmsg_bus.c +8 -0
+                          txbuf_size, &vrp->rbuf_size);
+             virtio_cread(vdev, struct virtio_rpmsg_config,
+                          rxbuf_size, &vrp->sbuf_size);
+     }
+
+     /* use the default if resource table doesn't provide one */
+     if (!vrp->rbuf_size)
+             vrp->rbuf_size = MAX_RPMSG_BUF_SIZE;

In this case constant should be renamed DEFAULT_RPMSG_BUF_SIZE as it is no more a max value

Yes, DEFAULT_RPMSG_BUF_SIZE is more reasonable now.

Quoted 4 lines
Quoted 2 lines
Quoted 2 lines

drivers/rpmsg/virtio_rpmsg_bus.c static int rpmsg_probe(struct virtio_device *vdev) · line 923

Quoted 2 lines
drivers/rpmsg/virtio_rpmsg_bus.c +2 -0
+     if (!vrp->sbuf_size)
+             vrp->sbuf_size = MAX_RPMSG_BUF_SIZE;

Here, if the config space exists you need to update it in consequence to ensure coherency with the remote processor config.

Quoted 9 lines

The update is already done in if (virtio_has_feature(vdev, VIRTIO_RPMSG_F_BUFSZ)), here just handle the zero value in config space which mean the remote side want to use the default value even VIRTIO_RPMSG_F_BUFSZ set.
For example:
1.remote side want to change one direction buffer size, but keep another direction as default
2.or remote side want to change other config options(define in the furture) not the buffer size

Quoted 6 lines

In code above i can see a virtio_cread of the config structure, but no writing of it...
I mentioned the configs space in the resource table itself. Without an update, you must ensure that both have the same default value... In addition, it makes sense that the master can update the buffer size according to some other constraints.

Get your point, thanks.

Quoted 127 lines
Quoted 127 lines
Quoted 66 lines
Quoted 66 lines
Quoted 66 lines
Quoted 66 lines
drivers/rpmsg/virtio_rpmsg_bus.c +9 -8
      /* allocate coherent memory for the buffers */
      vrp->rbufs = dma_alloc_coherent(vdev->dev.parent->parent,
-                                     vrp->num_rbufs * vrp->buf_size,
+                                     vrp->num_rbufs * vrp->rbuf_size,
                                      &vrp->rbufs_dma, GFP_KERNEL);
      if (!vrp->rbufs) {
              err = -ENOMEM;
@@ -922,7 +935,7 @@ static int rpmsg_probe(struct virtio_device *vdev)
              vrp->rbufs, &vrp->rbufs_dma);
 
      vrp->sbufs = dma_alloc_coherent(vdev->dev.parent->parent,
-                                     vrp->num_sbufs * vrp->buf_size,
+                                     vrp->num_sbufs * vrp->sbuf_size,
                                      &vrp->sbufs_dma, GFP_KERNEL);
      if (!vrp->sbufs) {
              err = -ENOMEM;
@@ -935,9 +948,9 @@ static int rpmsg_probe(struct virtio_device *vdev)
      /* set up the receive buffers */
      for (i = 0; i < vrp->num_rbufs; i++) {
              struct scatterlist sg;
-             void *cpu_addr = vrp->rbufs + i * vrp->buf_size;
+             void *cpu_addr = vrp->rbufs + i * vrp->rbuf_size;
 
-             rpmsg_sg_init(&sg, cpu_addr, vrp->buf_size);
+             rpmsg_sg_init(&sg, cpu_addr, vrp->rbuf_size);
 
              err = virtqueue_add_inbuf(vrp->rvq, &sg, 1, cpu_addr,
                                        GFP_KERNEL);
@@ -984,11 +997,11 @@ static int rpmsg_probe(struct virtio_device *vdev)
 
 free_sbufs:
      dma_free_coherent(vdev->dev.parent->parent,
-                       vrp->num_sbufs * vrp->buf_size,
+                       vrp->num_sbufs * vrp->sbuf_size,
                        vrp->sbufs, vrp->sbufs_dma);
 free_rbufs:
      dma_free_coherent(vdev->dev.parent->parent,
-                       vrp->num_rbufs * vrp->buf_size,
+                       vrp->num_rbufs * vrp->rbuf_size,
                        vrp->rbufs, vrp->rbufs_dma);
 vqs_del:
      vdev->config->del_vqs(vrp->vdev);
@@ -1023,10 +1036,10 @@ static void rpmsg_remove(struct virtio_device *vdev)
      vdev->config->del_vqs(vrp->vdev);
 
      dma_free_coherent(vdev->dev.parent->parent,
-                       vrp->num_sbufs * vrp->buf_size,
+                       vrp->num_sbufs * vrp->sbuf_size,
                        vrp->sbufs, vrp->sbufs_dma);
      dma_free_coherent(vdev->dev.parent->parent,
-                       vrp->num_rbufs * vrp->buf_size,
+                       vrp->num_rbufs * vrp->rbuf_size,
                        vrp->rbufs, vrp->rbufs_dma);
 
      kfree(vrp);
@@ -1039,6 +1052,7 @@ static struct virtio_device_id id_table[] = {
 
 static unsigned int features[] = {
      VIRTIO_RPMSG_F_NS,
+     VIRTIO_RPMSG_F_BUFSZ,
 };
 
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines

include/uapi/linux/virtio_rpmsg.h

Quoted 6 lines
drivers/rpmsg/virtio_rpmsg_bus.c +0 -0
 static struct virtio_driver virtio_ipc_driver = {
include/uapi/linux/virtio_rpmsg.h new file +0 -0
new file mode 100644
index 0000000..24fa0dd

Strange to define a user space API for kernel usage need. Could you elaborate?

I just follow the practice other virtio drivers(e.g. include/uapi/virtio_net.h) applied, but rpmsg driver don't need to talk with the host VM software like other virtio driver, yes this header file isn't really needed.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
include/uapi/linux/virtio_rpmsg.h new file +7 -0
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Copyright (C) Pinecone Inc. 2019
+ * Copyright (C) Xiang Xiao <[email protected]>
+ */
+
+#ifndef _UAPI_LINUX_VIRTIO_RPMSG_H
Quoted 8 lines
Quoted 7 lines
Quoted 7 lines

include/uapi/linux/virtio_rpmsg.h line 14

Quoted 7 lines
include/uapi/linux/virtio_rpmsg.h new file +7 -0
+#define _UAPI_LINUX_VIRTIO_RPMSG_H
+
+#include <linux/types.h>
+
+/* The feature bitmap for virtio rpmsg */
+#define VIRTIO_RPMSG_F_NS    0 /* RP supports name service notifications */
+#define VIRTIO_RPMSG_F_BUFSZ 2 /* RP get buffer size from config space */

Would be useful to document it in rpmsg.txt

Good point, but it is better to put them into this document:
https://docs.oasis-open.org/virtio/virtio/v1.1/csprd01/virtio-v1.1-csprd01.html
like other vritio driver spec.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
include/uapi/linux/virtio_rpmsg.h new file +4 -0
+
+struct virtio_rpmsg_config {
+     /* The tx/rx individual buffer size(if VIRTIO_RPMSG_F_BUFSZ) */
+     __u32 txbuf_size;
Quoted 9 lines
Quoted 5 lines
Quoted 5 lines

include/uapi/linux/virtio_rpmsg.h line 23

Quoted 5 lines
include/uapi/linux/virtio_rpmsg.h new file +5 -0
+     __u32 rxbuf_size;
+     __u32 reserved[14]; /* Reserve for the future use */
+     /* Put the customize config here */
+} __attribute__((packed));
+

Wouldn't it be better to add an identifier and a version fields at the beginning of the structure? Idea would be to simplify a future extension In this case is VIRTIO_RPMSG_F_BUFSZ still useful?

Yes, I consider this option before, but after review all include/uapi/virtio_*.h, I found that virito driver prefer feature bits than version number to handle the compability issue. For example, if we need introduce more options in the furture, we need:
1.Add new feature bit to notice the option exist
2.Allocate the field from reserved space

Quoted 5 lines
Quoted 2 lines
Quoted 2 lines

include/uapi/linux/virtio_rpmsg.h line 24

Quoted 2 lines
include/uapi/linux/virtio_rpmsg.h new file +1 -0
+#endif /* _UAPI_LINUX_VIRTIO_RPMSG_H */

--
Thanks
Arnaud

#13 lore raw
In reply to Xiang Xiao
Quoted 8 lines · Xiang Xiao

Hi,
This series enhance the buffer allocation by:
1.Support the different buffer number in rx/tx direction 2.Get the individual rx/tx buffer size from config space

Here is the related OpenAMP change:
https://github.com/OpenAMP/open-amp/pull/155

This looks pretty reasonable, but can you confirm that it's possible to use new firmware with an old Linux kernel when introducing this?

But ever since we discussed Loic's similar proposal earlier I've been questioning if the fixed buffer size isn't just an artifact of how we preallocate our buffers. The virtqueue seems to support arbitrary sizes of buffers and I see that the receive function in OpenAMP has been fixed to put back the buffer of the size that was received, rather than 512 bytes. So it seems like Linux would be able to send whatever size messages to OpenAMP it would handle it.

The question is if we could do the same in the other direction, perhaps by letting the OpenAMP side do it's message allocation when it's sending, rather than Linux pushing inbufs to be filled by the remote.

This would remove the problem of always having suboptimal buffer sizes.

Regards,
Bjorn

Quoted 13 lines

Xiang Xiao (3):
rpmsg: virtio_rpmsg_bus: allow the different vring size for send/recv
rpmsg: virtio_rpmsg_bus: allocate rx/tx buffer separately
rpmsg: virtio_rpmsg_bus: get buffer size from config space

 drivers/rpmsg/virtio_rpmsg_bus.c  | 127 +++++++++++++++++++++++---------------
 include/uapi/linux/virtio_rpmsg.h |  24 +++++++
 2 files changed, 100 insertions(+), 51 deletions(-)
 create mode 100644 include/uapi/linux/virtio_rpmsg.h
Signature · 1 line
2.7.4
#14 lore raw

Hi Bjorn,

Quoted 23 lines · Bjorn Andersson
Quoted 8 lines · Xiang Xiao

Hi,
This series enhance the buffer allocation by:
1.Support the different buffer number in rx/tx direction 2.Get the individual rx/tx buffer size from config space

Here is the related OpenAMP change:
https://github.com/OpenAMP/open-amp/pull/155

This looks pretty reasonable, but can you confirm that it's possible to use new firmware with an old Linux kernel when introducing this?

But ever since we discussed Loic's similar proposal earlier I've been questioning if the fixed buffer size isn't just an artifact of how we preallocate our buffers. The virtqueue seems to support arbitrary sizes of buffers and I see that the receive function in OpenAMP has been fixed to put back the buffer of the size that was received, rather than 512 bytes. So it seems like Linux would be able to send whatever size messages to OpenAMP it would handle it.

Quoted 3 lines

The question is if we could do the same in the other direction, perhaps by letting the OpenAMP side do it's message allocation when it's sending, rather than Linux pushing inbufs to be filled by the remote.

IMHO, both could be useful and could be not correlated. On-the fly buffer allocation seems more efficient but needs an allocator.This can be a generic allocator (with a va to da) for system where large amount of memories are accessible from both side.

Now what about system with small shared memory? In this case you have to deal with a limited/optimized memory chunk. To avoid memory fragmentation the allocator should have a pre-reserved buffers pool(so similar to existing implementation). This serie seems useful to optimize the size of the pre-reserved pool.

Quoted 19 lines

This would remove the problem of always having suboptimal buffer sizes.

Regards,
Bjorn

Quoted 13 lines

Xiang Xiao (3):
rpmsg: virtio_rpmsg_bus: allow the different vring size for send/recv
rpmsg: virtio_rpmsg_bus: allocate rx/tx buffer separately
rpmsg: virtio_rpmsg_bus: get buffer size from config space

 drivers/rpmsg/virtio_rpmsg_bus.c  | 127 +++++++++++++++++++++++---------------
 include/uapi/linux/virtio_rpmsg.h |  24 +++++++
 2 files changed, 100 insertions(+), 51 deletions(-)
 create mode 100644 include/uapi/linux/virtio_rpmsg.h
Signature · 1 line
2.7.4

--

Regards,
Arnaud

author
#15 lore raw
Quoted 36 lines · Arnaud Pouliquen

Hi Bjorn,

Quoted 23 lines · Bjorn Andersson
Quoted 8 lines · Xiang Xiao

Hi,
This series enhance the buffer allocation by:
1.Support the different buffer number in rx/tx direction 2.Get the individual rx/tx buffer size from config space

Here is the related OpenAMP change:
https://github.com/OpenAMP/open-amp/pull/155

This looks pretty reasonable, but can you confirm that it's possible to use new firmware with an old Linux kernel when introducing this?

But ever since we discussed Loic's similar proposal earlier I've been questioning if the fixed buffer size isn't just an artifact of how we preallocate our buffers. The virtqueue seems to support arbitrary sizes of buffers and I see that the receive function in OpenAMP has been fixed to put back the buffer of the size that was received, rather than 512 bytes. So it seems like Linux would be able to send whatever size messages to OpenAMP it would handle it.

Quoted 3 lines

The question is if we could do the same in the other direction, perhaps by letting the OpenAMP side do it's message allocation when it's sending, rather than Linux pushing inbufs to be filled by the remote.

IMHO, both could be useful and could be not correlated. On-the fly buffer allocation seems more efficient but needs an allocator.This can be a generic allocator (with a va to da) for system where large amount of memories are accessible from both side.

Quoted 6 lines

Now what about system with small shared memory? In this case you have to deal with a limited/optimized memory chunk. To avoid memory fragmentation the allocator should have a pre-reserved buffers pool(so similar to existing implementation). This serie seems useful to optimize the size of the pre-reserved pool.

Maybe we can reuse rxbuf_size/txbuf_size to trigger the different allocation policy:
1.If buf_size equal 0xfffffff, turn on the dynamic allocator
2.If buf_size equall 0, turn on the fixed allocator with the default buffer size 3.otherwise, turn on the fixed allocator with the configed buffer size So, both requirement could be satisfied without breaking the compatibility.

Quoted 24 lines
Quoted 19 lines

This would remove the problem of always having suboptimal buffer sizes.

Regards,
Bjorn

Quoted 13 lines

Xiang Xiao (3):
rpmsg: virtio_rpmsg_bus: allow the different vring size for send/recv
rpmsg: virtio_rpmsg_bus: allocate rx/tx buffer separately
rpmsg: virtio_rpmsg_bus: get buffer size from config space

 drivers/rpmsg/virtio_rpmsg_bus.c  | 127 +++++++++++++++++++++++---------------
 include/uapi/linux/virtio_rpmsg.h |  24 +++++++
 2 files changed, 100 insertions(+), 51 deletions(-)
 create mode 100644 include/uapi/linux/virtio_rpmsg.h

--
2.7.4

--

Regards,
Arnaud

#16 lore raw
In reply to Arnaud Pouliquen
Quoted 35 lines · Arnaud Pouliquen

Hi Bjorn,

Quoted 23 lines · Bjorn Andersson
Quoted 8 lines · Xiang Xiao

Hi,
This series enhance the buffer allocation by:
1.Support the different buffer number in rx/tx direction 2.Get the individual rx/tx buffer size from config space

Here is the related OpenAMP change:
https://github.com/OpenAMP/open-amp/pull/155

This looks pretty reasonable, but can you confirm that it's possible to use new firmware with an old Linux kernel when introducing this?

But ever since we discussed Loic's similar proposal earlier I've been questioning if the fixed buffer size isn't just an artifact of how we preallocate our buffers. The virtqueue seems to support arbitrary sizes of buffers and I see that the receive function in OpenAMP has been fixed to put back the buffer of the size that was received, rather than 512 bytes. So it seems like Linux would be able to send whatever size messages to OpenAMP it would handle it.

Quoted 3 lines

The question is if we could do the same in the other direction, perhaps by letting the OpenAMP side do it's message allocation when it's sending, rather than Linux pushing inbufs to be filled by the remote.

IMHO, both could be useful and could be not correlated. On-the fly buffer allocation seems more efficient but needs an allocator.This can be a generic allocator (with a va to da) for system where large amount of memories are accessible from both side.

Quoted 6 lines

Now what about system with small shared memory? In this case you have to deal with a limited/optimized memory chunk. To avoid memory fragmentation the allocator should have a pre-reserved buffers pool(so similar to existing implementation). This serie seems useful to optimize the size of the pre-reserved pool.

Having an implementation that uses small fixed size buffers seems very reasonable and I'm in favour of making the message size configurable.

I would however prefer to have this implemented in a way where the remote side should not be receiving messages in a way that's based on the remote side's allocation parameters.

I don't think this series prevents the introduction of such isolation, but it would render this code unnecessary.

Regards,
Bjorn