Enhance virtio rpmsg bus driver buffer allocation
Subsystems (5)
cover Enhance virtio rpmsg bus driver buffer allocation
Hi,
This series enhance the buffer allocation by:
1.Support the different buffer number in rx/tx direction 2.Get the individual rx/tx buffer size from config space
Here is the related OpenAMP change:
https://github.com/OpenAMP/open-amp/pull/155
Xiang Xiao (3):
rpmsg: virtio_rpmsg_bus: allow the different vring size for send/recv
rpmsg: virtio_rpmsg_bus: allocate rx/tx buffer separately
rpmsg: virtio_rpmsg_bus: get buffer size from config space
drivers/rpmsg/virtio_rpmsg_bus.c | 127 +++++++++++++++++++++++--------------- include/uapi/linux/virtio_rpmsg.h | 24 +++++++ 2 files changed, 100 insertions(+), 51 deletions(-) create mode 100644 include/uapi/linux/virtio_rpmsg.h
Signature · 1 line
2.7.4
PATCH 1/3 rpmsg: virtio_rpmsg_bus: allow the different vring size for send/recv
it's useful if the communication throughput is different from each side
--- drivers/rpmsg/virtio_rpmsg_bus.c | 47 ++++++++++++++++++++-------------------- 1 file changed, 24 insertions(+), 23 deletions(-)
drivers/rpmsg/virtio_rpmsg_bus.c +24 -23
@@ -36,8 +36,9 @@* @svq: tx virtqueue * @rbufs: kernel address of rx buffers * @sbufs: kernel address of tx buffers -* @num_bufs: total number of buffers for rx and tx -* @buf_size: size of one rx or tx buffer +* @num_rbufs: total number of buffers for rx +* @num_sbufs: total number of buffers for tx +* @buf_size: size of one rx or tx buffer * @last_sbuf: index of last tx buffer used * @bufs_dma: dma base addr of the buffers * @tx_lock: protects svq, sbufs and sleepers, to allow concurrent senders.
@@ -57,7 +58,8 @@ struct virtproc_info {struct virtio_device *vdev;struct virtqueue *rvq, *svq;void *rbufs, *sbufs; -unsigned int num_bufs; +unsigned int num_rbufs; +unsigned int num_sbufs;unsigned int buf_size;int last_sbuf;dma_addr_t bufs_dma;
@@ -136,7 +138,7 @@ struct virtio_rpmsg_channel {/* * We're allocating buffers of 512 bytes each for communications. The * number of buffers will be computed from the number of buffers supported -* by the vring, upto a maximum of 512 buffers (256 in each direction). +* by the vring, up to a maximum of 256 in each direction. * * Each buffer will have 16 bytes for the msg header and 496 bytes for * the payload.
@@ -151,7 +153,7 @@ struct virtio_rpmsg_channel {* can change this without changing anything in the firmware of the remote * processor. */ -#define MAX_RPMSG_NUM_BUFS (512) +#define MAX_RPMSG_NUM_BUFS (256)#define MAX_RPMSG_BUF_SIZE (512)/*
@@ -446,11 +448,8 @@ static void *get_a_tx_buf(struct virtproc_info *vrp)/* support multiple concurrent senders */ mutex_lock (&vrp->tx_lock); -/* -* either pick the next unused tx buffer -* (half of our buffers are used for sending messages) -*/ -if (vrp->last_sbuf < vrp->num_bufs /2 ) +/* either pick the next unused tx buffer */ +if (vrp->last_sbuf < vrp->num_sbufs) ret = vrp->sbufs + vrp->buf_size * vrp->last_sbuf++;/* or recycle a used one */ else
@@ -897,19 +896,20 @@ static int rpmsg_probe(struct virtio_device *vdev) vrp->rvq = vqs[0 ]; vrp->svq = vqs[1 ]; -/* we expect symmetric tx/rx vrings */ -WARN_ON (virtqueue_get_vring_size (vrp->rvq) != -virtqueue_get_vring_size (vrp->svq)); -/* we need less buffers if vrings are small */ -if (virtqueue_get_vring_size (vrp->rvq) <MAX_RPMSG_NUM_BUFS /2 ) - vrp->num_bufs =virtqueue_get_vring_size (vrp->rvq) *2 ; +if (virtqueue_get_vring_size (vrp->rvq) <MAX_RPMSG_NUM_BUFS ) + vrp->num_rbufs =virtqueue_get_vring_size (vrp->rvq); +else + vrp->num_rbufs =MAX_RPMSG_NUM_BUFS ; + +if (virtqueue_get_vring_size (vrp->svq) <MAX_RPMSG_NUM_BUFS ) + vrp->num_sbufs =virtqueue_get_vring_size (vrp->svq);else - vrp->num_bufs =MAX_RPMSG_NUM_BUFS ; + vrp->num_sbufs =MAX_RPMSG_NUM_BUFS ; vrp->buf_size =MAX_RPMSG_BUF_SIZE ; - total_buf_space = vrp->num_bufs * vrp->buf_size; + total_buf_space = (vrp->num_rbufs + vrp->num_sbufs) * vrp->buf_size;/* allocate coherent memory for the buffers */ bufs_va = dma_alloc_coherent(vdev->dev.parent->parent,
@@ -923,14 +923,14 @@ static int rpmsg_probe(struct virtio_device *vdev)dev_dbg (&vdev->dev,"buffers: va %p, dma %pad\n" , bufs_va, &vrp->bufs_dma); -/* half of the buffers is dedicated for RX */ +/* first part of the buffers is dedicated for RX */ vrp->rbufs = bufs_va; -/* and half is dedicated for TX */ - vrp->sbufs = bufs_va + total_buf_space /2 ; +/* and second part is dedicated for TX */ + vrp->sbufs = bufs_va + vrp->num_rbufs * vrp->buf_size;/* set up the receive buffers */ -for (i =0 ; i < vrp->num_bufs /2 ; i++) { +for (i =0 ; i < vrp->num_rbufs; i++) {struct scatterlist sg;void *cpu_addr = vrp->rbufs + i * vrp->buf_size;
@@ -999,7 +999,8 @@ static int rpmsg_remove_device(struct device *dev, void *data)static void rpmsg_remove (struct virtio_device *vdev) {struct virtproc_info *vrp = vdev->priv; -size_t total_buf_space = vrp->num_bufs * vrp->buf_size; +unsigned int num_bufs = vrp->num_rbufs + vrp->num_sbufs; +size_t total_buf_space = num_bufs * vrp->buf_size;int ret; vdev->config->reset (vdev);
Signature · 1 line
2.7.4
Re: [PATCH 1/3] rpmsg: virtio_rpmsg_bus: allow the different vring size for send/recv
Hello Xiang,
I tested your patch on stm32 platform, no issue.
No remark on you code.
Thanks,
Arnaud
Quoted 124 lines · Xiang Xiao
it's useful if the communication throughput is different from each side
Signed-off-by Xiang Xiao <[email protected]>--- drivers/rpmsg/virtio_rpmsg_bus.c | 47 ++++++++++++++++++++-------------------- 1 file changed, 24 insertions(+), 23 deletions(-)drivers/rpmsg/virtio_rpmsg_bus.c +24
-23@@ -36,8 +36,9 @@* @svq: tx virtqueue * @rbufs: kernel address of rx buffers * @sbufs: kernel address of tx buffers -* @num_bufs: total number of buffers for rx and tx -* @buf_size: size of one rx or tx buffer +* @num_rbufs: total number of buffers for rx +* @num_sbufs: total number of buffers for tx +* @buf_size: size of one rx or tx buffer * @last_sbuf: index of last tx buffer used * @bufs_dma: dma base addr of the buffers * @tx_lock: protects svq, sbufs and sleepers, to allow concurrent senders. @@ -57,7 +58,8 @@ struct virtproc_info {struct virtio_device *vdev;struct virtqueue *rvq, *svq;void *rbufs, *sbufs; -unsigned int num_bufs; +unsigned int num_rbufs; +unsigned int num_sbufs;unsigned int buf_size;int last_sbuf;dma_addr_t bufs_dma;@@ -136,7 +138,7 @@ struct virtio_rpmsg_channel {/* * We're allocating buffers of 512 bytes each for communications. The * number of buffers will be computed from the number of buffers supported -* by the vring, upto a maximum of 512 buffers (256 in each direction). +* by the vring, up to a maximum of 256 in each direction. * * Each buffer will have 16 bytes for the msg header and 496 bytes for * the payload. @@ -151,7 +153,7 @@ struct virtio_rpmsg_channel {* can change this without changing anything in the firmware of the remote * processor. */ -#define MAX_RPMSG_NUM_BUFS (512) +#define MAX_RPMSG_NUM_BUFS (256)#define MAX_RPMSG_BUF_SIZE (512)/* @@ -446,11 +448,8 @@ static void *get_a_tx_buf(struct virtproc_info *vrp)/* support multiple concurrent senders */ mutex_lock (&vrp->tx_lock); -/* -* either pick the next unused tx buffer -* (half of our buffers are used for sending messages) -*/ -if (vrp->last_sbuf < vrp->num_bufs /2 ) +/* either pick the next unused tx buffer */ +if (vrp->last_sbuf < vrp->num_sbufs) ret = vrp->sbufs + vrp->buf_size * vrp->last_sbuf++;/* or recycle a used one */ else @@ -897,19 +896,20 @@ static int rpmsg_probe(struct virtio_device *vdev) vrp->rvq = vqs[0 ]; vrp->svq = vqs[1 ]; -/* we expect symmetric tx/rx vrings */ -WARN_ON (virtqueue_get_vring_size (vrp->rvq) != -virtqueue_get_vring_size (vrp->svq)); -/* we need less buffers if vrings are small */ -if (virtqueue_get_vring_size (vrp->rvq) <MAX_RPMSG_NUM_BUFS /2 ) - vrp->num_bufs =virtqueue_get_vring_size (vrp->rvq) *2 ; +if (virtqueue_get_vring_size (vrp->rvq) <MAX_RPMSG_NUM_BUFS ) + vrp->num_rbufs =virtqueue_get_vring_size (vrp->rvq); +else + vrp->num_rbufs =MAX_RPMSG_NUM_BUFS ; + +if (virtqueue_get_vring_size (vrp->svq) <MAX_RPMSG_NUM_BUFS ) + vrp->num_sbufs =virtqueue_get_vring_size (vrp->svq);else - vrp->num_bufs =MAX_RPMSG_NUM_BUFS ; + vrp->num_sbufs =MAX_RPMSG_NUM_BUFS ; vrp->buf_size =MAX_RPMSG_BUF_SIZE ; - total_buf_space = vrp->num_bufs * vrp->buf_size; + total_buf_space = (vrp->num_rbufs + vrp->num_sbufs) * vrp->buf_size;/* allocate coherent memory for the buffers */ bufs_va = dma_alloc_coherent(vdev->dev.parent->parent,@@ -923,14 +923,14 @@ static int rpmsg_probe(struct virtio_device *vdev)dev_dbg (&vdev->dev,"buffers: va %p, dma %pad\n" , bufs_va, &vrp->bufs_dma); -/* half of the buffers is dedicated for RX */ +/* first part of the buffers is dedicated for RX */ vrp->rbufs = bufs_va; -/* and half is dedicated for TX */ - vrp->sbufs = bufs_va + total_buf_space /2 ; +/* and second part is dedicated for TX */ + vrp->sbufs = bufs_va + vrp->num_rbufs * vrp->buf_size;/* set up the receive buffers */ -for (i =0 ; i < vrp->num_bufs /2 ; i++) { +for (i =0 ; i < vrp->num_rbufs; i++) {struct scatterlist sg;void *cpu_addr = vrp->rbufs + i * vrp->buf_size;@@ -999,7 +999,8 @@ static int rpmsg_remove_device(struct device *dev, void *data)static void rpmsg_remove (struct virtio_device *vdev) {struct virtproc_info *vrp = vdev->priv; -size_t total_buf_space = vrp->num_bufs * vrp->buf_size; +unsigned int num_bufs = vrp->num_rbufs + vrp->num_sbufs; +size_t total_buf_space = num_bufs * vrp->buf_size;int ret; vdev->config->reset (vdev);
PATCH 2/3 rpmsg: virtio_rpmsg_bus: allocate rx/tx buffer separately
In reply to Xiang Xiaomany dma allocator align the returned address with buffer size, so two small allocation could reduce the alignment requirement and save the the memory space wasted by the potential alignment.
--- drivers/rpmsg/virtio_rpmsg_bus.c | 58 +++++++++++++++++++++++----------------- 1 file changed, 34 insertions(+), 24 deletions(-)
drivers/rpmsg/virtio_rpmsg_bus.c +34 -24
@@ -40,7 +40,8 @@* @num_sbufs: total number of buffers for tx * @buf_size: size of one rx or tx buffer * @last_sbuf: index of last tx buffer used -* @bufs_dma: dma base addr of the buffers +* @rbufs_dma: dma base addr of rx buffers +* @sbufs_dma: dma base addr of tx buffers * @tx_lock: protects svq, sbufs and sleepers, to allow concurrent senders. * sending a message might require waking up a dozing remote * processor, which involves sleeping, hence the mutex.
@@ -62,7 +63,8 @@ struct virtproc_info {unsigned int num_sbufs;unsigned int buf_size;int last_sbuf; -dma_addr_t bufs_dma; +dma_addr_t rbufs_dma; +dma_addr_t sbufs_dma;struct mutex tx_lock;struct idr endpoints;struct mutex endpoints_lock;
@@ -872,9 +874,7 @@ static int rpmsg_probe(struct virtio_device *vdev)static const char *const names[] = {"input" ,"output" };struct virtqueue *vqs[2 ];struct virtproc_info *vrp; -void *bufs_va;int err =0 , i; -size_t total_buf_space;bool notify; vrp =kzalloc (sizeof (*vrp),GFP_KERNEL );
@@ -909,25 +909,28 @@ static int rpmsg_probe(struct virtio_device *vdev) vrp->buf_size =MAX_RPMSG_BUF_SIZE ; - total_buf_space = (vrp->num_rbufs + vrp->num_sbufs) * vrp->buf_size; -/* allocate coherent memory for the buffers */ - bufs_va =dma_alloc_coherent (vdev->dev.parent->parent, - total_buf_space, &vrp->bufs_dma, -GFP_KERNEL ); -if (!bufs_va) { + vrp->rbufs =dma_alloc_coherent (vdev->dev.parent->parent, + vrp->num_rbufs * vrp->buf_size, + &vrp->rbufs_dma,GFP_KERNEL ); +if (!vrp->rbufs) { err = -ENOMEM ; goto vqs_del; } -dev_dbg (&vdev->dev,"buffers: va %p, dma %pad\n" , - bufs_va, &vrp->bufs_dma); +dev_dbg (&vdev->dev,"rx buffers: va %p, dma 0x%pad\n" , + vrp->rbufs, &vrp->rbufs_dma); -/* first part of the buffers is dedicated for RX */ - vrp->rbufs = bufs_va; + vrp->sbufs =dma_alloc_coherent (vdev->dev.parent->parent, + vrp->num_sbufs * vrp->buf_size, + &vrp->sbufs_dma,GFP_KERNEL ); +if (!vrp->sbufs) { + err = -ENOMEM ; + goto free_rbufs; + } -/* and second part is dedicated for TX */ - vrp->sbufs = bufs_va + vrp->num_rbufs * vrp->buf_size; +dev_dbg (&vdev->dev,"tx buffers: va %p, dma 0x%pad\n" , + vrp->sbufs, &vrp->sbufs_dma);/* set up the receive buffers */ for (i =0 ; i < vrp->num_rbufs; i++) {
@@ -954,7 +957,7 @@ static int rpmsg_probe(struct virtio_device *vdev)if (!vrp->ns_ept) {dev_err (&vdev->dev,"failed to create the ns ept\n" ); err = -ENOMEM ; - goto free_coherent; + goto free_sbufs; } }
@@ -979,9 +982,14 @@ static int rpmsg_probe(struct virtio_device *vdev)return 0 ; -free_coherent: -dma_free_coherent (vdev->dev.parent->parent, total_buf_space, - bufs_va, vrp->bufs_dma); +free_sbufs: +dma_free_coherent (vdev->dev.parent->parent, + vrp->num_sbufs * vrp->buf_size, + vrp->sbufs, vrp->sbufs_dma); +free_rbufs: +dma_free_coherent (vdev->dev.parent->parent, + vrp->num_rbufs * vrp->buf_size, + vrp->rbufs, vrp->rbufs_dma); vqs_del: vdev->config->del_vqs (vrp->vdev); free_vrp:
@@ -999,8 +1007,6 @@ static int rpmsg_remove_device(struct device *dev, void *data)static void rpmsg_remove (struct virtio_device *vdev) {struct virtproc_info *vrp = vdev->priv; -unsigned int num_bufs = vrp->num_rbufs + vrp->num_sbufs; -size_t total_buf_space = num_bufs * vrp->buf_size;int ret; vdev->config->reset (vdev);
@@ -1016,8 +1022,12 @@ static void rpmsg_remove(struct virtio_device *vdev) vdev->config->del_vqs (vrp->vdev); -dma_free_coherent (vdev->dev.parent->parent, total_buf_space, - vrp->rbufs, vrp->bufs_dma); +dma_free_coherent (vdev->dev.parent->parent, + vrp->num_sbufs * vrp->buf_size, + vrp->sbufs, vrp->sbufs_dma); +dma_free_coherent (vdev->dev.parent->parent, + vrp->num_rbufs * vrp->buf_size, + vrp->rbufs, vrp->rbufs_dma);kfree (vrp); }
Signature · 1 line
2.7.4
Re: [PATCH 2/3] rpmsg: virtio_rpmsg_bus: allocate rx/tx buffer separately
Hello Xiang,
This patch has the opposite effect on my platform as DMA allocation is aligned on 4k page.
For instance i declared:
- in RX 6 buffers (of 512 bytes)
- in TX 4 buffers ( of 512 bytes)
The result is (kernel trace)
[ 41.915896] virtio_rpmsg_bus virtio0: rx buffers: va ebb5f5ca, dma 0x0x10042000 [ 41.915922] virtio_rpmsg_bus virtio0: tx buffers: va a7865153, dma
0x0x10043000
The TX buffer memory is allocated on next 4k page...
Anyway separate the RX and TX allocation makes sense. This could also allow to allocate buffers in 2 different memories.
For time being, issue is that only one memory area can be attached to the virtio device for DMA allocation... and PA/DA translations are missing. This means that we probably need (in a first step) a new remoteproc API for memory allocation.
These memories should be declared and mmaped in rproc platform drivers (memory region) or in resource table (carveout).
This is partially done in the API for the platform driver (rproc_mem_entry_init) but not available for rproc clients.
Regards
Arnaud
Quoted 136 lines · Xiang Xiao
many dma allocator align the returned address with buffer size, so two small allocation could reduce the alignment requirement and save the the memory space wasted by the potential alignment.
Signed-off-by Xiang Xiao <[email protected]>--- drivers/rpmsg/virtio_rpmsg_bus.c | 58 +++++++++++++++++++++++----------------- 1 file changed, 34 insertions(+), 24 deletions(-)drivers/rpmsg/virtio_rpmsg_bus.c +34
-24@@ -40,7 +40,8 @@* @num_sbufs: total number of buffers for tx * @buf_size: size of one rx or tx buffer * @last_sbuf: index of last tx buffer used -* @bufs_dma: dma base addr of the buffers +* @rbufs_dma: dma base addr of rx buffers +* @sbufs_dma: dma base addr of tx buffers * @tx_lock: protects svq, sbufs and sleepers, to allow concurrent senders. * sending a message might require waking up a dozing remote * processor, which involves sleeping, hence the mutex. @@ -62,7 +63,8 @@ struct virtproc_info {unsigned int num_sbufs;unsigned int buf_size;int last_sbuf; -dma_addr_t bufs_dma; +dma_addr_t rbufs_dma; +dma_addr_t sbufs_dma;struct mutex tx_lock;struct idr endpoints;struct mutex endpoints_lock;@@ -872,9 +874,7 @@ static int rpmsg_probe(struct virtio_device *vdev)static const char *const names[] = {"input" ,"output" };struct virtqueue *vqs[2 ];struct virtproc_info *vrp; -void *bufs_va;int err =0 , i; -size_t total_buf_space;bool notify; vrp =kzalloc (sizeof (*vrp),GFP_KERNEL );@@ -909,25 +909,28 @@ static int rpmsg_probe(struct virtio_device *vdev) vrp->buf_size =MAX_RPMSG_BUF_SIZE ; - total_buf_space = (vrp->num_rbufs + vrp->num_sbufs) * vrp->buf_size; -/* allocate coherent memory for the buffers */ - bufs_va =dma_alloc_coherent (vdev->dev.parent->parent, - total_buf_space, &vrp->bufs_dma, -GFP_KERNEL ); -if (!bufs_va) { + vrp->rbufs =dma_alloc_coherent (vdev->dev.parent->parent, + vrp->num_rbufs * vrp->buf_size, + &vrp->rbufs_dma,GFP_KERNEL ); +if (!vrp->rbufs) { err = -ENOMEM ; goto vqs_del; } -dev_dbg (&vdev->dev,"buffers: va %p, dma %pad\n" , - bufs_va, &vrp->bufs_dma); +dev_dbg (&vdev->dev,"rx buffers: va %p, dma 0x%pad\n" , + vrp->rbufs, &vrp->rbufs_dma); -/* first part of the buffers is dedicated for RX */ - vrp->rbufs = bufs_va; + vrp->sbufs =dma_alloc_coherent (vdev->dev.parent->parent, + vrp->num_sbufs * vrp->buf_size, + &vrp->sbufs_dma,GFP_KERNEL ); +if (!vrp->sbufs) { + err = -ENOMEM ; + goto free_rbufs; + } -/* and second part is dedicated for TX */ - vrp->sbufs = bufs_va + vrp->num_rbufs * vrp->buf_size; +dev_dbg (&vdev->dev,"tx buffers: va %p, dma 0x%pad\n" , + vrp->sbufs, &vrp->sbufs_dma);/* set up the receive buffers */ for (i =0 ; i < vrp->num_rbufs; i++) {@@ -954,7 +957,7 @@ static int rpmsg_probe(struct virtio_device *vdev)if (!vrp->ns_ept) {dev_err (&vdev->dev,"failed to create the ns ept\n" ); err = -ENOMEM ; - goto free_coherent; + goto free_sbufs; } }@@ -979,9 +982,14 @@ static int rpmsg_probe(struct virtio_device *vdev)return 0 ; -free_coherent: -dma_free_coherent (vdev->dev.parent->parent, total_buf_space, - bufs_va, vrp->bufs_dma); +free_sbufs: +dma_free_coherent (vdev->dev.parent->parent, + vrp->num_sbufs * vrp->buf_size, + vrp->sbufs, vrp->sbufs_dma); +free_rbufs: +dma_free_coherent (vdev->dev.parent->parent, + vrp->num_rbufs * vrp->buf_size, + vrp->rbufs, vrp->rbufs_dma); vqs_del: vdev->config->del_vqs (vrp->vdev); free_vrp:@@ -999,8 +1007,6 @@ static int rpmsg_remove_device(struct device *dev, void *data)static void rpmsg_remove (struct virtio_device *vdev) {struct virtproc_info *vrp = vdev->priv; -unsigned int num_bufs = vrp->num_rbufs + vrp->num_sbufs; -size_t total_buf_space = num_bufs * vrp->buf_size;int ret; vdev->config->reset (vdev);@@ -1016,8 +1022,12 @@ static void rpmsg_remove(struct virtio_device *vdev) vdev->config->del_vqs (vrp->vdev); -dma_free_coherent (vdev->dev.parent->parent, total_buf_space, - vrp->rbufs, vrp->bufs_dma); +dma_free_coherent (vdev->dev.parent->parent, + vrp->num_sbufs * vrp->buf_size, + vrp->sbufs, vrp->sbufs_dma); +dma_free_coherent (vdev->dev.parent->parent, + vrp->num_rbufs * vrp->buf_size, + vrp->rbufs, vrp->rbufs_dma);kfree (vrp); }
Re: [PATCH 2/3] rpmsg: virtio_rpmsg_bus: allocate rx/tx buffer separately
Quoted 9 lines · Arnaud Pouliquen
Hello Xiang,
This patch has the opposite effect on my platform as DMA allocation is aligned on 4k page.
For instance i declared:
- in RX 6 buffers (of 512 bytes)
- in TX 4 buffers ( of 512 bytes)
Yes, dma_init_coherent_memory always allocate memory by 4KB unit, but this limitation is too waste memory for remoteproc/rpmsg. The attached patch fix this problem by adding a new device tree option to customize the unit size.
Quoted 157 lines
The result is (kernel trace)
[ 41.915896] virtio_rpmsg_bus virtio0: rx buffers: va ebb5f5ca, dma 0x0x10042000 [ 41.915922] virtio_rpmsg_bus virtio0: tx buffers: va a7865153, dma0x0x10043000
The TX buffer memory is allocated on next 4k page...
Anyway separate the RX and TX allocation makes sense. This could also allow to allocate buffers in 2 different memories.
For time being, issue is that only one memory area can be attached to the virtio device for DMA allocation... and PA/DA translations are missing. This means that we probably need (in a first step) a new remoteproc API for memory allocation.
These memories should be declared and mmaped in rproc platform drivers (memory region) or in resource table (carveout).
This is partially done in the API for the platform driver (rproc_mem_entry_init) but not available for rproc clients.Regards
ArnaudQuoted 133 lines · Xiang Xiao
many dma allocator align the returned address with buffer size, so two small allocation could reduce the alignment requirement and save the the memory space wasted by the potential alignment.
Signed-off-by Xiang Xiao <[email protected]>--- drivers/rpmsg/virtio_rpmsg_bus.c | 58 +++++++++++++++++++++++----------------- 1 file changed, 34 insertions(+), 24 deletions(-)drivers/rpmsg/virtio_rpmsg_bus.c +34
-24@@ -40,7 +40,8 @@* @num_sbufs: total number of buffers for tx * @buf_size: size of one rx or tx buffer * @last_sbuf: index of last tx buffer used -* @bufs_dma: dma base addr of the buffers +* @rbufs_dma: dma base addr of rx buffers +* @sbufs_dma: dma base addr of tx buffers * @tx_lock: protects svq, sbufs and sleepers, to allow concurrent senders. * sending a message might require waking up a dozing remote * processor, which involves sleeping, hence the mutex. @@ -62,7 +63,8 @@ struct virtproc_info {unsigned int num_sbufs;unsigned int buf_size;int last_sbuf; -dma_addr_t bufs_dma; +dma_addr_t rbufs_dma; +dma_addr_t sbufs_dma;struct mutex tx_lock;struct idr endpoints;struct mutex endpoints_lock;@@ -872,9 +874,7 @@ static int rpmsg_probe(struct virtio_device *vdev)static const char *const names[] = {"input" ,"output" };struct virtqueue *vqs[2 ];struct virtproc_info *vrp; -void *bufs_va;int err =0 , i; -size_t total_buf_space;bool notify; vrp =kzalloc (sizeof (*vrp),GFP_KERNEL );@@ -909,25 +909,28 @@ static int rpmsg_probe(struct virtio_device *vdev) vrp->buf_size =MAX_RPMSG_BUF_SIZE ; - total_buf_space = (vrp->num_rbufs + vrp->num_sbufs) * vrp->buf_size; -/* allocate coherent memory for the buffers */ - bufs_va =dma_alloc_coherent (vdev->dev.parent->parent, - total_buf_space, &vrp->bufs_dma, -GFP_KERNEL ); -if (!bufs_va) { + vrp->rbufs =dma_alloc_coherent (vdev->dev.parent->parent, + vrp->num_rbufs * vrp->buf_size, + &vrp->rbufs_dma,GFP_KERNEL ); +if (!vrp->rbufs) { err = -ENOMEM ; goto vqs_del; } -dev_dbg (&vdev->dev,"buffers: va %p, dma %pad\n" , - bufs_va, &vrp->bufs_dma); +dev_dbg (&vdev->dev,"rx buffers: va %p, dma 0x%pad\n" , + vrp->rbufs, &vrp->rbufs_dma); -/* first part of the buffers is dedicated for RX */ - vrp->rbufs = bufs_va; + vrp->sbufs =dma_alloc_coherent (vdev->dev.parent->parent, + vrp->num_sbufs * vrp->buf_size, + &vrp->sbufs_dma,GFP_KERNEL ); +if (!vrp->sbufs) { + err = -ENOMEM ; + goto free_rbufs; + } -/* and second part is dedicated for TX */ - vrp->sbufs = bufs_va + vrp->num_rbufs * vrp->buf_size; +dev_dbg (&vdev->dev,"tx buffers: va %p, dma 0x%pad\n" , + vrp->sbufs, &vrp->sbufs_dma);/* set up the receive buffers */ for (i =0 ; i < vrp->num_rbufs; i++) {@@ -954,7 +957,7 @@ static int rpmsg_probe(struct virtio_device *vdev)if (!vrp->ns_ept) {dev_err (&vdev->dev,"failed to create the ns ept\n" ); err = -ENOMEM ; - goto free_coherent; + goto free_sbufs; } }@@ -979,9 +982,14 @@ static int rpmsg_probe(struct virtio_device *vdev)return 0 ; -free_coherent: -dma_free_coherent (vdev->dev.parent->parent, total_buf_space, - bufs_va, vrp->bufs_dma); +free_sbufs: +dma_free_coherent (vdev->dev.parent->parent, + vrp->num_sbufs * vrp->buf_size, + vrp->sbufs, vrp->sbufs_dma); +free_rbufs: +dma_free_coherent (vdev->dev.parent->parent, + vrp->num_rbufs * vrp->buf_size, + vrp->rbufs, vrp->rbufs_dma); vqs_del: vdev->config->del_vqs (vrp->vdev); free_vrp:@@ -999,8 +1007,6 @@ static int rpmsg_remove_device(struct device *dev, void *data)static void rpmsg_remove (struct virtio_device *vdev) {struct virtproc_info *vrp = vdev->priv; -unsigned int num_bufs = vrp->num_rbufs + vrp->num_sbufs; -size_t total_buf_space = num_bufs * vrp->buf_size;int ret; vdev->config->reset (vdev);@@ -1016,8 +1022,12 @@ static void rpmsg_remove(struct virtio_device *vdev) vdev->config->del_vqs (vrp->vdev); -dma_free_coherent (vdev->dev.parent->parent, total_buf_space, - vrp->rbufs, vrp->bufs_dma); +dma_free_coherent (vdev->dev.parent->parent, + vrp->num_sbufs * vrp->buf_size, + vrp->sbufs, vrp->sbufs_dma); +dma_free_coherent (vdev->dev.parent->parent, + vrp->num_rbufs * vrp->buf_size, + vrp->rbufs, vrp->rbufs_dma);
Quoted 3 lines
drivers/rpmsg/virtio_rpmsg_bus.c static void rpmsg_remove(struct virtio_device *vdev) · line 1033
Quoted 3 lines
drivers/rpmsg/virtio_rpmsg_bus.c +0
-0kfree (vrp); }
From 62672e1df79c9ffccb062f5aa0dcb18e938ced0c Mon Sep 17 00:00:00 2001
From: Xiang Xiao <[email protected]>
Date: Thu, 22 Jun 2017 14:30:50 +0800
Subject: [PATCH] dma-coherent: support the alignment smaller than PAGE_SIZE
and read the alignment(default PAGE_SIZE) from device tree
--- .../bindings/reserved-memory/reserved-memory.txt | 2 + drivers/base/dma-coherent.c | 74 ++++++++++++++-------- include/linux/dma-mapping.h | 6 +- 3 files changed, 51 insertions(+), 31 deletions(-)
Documentation/devicetree/bindings/reserved-memory/reserved-memory.txt +2 -0
@@ -63,6 +63,8 @@ reusable (optional) - empty property able to reclaim it back. Typically that means that the operating system can use that region to store volatile or cached data that can be otherwise regenerated or migrated elsewhere. +align-shift (optional) - the allocation alignment (1 << align-shift) + - The default value is PAGE_SHIFT. Linux implementation note: - If a "linux,cma-default" property is present, then Linux will use the
drivers/base/dma-coherent.c +46 -28
179 lines not on this page · raw message
include/linux/dma-mapping.h +3 -3
@@ -152,13 +152,13 @@ static inline int is_device_dma_capable(struct device *dev)*/ int dma_alloc_from_coherent (struct device *dev,ssize_t size,dma_addr_t *dma_handle,void **ret); -int dma_release_from_coherent (struct device *dev,int order,void *vaddr); +int dma_release_from_coherent (struct device *dev,int size,void *vaddr);int dma_mmap_from_coherent (struct device *dev,struct vm_area_struct *vma,void *cpu_addr,size_t size,int *ret);#else #define dma_alloc_from_coherent (dev, size, handle, ret) (0) -#define dma_release_from_coherent (dev, order, vaddr) (0) +#define dma_release_from_coherent (dev, size, vaddr) (0)#define dma_mmap_from_coherent (dev, vma, vaddr, order, ret) (0)#endif /* CONFIG_HAVE_GENERIC_DMA_COHERENT */
@@ -478,7 +478,7 @@ static inline void dma_free_attrs(struct device *dev, size_t size,BUG_ON (!ops);WARN_ON (irqs_disabled ()); -if (dma_release_from_coherent (dev,get_order (size), cpu_addr)) +if (dma_release_from_coherent (dev, size, cpu_addr))return ;if (!ops->free || !cpu_addr)
Signature · 1 line
2.16.2
PATCH 3/3 rpmsg: virtio_rpmsg_bus: get buffer size from config space
In reply to Xiang Xiao512 bytes isn't always suitable for all case, let firmware maker decide the best value from resource table. enable by VIRTIO_RPMSG_F_BUFSZ feature bit.
--- drivers/rpmsg/virtio_rpmsg_bus.c | 50 +++++++++++++++++++++++++-------------- include/uapi/linux/virtio_rpmsg.h | 24 +++++++++++++++++++ 2 files changed, 56 insertions(+), 18 deletions(-) create mode 100644 include/uapi/linux/virtio_rpmsg.h
drivers/rpmsg/virtio_rpmsg_bus.c +32 -18
@@ -16,6 +16,7 @@#include <linux/virtio.h> #include <linux/virtio_ids.h> #include <linux/virtio_config.h> +#include <linux/virtio_rpmsg.h> #include <linux/scatterlist.h> #include <linux/dma-mapping.h> #include <linux/slab.h>
@@ -38,7 +39,8 @@* @sbufs: kernel address of tx buffers * @num_rbufs: total number of buffers for rx * @num_sbufs: total number of buffers for tx -* @buf_size: size of one rx or tx buffer +* @rbuf_size: size of one rx buffer +* @sbuf_size: size of one tx buffer * @last_sbuf: index of last tx buffer used * @rbufs_dma: dma base addr of rx buffers * @sbufs_dma: dma base addr of tx buffers
@@ -61,7 +63,8 @@ struct virtproc_info {void *rbufs, *sbufs;unsigned int num_rbufs;unsigned int num_sbufs; -unsigned int buf_size; +unsigned int rbuf_size; +unsigned int sbuf_size;int last_sbuf;dma_addr_t rbufs_dma;dma_addr_t sbufs_dma;
@@ -73,9 +76,6 @@ struct virtproc_info {struct rpmsg_endpoint *ns_ept; }; -/* The feature bitmap for virtio rpmsg */ -#define VIRTIO_RPMSG_F_NS 0/* RP supports name service notifications */ -/** * struct rpmsg_hdr - common header for all rpmsg messages * @src: source address
@@ -452,7 +452,7 @@ static void *get_a_tx_buf(struct virtproc_info *vrp)/* either pick the next unused tx buffer */ if (vrp->last_sbuf < vrp->num_sbufs) - ret = vrp->sbufs + vrp->buf_size * vrp->last_sbuf++; + ret = vrp->sbufs + vrp->sbuf_size * vrp->last_sbuf++;/* or recycle a used one */ else ret =virtqueue_get_buf (vrp->svq, &len);
@@ -578,7 +578,7 @@ static int rpmsg_send_offchannel_raw(struct rpmsg_device *rpdev,* messaging), or to improve the buffer allocator, to support * variable-length buffer sizes. */ -if (len > vrp->buf_size -sizeof (struct rpmsg_hdr )) { +if (len > vrp->sbuf_size -sizeof (struct rpmsg_hdr )) {dev_err (dev,"message is too big (%d)\n" , len);return -EMSGSIZE ; }
@@ -718,7 +718,7 @@ static int rpmsg_recv_single(struct virtproc_info *vrp, struct device *dev,* We currently use fixed-sized buffers, so trivially sanitize * the reported payload length. */ -if (len > vrp->buf_size || +if (len > vrp->rbuf_size || msg->len > (len -sizeof (struct rpmsg_hdr ))) {dev_warn (dev,"inbound msg too big: (%d, %d)\n" , len, msg->len);return -EINVAL ;
@@ -751,7 +751,7 @@ static int rpmsg_recv_single(struct virtproc_info *vrp, struct device *dev,dev_warn (dev,"msg received with no recipient\n" );/* publish the real size of the buffer */ -rpmsg_sg_init (&sg, msg, vrp->buf_size); +rpmsg_sg_init (&sg, msg, vrp->rbuf_size);/* add the buffer back to the remote processor's virtqueue */ err =virtqueue_add_inbuf (vrp->rvq, &sg,1 , msg,GFP_KERNEL );
@@ -907,11 +907,24 @@ static int rpmsg_probe(struct virtio_device *vdev)else vrp->num_sbufs =MAX_RPMSG_NUM_BUFS ; - vrp->buf_size =MAX_RPMSG_BUF_SIZE ; +/* try to get buffer size from config space */ +if (virtio_has_feature (vdev,VIRTIO_RPMSG_F_BUFSZ )) { +/* note: virtio_rpmsg_config is defined from remote view */ + virtio_cread(vdev, struct virtio_rpmsg_config, + txbuf_size, &vrp->rbuf_size); + virtio_cread(vdev, struct virtio_rpmsg_config, + rxbuf_size, &vrp->sbuf_size); + } + +/* use the default if resource table doesn't provide one */ +if (!vrp->rbuf_size) + vrp->rbuf_size =MAX_RPMSG_BUF_SIZE ; +if (!vrp->sbuf_size) + vrp->sbuf_size =MAX_RPMSG_BUF_SIZE ;/* allocate coherent memory for the buffers */ vrp->rbufs =dma_alloc_coherent (vdev->dev.parent->parent, - vrp->num_rbufs * vrp->buf_size, + vrp->num_rbufs * vrp->rbuf_size, &vrp->rbufs_dma,GFP_KERNEL );if (!vrp->rbufs) { err = -ENOMEM ;
@@ -922,7 +935,7 @@ static int rpmsg_probe(struct virtio_device *vdev) vrp->rbufs, &vrp->rbufs_dma); vrp->sbufs =dma_alloc_coherent (vdev->dev.parent->parent, - vrp->num_sbufs * vrp->buf_size, + vrp->num_sbufs * vrp->sbuf_size, &vrp->sbufs_dma,GFP_KERNEL );if (!vrp->sbufs) { err = -ENOMEM ;
@@ -935,9 +948,9 @@ static int rpmsg_probe(struct virtio_device *vdev)/* set up the receive buffers */ for (i =0 ; i < vrp->num_rbufs; i++) {struct scatterlist sg; -void *cpu_addr = vrp->rbufs + i * vrp->buf_size; +void *cpu_addr = vrp->rbufs + i * vrp->rbuf_size; -rpmsg_sg_init (&sg, cpu_addr, vrp->buf_size); +rpmsg_sg_init (&sg, cpu_addr, vrp->rbuf_size); err =virtqueue_add_inbuf (vrp->rvq, &sg,1 , cpu_addr,GFP_KERNEL );
@@ -984,11 +997,11 @@ static int rpmsg_probe(struct virtio_device *vdev) free_sbufs:dma_free_coherent (vdev->dev.parent->parent, - vrp->num_sbufs * vrp->buf_size, + vrp->num_sbufs * vrp->sbuf_size, vrp->sbufs, vrp->sbufs_dma); free_rbufs:dma_free_coherent (vdev->dev.parent->parent, - vrp->num_rbufs * vrp->buf_size, + vrp->num_rbufs * vrp->rbuf_size, vrp->rbufs, vrp->rbufs_dma); vqs_del: vdev->config->del_vqs (vrp->vdev);
@@ -1023,10 +1036,10 @@ static void rpmsg_remove(struct virtio_device *vdev) vdev->config->del_vqs (vrp->vdev);dma_free_coherent (vdev->dev.parent->parent, - vrp->num_sbufs * vrp->buf_size, + vrp->num_sbufs * vrp->sbuf_size, vrp->sbufs, vrp->sbufs_dma);dma_free_coherent (vdev->dev.parent->parent, - vrp->num_rbufs * vrp->buf_size, + vrp->num_rbufs * vrp->rbuf_size, vrp->rbufs, vrp->rbufs_dma);kfree (vrp);
@@ -1039,6 +1052,7 @@ static struct virtio_device_id id_table[] = {static unsigned int features[] = {VIRTIO_RPMSG_F_NS , +VIRTIO_RPMSG_F_BUFSZ , };static struct virtio_driver virtio_ipc_driver = {
include/uapi/linux/virtio_rpmsg.h new file +24 -0
@@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +/* +* Copyright (C) Pinecone Inc. 2019 +* Copyright (C) Xiang Xiao <[email protected]> +*/ + +#ifndef _UAPI_LINUX_VIRTIO_RPMSG_H +#define _UAPI_LINUX_VIRTIO_RPMSG_H + +#include <linux/types.h> + +/* The feature bitmap for virtio rpmsg */ +#define VIRTIO_RPMSG_F_NS 0/* RP supports name service notifications */ +#define VIRTIO_RPMSG_F_BUFSZ 2/* RP get buffer size from config space */ + +struct virtio_rpmsg_config { +/* The tx/rx individual buffer size(if VIRTIO_RPMSG_F_BUFSZ) */ +__u32 txbuf_size; +__u32 rxbuf_size; +__u32 reserved[14 ];/* Reserve for the future use */ +/* Put the customize config here */ +} __attribute__((packed)); + +#endif /* _UAPI_LINUX_VIRTIO_RPMSG_H */
Signature · 1 line
2.7.4
Re: [PATCH 3/3] rpmsg: virtio_rpmsg_bus: get buffer size from config space
Hello Xiang,
Similar mechanism has been proposed by Loic 2 years ago (link to the series here https://lkml.org/lkml/2017/3/28/349).
Did you see them? Regarding history, patches seem just on hold...
Main differences (except interesting RX/TX size split) seems that you
- don't use the virtio_config_ops->get
- define a new feature VIRTIO_RPMSG_F_NS.
Regards
Arnaud
Quoted 203 lines · Xiang Xiao
512 bytes isn't always suitable for all case, let firmware maker decide the best value from resource table. enable by
VIRTIO_RPMSG_F_BUFSZfeature bit.Signed-off-by Xiang Xiao <[email protected]>--- drivers/rpmsg/virtio_rpmsg_bus.c | 50 +++++++++++++++++++++++++-------------- include/uapi/linux/virtio_rpmsg.h | 24 +++++++++++++++++++ 2 files changed, 56 insertions(+), 18 deletions(-) create mode 100644 include/uapi/linux/virtio_rpmsg.hdrivers/rpmsg/virtio_rpmsg_bus.c +32
-18@@ -16,6 +16,7 @@#include <linux/virtio.h> #include <linux/virtio_ids.h> #include <linux/virtio_config.h> +#include <linux/virtio_rpmsg.h> #include <linux/scatterlist.h> #include <linux/dma-mapping.h> #include <linux/slab.h> @@ -38,7 +39,8 @@* @sbufs: kernel address of tx buffers * @num_rbufs: total number of buffers for rx * @num_sbufs: total number of buffers for tx -* @buf_size: size of one rx or tx buffer +* @rbuf_size: size of one rx buffer +* @sbuf_size: size of one tx buffer * @last_sbuf: index of last tx buffer used * @rbufs_dma: dma base addr of rx buffers * @sbufs_dma: dma base addr of tx buffers @@ -61,7 +63,8 @@ struct virtproc_info {void *rbufs, *sbufs;unsigned int num_rbufs;unsigned int num_sbufs; -unsigned int buf_size; +unsigned int rbuf_size; +unsigned int sbuf_size;int last_sbuf;dma_addr_t rbufs_dma;dma_addr_t sbufs_dma;@@ -73,9 +76,6 @@ struct virtproc_info {struct rpmsg_endpoint *ns_ept; }; -/* The feature bitmap for virtio rpmsg */ -#define VIRTIO_RPMSG_F_NS 0/* RP supports name service notifications */ -/** * struct rpmsg_hdr - common header for all rpmsg messages * @src: source address @@ -452,7 +452,7 @@ static void *get_a_tx_buf(struct virtproc_info *vrp)/* either pick the next unused tx buffer */ if (vrp->last_sbuf < vrp->num_sbufs) - ret = vrp->sbufs + vrp->buf_size * vrp->last_sbuf++; + ret = vrp->sbufs + vrp->sbuf_size * vrp->last_sbuf++;/* or recycle a used one */ else ret =virtqueue_get_buf (vrp->svq, &len);@@ -578,7 +578,7 @@ static int rpmsg_send_offchannel_raw(struct rpmsg_device *rpdev,* messaging), or to improve the buffer allocator, to support * variable-length buffer sizes. */ -if (len > vrp->buf_size -sizeof (struct rpmsg_hdr )) { +if (len > vrp->sbuf_size -sizeof (struct rpmsg_hdr )) {dev_err (dev,"message is too big (%d)\n" , len);return -EMSGSIZE ; }@@ -718,7 +718,7 @@ static int rpmsg_recv_single(struct virtproc_info *vrp, struct device *dev,* We currently use fixed-sized buffers, so trivially sanitize * the reported payload length. */ -if (len > vrp->buf_size || +if (len > vrp->rbuf_size || msg->len > (len -sizeof (struct rpmsg_hdr ))) {dev_warn (dev,"inbound msg too big: (%d, %d)\n" , len, msg->len);return -EINVAL ;@@ -751,7 +751,7 @@ static int rpmsg_recv_single(struct virtproc_info *vrp, struct device *dev,dev_warn (dev,"msg received with no recipient\n" );/* publish the real size of the buffer */ -rpmsg_sg_init (&sg, msg, vrp->buf_size); +rpmsg_sg_init (&sg, msg, vrp->rbuf_size);/* add the buffer back to the remote processor's virtqueue */ err =virtqueue_add_inbuf (vrp->rvq, &sg,1 , msg,GFP_KERNEL );@@ -907,11 +907,24 @@ static int rpmsg_probe(struct virtio_device *vdev)else vrp->num_sbufs =MAX_RPMSG_NUM_BUFS ; - vrp->buf_size =MAX_RPMSG_BUF_SIZE ; +/* try to get buffer size from config space */ +if (virtio_has_feature (vdev,VIRTIO_RPMSG_F_BUFSZ )) { +/* note: virtio_rpmsg_config is defined from remote view */ + virtio_cread(vdev, struct virtio_rpmsg_config, + txbuf_size, &vrp->rbuf_size); + virtio_cread(vdev, struct virtio_rpmsg_config, + rxbuf_size, &vrp->sbuf_size); + } + +/* use the default if resource table doesn't provide one */ +if (!vrp->rbuf_size) + vrp->rbuf_size =MAX_RPMSG_BUF_SIZE ; +if (!vrp->sbuf_size) + vrp->sbuf_size =MAX_RPMSG_BUF_SIZE ;/* allocate coherent memory for the buffers */ vrp->rbufs =dma_alloc_coherent (vdev->dev.parent->parent, - vrp->num_rbufs * vrp->buf_size, + vrp->num_rbufs * vrp->rbuf_size, &vrp->rbufs_dma,GFP_KERNEL );if (!vrp->rbufs) { err = -ENOMEM ;@@ -922,7 +935,7 @@ static int rpmsg_probe(struct virtio_device *vdev) vrp->rbufs, &vrp->rbufs_dma); vrp->sbufs =dma_alloc_coherent (vdev->dev.parent->parent, - vrp->num_sbufs * vrp->buf_size, + vrp->num_sbufs * vrp->sbuf_size, &vrp->sbufs_dma,GFP_KERNEL );if (!vrp->sbufs) { err = -ENOMEM ;@@ -935,9 +948,9 @@ static int rpmsg_probe(struct virtio_device *vdev)/* set up the receive buffers */ for (i =0 ; i < vrp->num_rbufs; i++) {struct scatterlist sg; -void *cpu_addr = vrp->rbufs + i * vrp->buf_size; +void *cpu_addr = vrp->rbufs + i * vrp->rbuf_size; -rpmsg_sg_init (&sg, cpu_addr, vrp->buf_size); +rpmsg_sg_init (&sg, cpu_addr, vrp->rbuf_size); err =virtqueue_add_inbuf (vrp->rvq, &sg,1 , cpu_addr,GFP_KERNEL );@@ -984,11 +997,11 @@ static int rpmsg_probe(struct virtio_device *vdev) free_sbufs:dma_free_coherent (vdev->dev.parent->parent, - vrp->num_sbufs * vrp->buf_size, + vrp->num_sbufs * vrp->sbuf_size, vrp->sbufs, vrp->sbufs_dma); free_rbufs:dma_free_coherent (vdev->dev.parent->parent, - vrp->num_rbufs * vrp->buf_size, + vrp->num_rbufs * vrp->rbuf_size, vrp->rbufs, vrp->rbufs_dma); vqs_del: vdev->config->del_vqs (vrp->vdev);@@ -1023,10 +1036,10 @@ static void rpmsg_remove(struct virtio_device *vdev) vdev->config->del_vqs (vrp->vdev);dma_free_coherent (vdev->dev.parent->parent, - vrp->num_sbufs * vrp->buf_size, + vrp->num_sbufs * vrp->sbuf_size, vrp->sbufs, vrp->sbufs_dma);dma_free_coherent (vdev->dev.parent->parent, - vrp->num_rbufs * vrp->buf_size, + vrp->num_rbufs * vrp->rbuf_size, vrp->rbufs, vrp->rbufs_dma);kfree (vrp);@@ -1039,6 +1052,7 @@ static struct virtio_device_id id_table[] = {static unsigned int features[] = {VIRTIO_RPMSG_F_NS , +VIRTIO_RPMSG_F_BUFSZ , };static struct virtio_driver virtio_ipc_driver = {include/uapi/linux/virtio_rpmsg.h new file +24
-0@@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +/* +* Copyright (C) Pinecone Inc. 2019 +* Copyright (C) Xiang Xiao <[email protected]> +*/ + +#ifndef _UAPI_LINUX_VIRTIO_RPMSG_H +#define _UAPI_LINUX_VIRTIO_RPMSG_H + +#include <linux/types.h> + +/* The feature bitmap for virtio rpmsg */ +#define VIRTIO_RPMSG_F_NS 0/* RP supports name service notifications */ +#define VIRTIO_RPMSG_F_BUFSZ 2/* RP get buffer size from config space */ + +struct virtio_rpmsg_config { +/* The tx/rx individual buffer size(if VIRTIO_RPMSG_F_BUFSZ) */ +__u32 txbuf_size; +__u32 rxbuf_size; +__u32 reserved[14 ];/* Reserve for the future use */ +/* Put the customize config here */ +} __attribute__((packed)); + +#endif /* _UAPI_LINUX_VIRTIO_RPMSG_H */
Re: [PATCH 3/3] rpmsg: virtio_rpmsg_bus: get buffer size from config space
Quoted 8 lines · Arnaud Pouliquen
Hello Xiang,
Similar mechanism has been proposed by Loic 2 years ago (link to the series here https://lkml.org/lkml/2017/3/28/349).
Did you see them? Regarding history, patches seem just on hold...
Just saw this patchset, so it's common problem hit by many vendor, rpmsg framework need to address it.:)
Quoted 2 lines
Main differences (except interesting RX/TX size split) seems that you
- don't use thevirtio_config_ops->get
virtio_cread call virtio_config_ops->get internally, the ideal is same for both patch, just the implementation detail is different.
Quoted 1 line
- define a new feature
VIRTIO_RPMSG_F_NS.
I add this flag to keep the compatibility with old remote peer, and also follow the common virito driver practice.
Quoted 209 lines
Regards
ArnaudQuoted 203 lines · Xiang Xiao
512 bytes isn't always suitable for all case, let firmware maker decide the best value from resource table. enable by
VIRTIO_RPMSG_F_BUFSZfeature bit.Signed-off-by Xiang Xiao <[email protected]>--- drivers/rpmsg/virtio_rpmsg_bus.c | 50 +++++++++++++++++++++++++-------------- include/uapi/linux/virtio_rpmsg.h | 24 +++++++++++++++++++ 2 files changed, 56 insertions(+), 18 deletions(-) create mode 100644 include/uapi/linux/virtio_rpmsg.hdrivers/rpmsg/virtio_rpmsg_bus.c +32
-18@@ -16,6 +16,7 @@#include <linux/virtio.h> #include <linux/virtio_ids.h> #include <linux/virtio_config.h> +#include <linux/virtio_rpmsg.h> #include <linux/scatterlist.h> #include <linux/dma-mapping.h> #include <linux/slab.h> @@ -38,7 +39,8 @@* @sbufs: kernel address of tx buffers * @num_rbufs: total number of buffers for rx * @num_sbufs: total number of buffers for tx -* @buf_size: size of one rx or tx buffer +* @rbuf_size: size of one rx buffer +* @sbuf_size: size of one tx buffer * @last_sbuf: index of last tx buffer used * @rbufs_dma: dma base addr of rx buffers * @sbufs_dma: dma base addr of tx buffers @@ -61,7 +63,8 @@ struct virtproc_info {void *rbufs, *sbufs;unsigned int num_rbufs;unsigned int num_sbufs; -unsigned int buf_size; +unsigned int rbuf_size; +unsigned int sbuf_size;int last_sbuf;dma_addr_t rbufs_dma;dma_addr_t sbufs_dma;@@ -73,9 +76,6 @@ struct virtproc_info {struct rpmsg_endpoint *ns_ept; }; -/* The feature bitmap for virtio rpmsg */ -#define VIRTIO_RPMSG_F_NS 0/* RP supports name service notifications */ -/** * struct rpmsg_hdr - common header for all rpmsg messages * @src: source address @@ -452,7 +452,7 @@ static void *get_a_tx_buf(struct virtproc_info *vrp)/* either pick the next unused tx buffer */ if (vrp->last_sbuf < vrp->num_sbufs) - ret = vrp->sbufs + vrp->buf_size * vrp->last_sbuf++; + ret = vrp->sbufs + vrp->sbuf_size * vrp->last_sbuf++;/* or recycle a used one */ else ret =virtqueue_get_buf (vrp->svq, &len);@@ -578,7 +578,7 @@ static int rpmsg_send_offchannel_raw(struct rpmsg_device *rpdev,* messaging), or to improve the buffer allocator, to support * variable-length buffer sizes. */ -if (len > vrp->buf_size -sizeof (struct rpmsg_hdr )) { +if (len > vrp->sbuf_size -sizeof (struct rpmsg_hdr )) {dev_err (dev,"message is too big (%d)\n" , len);return -EMSGSIZE ; }@@ -718,7 +718,7 @@ static int rpmsg_recv_single(struct virtproc_info *vrp, struct device *dev,* We currently use fixed-sized buffers, so trivially sanitize * the reported payload length. */ -if (len > vrp->buf_size || +if (len > vrp->rbuf_size || msg->len > (len -sizeof (struct rpmsg_hdr ))) {dev_warn (dev,"inbound msg too big: (%d, %d)\n" , len, msg->len);return -EINVAL ;@@ -751,7 +751,7 @@ static int rpmsg_recv_single(struct virtproc_info *vrp, struct device *dev,dev_warn (dev,"msg received with no recipient\n" );/* publish the real size of the buffer */ -rpmsg_sg_init (&sg, msg, vrp->buf_size); +rpmsg_sg_init (&sg, msg, vrp->rbuf_size);/* add the buffer back to the remote processor's virtqueue */ err =virtqueue_add_inbuf (vrp->rvq, &sg,1 , msg,GFP_KERNEL );@@ -907,11 +907,24 @@ static int rpmsg_probe(struct virtio_device *vdev)else vrp->num_sbufs =MAX_RPMSG_NUM_BUFS ; - vrp->buf_size =MAX_RPMSG_BUF_SIZE ; +/* try to get buffer size from config space */ +if (virtio_has_feature (vdev,VIRTIO_RPMSG_F_BUFSZ )) { +/* note: virtio_rpmsg_config is defined from remote view */ + virtio_cread(vdev, struct virtio_rpmsg_config, + txbuf_size, &vrp->rbuf_size); + virtio_cread(vdev, struct virtio_rpmsg_config, + rxbuf_size, &vrp->sbuf_size); + } + +/* use the default if resource table doesn't provide one */ +if (!vrp->rbuf_size) + vrp->rbuf_size =MAX_RPMSG_BUF_SIZE ; +if (!vrp->sbuf_size) + vrp->sbuf_size =MAX_RPMSG_BUF_SIZE ;/* allocate coherent memory for the buffers */ vrp->rbufs =dma_alloc_coherent (vdev->dev.parent->parent, - vrp->num_rbufs * vrp->buf_size, + vrp->num_rbufs * vrp->rbuf_size, &vrp->rbufs_dma,GFP_KERNEL );if (!vrp->rbufs) { err = -ENOMEM ;@@ -922,7 +935,7 @@ static int rpmsg_probe(struct virtio_device *vdev) vrp->rbufs, &vrp->rbufs_dma); vrp->sbufs =dma_alloc_coherent (vdev->dev.parent->parent, - vrp->num_sbufs * vrp->buf_size, + vrp->num_sbufs * vrp->sbuf_size, &vrp->sbufs_dma,GFP_KERNEL );if (!vrp->sbufs) { err = -ENOMEM ;@@ -935,9 +948,9 @@ static int rpmsg_probe(struct virtio_device *vdev)/* set up the receive buffers */ for (i =0 ; i < vrp->num_rbufs; i++) {struct scatterlist sg; -void *cpu_addr = vrp->rbufs + i * vrp->buf_size; +void *cpu_addr = vrp->rbufs + i * vrp->rbuf_size; -rpmsg_sg_init (&sg, cpu_addr, vrp->buf_size); +rpmsg_sg_init (&sg, cpu_addr, vrp->rbuf_size); err =virtqueue_add_inbuf (vrp->rvq, &sg,1 , cpu_addr,GFP_KERNEL );@@ -984,11 +997,11 @@ static int rpmsg_probe(struct virtio_device *vdev) free_sbufs:dma_free_coherent (vdev->dev.parent->parent, - vrp->num_sbufs * vrp->buf_size, + vrp->num_sbufs * vrp->sbuf_size, vrp->sbufs, vrp->sbufs_dma); free_rbufs:dma_free_coherent (vdev->dev.parent->parent, - vrp->num_rbufs * vrp->buf_size, + vrp->num_rbufs * vrp->rbuf_size, vrp->rbufs, vrp->rbufs_dma); vqs_del: vdev->config->del_vqs (vrp->vdev);@@ -1023,10 +1036,10 @@ static void rpmsg_remove(struct virtio_device *vdev) vdev->config->del_vqs (vrp->vdev);dma_free_coherent (vdev->dev.parent->parent, - vrp->num_sbufs * vrp->buf_size, + vrp->num_sbufs * vrp->sbuf_size, vrp->sbufs, vrp->sbufs_dma);dma_free_coherent (vdev->dev.parent->parent, - vrp->num_rbufs * vrp->buf_size, + vrp->num_rbufs * vrp->rbuf_size, vrp->rbufs, vrp->rbufs_dma);kfree (vrp);@@ -1039,6 +1052,7 @@ static struct virtio_device_id id_table[] = {static unsigned int features[] = {VIRTIO_RPMSG_F_NS , +VIRTIO_RPMSG_F_BUFSZ , };static struct virtio_driver virtio_ipc_driver = {include/uapi/linux/virtio_rpmsg.h new file +24
-0@@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +/* +* Copyright (C) Pinecone Inc. 2019 +* Copyright (C) Xiang Xiao <[email protected]> +*/ + +#ifndef _UAPI_LINUX_VIRTIO_RPMSG_H +#define _UAPI_LINUX_VIRTIO_RPMSG_H + +#include <linux/types.h> + +/* The feature bitmap for virtio rpmsg */ +#define VIRTIO_RPMSG_F_NS 0/* RP supports name service notifications */ +#define VIRTIO_RPMSG_F_BUFSZ 2/* RP get buffer size from config space */ + +struct virtio_rpmsg_config { +/* The tx/rx individual buffer size(if VIRTIO_RPMSG_F_BUFSZ) */ +__u32 txbuf_size; +__u32 rxbuf_size; +__u32 reserved[14 ];/* Reserve for the future use */ +/* Put the customize config here */ +} __attribute__((packed)); + +#endif /* _UAPI_LINUX_VIRTIO_RPMSG_H */
Re: [PATCH 3/3] rpmsg: virtio_rpmsg_bus: get buffer size from config space
Hello Xiang,
Quoted 16 lines · xiang xiao
Quoted 8 lines · Arnaud Pouliquen
Hello Xiang,
Similar mechanism has been proposed by Loic 2 years ago (link to the series here https://lkml.org/lkml/2017/3/28/349).
Did you see them? Regarding history, patches seem just on hold...
Just saw this patchset, so it's common problem hit by many vendor, rpmsg framework need to address it.:)
Quoted 2 lines
Main differences (except interesting RX/TX size split) seems that you
- don't use thevirtio_config_ops->get
Quoted 7 lines
virtio_creadcallvirtio_config_ops->getinternally, the ideal is same for both patch, just the implementation detail is different.Quoted 1 line
- define a new feature
VIRTIO_RPMSG_F_NS.I add this flag to keep the compatibility with old remote peer, and also follow the common virito driver practice.
I discussed with Loic, he is ok to go further with your patch and abandon his one. Please find some remarks below in-line
Quoted 100 lines
Quoted 100 lines
Regards
ArnaudQuoted 93 lines · Xiang Xiao
512 bytes isn't always suitable for all case, let firmware maker decide the best value from resource table. enable by
VIRTIO_RPMSG_F_BUFSZfeature bit.Signed-off-by Xiang Xiao <[email protected]>--- drivers/rpmsg/virtio_rpmsg_bus.c | 50 +++++++++++++++++++++++++-------------- include/uapi/linux/virtio_rpmsg.h | 24 +++++++++++++++++++ 2 files changed, 56 insertions(+), 18 deletions(-) create mode 100644 include/uapi/linux/virtio_rpmsg.hdrivers/rpmsg/virtio_rpmsg_bus.c +9
-9@@ -16,6 +16,7 @@#include <linux/virtio.h> #include <linux/virtio_ids.h> #include <linux/virtio_config.h> +#include <linux/virtio_rpmsg.h> #include <linux/scatterlist.h> #include <linux/dma-mapping.h> #include <linux/slab.h> @@ -38,7 +39,8 @@* @sbufs: kernel address of tx buffers * @num_rbufs: total number of buffers for rx * @num_sbufs: total number of buffers for tx -* @buf_size: size of one rx or tx buffer +* @rbuf_size: size of one rx buffer +* @sbuf_size: size of one tx buffer * @last_sbuf: index of last tx buffer used * @rbufs_dma: dma base addr of rx buffers * @sbufs_dma: dma base addr of tx buffers @@ -61,7 +63,8 @@ struct virtproc_info {void *rbufs, *sbufs;unsigned int num_rbufs;unsigned int num_sbufs; -unsigned int buf_size; +unsigned int rbuf_size; +unsigned int sbuf_size;int last_sbuf;dma_addr_t rbufs_dma;dma_addr_t sbufs_dma;@@ -73,9 +76,6 @@ struct virtproc_info {struct rpmsg_endpoint *ns_ept; }; -/* The feature bitmap for virtio rpmsg */ -#define VIRTIO_RPMSG_F_NS 0/* RP supports name service notifications */ -/** * struct rpmsg_hdr - common header for all rpmsg messages * @src: source address @@ -452,7 +452,7 @@ static void *get_a_tx_buf(struct virtproc_info *vrp)/* either pick the next unused tx buffer */ if (vrp->last_sbuf < vrp->num_sbufs) - ret = vrp->sbufs + vrp->buf_size * vrp->last_sbuf++; + ret = vrp->sbufs + vrp->sbuf_size * vrp->last_sbuf++;/* or recycle a used one */ else ret =virtqueue_get_buf (vrp->svq, &len);@@ -578,7 +578,7 @@ static int rpmsg_send_offchannel_raw(struct rpmsg_device *rpdev,* messaging), or to improve the buffer allocator, to support * variable-length buffer sizes. */ -if (len > vrp->buf_size -sizeof (struct rpmsg_hdr )) { +if (len > vrp->sbuf_size -sizeof (struct rpmsg_hdr )) {dev_err (dev,"message is too big (%d)\n" , len);return -EMSGSIZE ; }@@ -718,7 +718,7 @@ static int rpmsg_recv_single(struct virtproc_info *vrp, struct device *dev,* We currently use fixed-sized buffers, so trivially sanitize * the reported payload length. */ -if (len > vrp->buf_size || +if (len > vrp->rbuf_size || msg->len > (len -sizeof (struct rpmsg_hdr ))) {dev_warn (dev,"inbound msg too big: (%d, %d)\n" , len, msg->len);return -EINVAL ;@@ -751,7 +751,7 @@ static int rpmsg_recv_single(struct virtproc_info *vrp, struct device *dev,dev_warn (dev,"msg received with no recipient\n" );/* publish the real size of the buffer */ -rpmsg_sg_init (&sg, msg, vrp->buf_size); +rpmsg_sg_init (&sg, msg, vrp->rbuf_size);/* add the buffer back to the remote processor's virtqueue */ err =virtqueue_add_inbuf (vrp->rvq, &sg,1 , msg,GFP_KERNEL );@@ -907,11 +907,24 @@ static int rpmsg_probe(struct virtio_device *vdev)else vrp->num_sbufs =MAX_RPMSG_NUM_BUFS ;
Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
drivers/rpmsg/virtio_rpmsg_bus.c +4
-1- vrp->buf_size =MAX_RPMSG_BUF_SIZE ; +/* try to get buffer size from config space */ +if (virtio_has_feature (vdev,VIRTIO_RPMSG_F_BUFSZ )) { +/* note: virtio_rpmsg_config is defined from remote view */ + virtio_cread(vdev, struct virtio_rpmsg_config,Quoted 8 lines
drivers/rpmsg/virtio_rpmsg_bus.c static int rpmsg_probe(struct virtio_device *vdev) · line 921
Quoted 8 lines
drivers/rpmsg/virtio_rpmsg_bus.c +8
-0+ txbuf_size, &vrp->rbuf_size); + virtio_cread(vdev, struct virtio_rpmsg_config, + rxbuf_size, &vrp->sbuf_size); + } + +/* use the default if resource table doesn't provide one */ +if (!vrp->rbuf_size) + vrp->rbuf_size =MAX_RPMSG_BUF_SIZE ;
In this case constant should be renamed DEFAULT_RPMSG_BUF_SIZE as it is no more a max value
Quoted 2 lines
Quoted 2 lines
drivers/rpmsg/virtio_rpmsg_bus.c static int rpmsg_probe(struct virtio_device *vdev) · line 923
Quoted 2 lines
drivers/rpmsg/virtio_rpmsg_bus.c +2
-0+if (!vrp->sbuf_size) + vrp->sbuf_size =MAX_RPMSG_BUF_SIZE ;
Here, if the config space exists you need to update it in consequence to ensure coherency with the remote processor config.
Quoted 63 lines
Quoted 63 lines
Quoted 63 lines
drivers/rpmsg/virtio_rpmsg_bus.c +9
-8/* allocate coherent memory for the buffers */ vrp->rbufs =dma_alloc_coherent (vdev->dev.parent->parent, - vrp->num_rbufs * vrp->buf_size, + vrp->num_rbufs * vrp->rbuf_size, &vrp->rbufs_dma,GFP_KERNEL );if (!vrp->rbufs) { err = -ENOMEM ;@@ -922,7 +935,7 @@ static int rpmsg_probe(struct virtio_device *vdev) vrp->rbufs, &vrp->rbufs_dma); vrp->sbufs =dma_alloc_coherent (vdev->dev.parent->parent, - vrp->num_sbufs * vrp->buf_size, + vrp->num_sbufs * vrp->sbuf_size, &vrp->sbufs_dma,GFP_KERNEL );if (!vrp->sbufs) { err = -ENOMEM ;@@ -935,9 +948,9 @@ static int rpmsg_probe(struct virtio_device *vdev)/* set up the receive buffers */ for (i =0 ; i < vrp->num_rbufs; i++) {struct scatterlist sg; -void *cpu_addr = vrp->rbufs + i * vrp->buf_size; +void *cpu_addr = vrp->rbufs + i * vrp->rbuf_size; -rpmsg_sg_init (&sg, cpu_addr, vrp->buf_size); +rpmsg_sg_init (&sg, cpu_addr, vrp->rbuf_size); err =virtqueue_add_inbuf (vrp->rvq, &sg,1 , cpu_addr,GFP_KERNEL );@@ -984,11 +997,11 @@ static int rpmsg_probe(struct virtio_device *vdev) free_sbufs:dma_free_coherent (vdev->dev.parent->parent, - vrp->num_sbufs * vrp->buf_size, + vrp->num_sbufs * vrp->sbuf_size, vrp->sbufs, vrp->sbufs_dma); free_rbufs:dma_free_coherent (vdev->dev.parent->parent, - vrp->num_rbufs * vrp->buf_size, + vrp->num_rbufs * vrp->rbuf_size, vrp->rbufs, vrp->rbufs_dma); vqs_del: vdev->config->del_vqs (vrp->vdev);@@ -1023,10 +1036,10 @@ static void rpmsg_remove(struct virtio_device *vdev) vdev->config->del_vqs (vrp->vdev);dma_free_coherent (vdev->dev.parent->parent, - vrp->num_sbufs * vrp->buf_size, + vrp->num_sbufs * vrp->sbuf_size, vrp->sbufs, vrp->sbufs_dma);dma_free_coherent (vdev->dev.parent->parent, - vrp->num_rbufs * vrp->buf_size, + vrp->num_rbufs * vrp->rbuf_size, vrp->rbufs, vrp->rbufs_dma);kfree (vrp);@@ -1039,6 +1052,7 @@ static struct virtio_device_id id_table[] = {static unsigned int features[] = {VIRTIO_RPMSG_F_NS , +VIRTIO_RPMSG_F_BUFSZ , };
Quoted 6 lines
Quoted 6 lines
include/uapi/linux/virtio_rpmsg.h
Quoted 6 lines
drivers/rpmsg/virtio_rpmsg_bus.c +0
-0static struct virtio_driver virtio_ipc_driver = {include/uapi/linux/virtio_rpmsg.h new file +0
-0
Strange to define a user space API for kernel usage need. Could you elaborate?
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
include/uapi/linux/virtio_rpmsg.h new file +6
-0@@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +/* +* Copyright (C) Pinecone Inc. 2019 +* Copyright (C) Xiang Xiao <[email protected]> +*/ +
Quoted 8 lines
Quoted 8 lines
include/uapi/linux/virtio_rpmsg.h line 14
Quoted 8 lines
include/uapi/linux/virtio_rpmsg.h new file +8
-0+#ifndef _UAPI_LINUX_VIRTIO_RPMSG_H +#define _UAPI_LINUX_VIRTIO_RPMSG_H + +#include <linux/types.h> + +/* The feature bitmap for virtio rpmsg */ +#define VIRTIO_RPMSG_F_NS 0/* RP supports name service notifications */ +#define VIRTIO_RPMSG_F_BUFSZ 2/* RP get buffer size from config space */
Would be useful to document it in rpmsg.txt
Quoted 9 lines
Quoted 9 lines
include/uapi/linux/virtio_rpmsg.h line 23
Quoted 9 lines
include/uapi/linux/virtio_rpmsg.h new file +9
-0+ +struct virtio_rpmsg_config { +/* The tx/rx individual buffer size(if VIRTIO_RPMSG_F_BUFSZ) */ +__u32 txbuf_size; +__u32 rxbuf_size; +__u32 reserved[14 ];/* Reserve for the future use */ +/* Put the customize config here */ +} __attribute__((packed)); +
Wouldn't it be better to add an identifier and a version fields at the beginning of the structure? Idea would be to simplify a future extension In this case is VIRTIO_RPMSG_F_BUFSZ still useful?
Quoted 2 lines
Quoted 2 lines
include/uapi/linux/virtio_rpmsg.h line 24
Quoted 2 lines
include/uapi/linux/virtio_rpmsg.h new file +1
-0+#endif /* _UAPI_LINUX_VIRTIO_RPMSG_H */
--
Thanks
Arnaud
Re: [PATCH 3/3] rpmsg: virtio_rpmsg_bus: get buffer size from config space
Quoted 129 lines · Arnaud Pouliquen
Hello Xiang,
Quoted 16 lines · xiang xiao
Quoted 8 lines · Arnaud Pouliquen
Hello Xiang,
Similar mechanism has been proposed by Loic 2 years ago (link to the series here https://lkml.org/lkml/2017/3/28/349).
Did you see them? Regarding history, patches seem just on hold...
Just saw this patchset, so it's common problem hit by many vendor, rpmsg framework need to address it.:)
Quoted 2 lines
Main differences (except interesting RX/TX size split) seems that you
- don't use thevirtio_config_ops->getQuoted 7 lines
virtio_creadcallvirtio_config_ops->getinternally, the ideal is same for both patch, just the implementation detail is different.Quoted 1 line
- define a new feature
VIRTIO_RPMSG_F_NS.I add this flag to keep the compatibility with old remote peer, and also follow the common virito driver practice.
I discussed with Loic, he is ok to go further with your patch and abandon his one. Please find some remarks below in-line
Quoted 100 lines
Quoted 100 lines
Regards
ArnaudQuoted 93 lines · Xiang Xiao
512 bytes isn't always suitable for all case, let firmware maker decide the best value from resource table. enable by
VIRTIO_RPMSG_F_BUFSZfeature bit.Signed-off-by Xiang Xiao <[email protected]>--- drivers/rpmsg/virtio_rpmsg_bus.c | 50 +++++++++++++++++++++++++-------------- include/uapi/linux/virtio_rpmsg.h | 24 +++++++++++++++++++ 2 files changed, 56 insertions(+), 18 deletions(-) create mode 100644 include/uapi/linux/virtio_rpmsg.hdrivers/rpmsg/virtio_rpmsg_bus.c +9
-9@@ -16,6 +16,7 @@#include <linux/virtio.h> #include <linux/virtio_ids.h> #include <linux/virtio_config.h> +#include <linux/virtio_rpmsg.h> #include <linux/scatterlist.h> #include <linux/dma-mapping.h> #include <linux/slab.h> @@ -38,7 +39,8 @@* @sbufs: kernel address of tx buffers * @num_rbufs: total number of buffers for rx * @num_sbufs: total number of buffers for tx -* @buf_size: size of one rx or tx buffer +* @rbuf_size: size of one rx buffer +* @sbuf_size: size of one tx buffer * @last_sbuf: index of last tx buffer used * @rbufs_dma: dma base addr of rx buffers * @sbufs_dma: dma base addr of tx buffers @@ -61,7 +63,8 @@ struct virtproc_info {void *rbufs, *sbufs;unsigned int num_rbufs;unsigned int num_sbufs; -unsigned int buf_size; +unsigned int rbuf_size; +unsigned int sbuf_size;int last_sbuf;dma_addr_t rbufs_dma;dma_addr_t sbufs_dma;@@ -73,9 +76,6 @@ struct virtproc_info {struct rpmsg_endpoint *ns_ept; }; -/* The feature bitmap for virtio rpmsg */ -#define VIRTIO_RPMSG_F_NS 0/* RP supports name service notifications */ -/** * struct rpmsg_hdr - common header for all rpmsg messages * @src: source address @@ -452,7 +452,7 @@ static void *get_a_tx_buf(struct virtproc_info *vrp)/* either pick the next unused tx buffer */ if (vrp->last_sbuf < vrp->num_sbufs) - ret = vrp->sbufs + vrp->buf_size * vrp->last_sbuf++; + ret = vrp->sbufs + vrp->sbuf_size * vrp->last_sbuf++;/* or recycle a used one */ else ret =virtqueue_get_buf (vrp->svq, &len);@@ -578,7 +578,7 @@ static int rpmsg_send_offchannel_raw(struct rpmsg_device *rpdev,* messaging), or to improve the buffer allocator, to support * variable-length buffer sizes. */ -if (len > vrp->buf_size -sizeof (struct rpmsg_hdr )) { +if (len > vrp->sbuf_size -sizeof (struct rpmsg_hdr )) {dev_err (dev,"message is too big (%d)\n" , len);return -EMSGSIZE ; }@@ -718,7 +718,7 @@ static int rpmsg_recv_single(struct virtproc_info *vrp, struct device *dev,* We currently use fixed-sized buffers, so trivially sanitize * the reported payload length. */ -if (len > vrp->buf_size || +if (len > vrp->rbuf_size || msg->len > (len -sizeof (struct rpmsg_hdr ))) {dev_warn (dev,"inbound msg too big: (%d, %d)\n" , len, msg->len);return -EINVAL ;@@ -751,7 +751,7 @@ static int rpmsg_recv_single(struct virtproc_info *vrp, struct device *dev,dev_warn (dev,"msg received with no recipient\n" );/* publish the real size of the buffer */ -rpmsg_sg_init (&sg, msg, vrp->buf_size); +rpmsg_sg_init (&sg, msg, vrp->rbuf_size);/* add the buffer back to the remote processor's virtqueue */ err =virtqueue_add_inbuf (vrp->rvq, &sg,1 , msg,GFP_KERNEL );@@ -907,11 +907,24 @@ static int rpmsg_probe(struct virtio_device *vdev)else vrp->num_sbufs =MAX_RPMSG_NUM_BUFS ;
Quoted 15 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
drivers/rpmsg/virtio_rpmsg_bus.c +4
-1- vrp->buf_size =MAX_RPMSG_BUF_SIZE ; +/* try to get buffer size from config space */ +if (virtio_has_feature (vdev,VIRTIO_RPMSG_F_BUFSZ )) { +/* note: virtio_rpmsg_config is defined from remote view */ + virtio_cread(vdev, struct virtio_rpmsg_config,Quoted 8 lines
Quoted 8 lines
drivers/rpmsg/virtio_rpmsg_bus.c static int rpmsg_probe(struct virtio_device *vdev) · line 921
Quoted 8 lines
drivers/rpmsg/virtio_rpmsg_bus.c +8
-0+ txbuf_size, &vrp->rbuf_size); + virtio_cread(vdev, struct virtio_rpmsg_config, + rxbuf_size, &vrp->sbuf_size); + } + +/* use the default if resource table doesn't provide one */ +if (!vrp->rbuf_size) + vrp->rbuf_size =MAX_RPMSG_BUF_SIZE ;In this case constant should be renamed
DEFAULT_RPMSG_BUF_SIZEas it is no more a max value
Yes, DEFAULT_RPMSG_BUF_SIZE is more reasonable now.
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
drivers/rpmsg/virtio_rpmsg_bus.c static int rpmsg_probe(struct virtio_device *vdev) · line 923
Quoted 2 lines
drivers/rpmsg/virtio_rpmsg_bus.c +2
-0+if (!vrp->sbuf_size) + vrp->sbuf_size =MAX_RPMSG_BUF_SIZE ;Here, if the config space exists you need to update it in consequence to ensure coherency with the remote processor config.
The update is already done in if (virtio_has_feature(vdev, VIRTIO_RPMSG_F_BUFSZ)), here just handle the zero value in config space which mean the remote side want to use the default value even VIRTIO_RPMSG_F_BUFSZ set.
For example:
1.remote side want to change one direction buffer size, but keep another direction as default
2.or remote side want to change other config options(define in the furture) not the buffer size
Quoted 64 lines
Quoted 64 lines
Quoted 64 lines
Quoted 64 lines
drivers/rpmsg/virtio_rpmsg_bus.c +9
-8/* allocate coherent memory for the buffers */ vrp->rbufs =dma_alloc_coherent (vdev->dev.parent->parent, - vrp->num_rbufs * vrp->buf_size, + vrp->num_rbufs * vrp->rbuf_size, &vrp->rbufs_dma,GFP_KERNEL );if (!vrp->rbufs) { err = -ENOMEM ;@@ -922,7 +935,7 @@ static int rpmsg_probe(struct virtio_device *vdev) vrp->rbufs, &vrp->rbufs_dma); vrp->sbufs =dma_alloc_coherent (vdev->dev.parent->parent, - vrp->num_sbufs * vrp->buf_size, + vrp->num_sbufs * vrp->sbuf_size, &vrp->sbufs_dma,GFP_KERNEL );if (!vrp->sbufs) { err = -ENOMEM ;@@ -935,9 +948,9 @@ static int rpmsg_probe(struct virtio_device *vdev)/* set up the receive buffers */ for (i =0 ; i < vrp->num_rbufs; i++) {struct scatterlist sg; -void *cpu_addr = vrp->rbufs + i * vrp->buf_size; +void *cpu_addr = vrp->rbufs + i * vrp->rbuf_size; -rpmsg_sg_init (&sg, cpu_addr, vrp->buf_size); +rpmsg_sg_init (&sg, cpu_addr, vrp->rbuf_size); err =virtqueue_add_inbuf (vrp->rvq, &sg,1 , cpu_addr,GFP_KERNEL );@@ -984,11 +997,11 @@ static int rpmsg_probe(struct virtio_device *vdev) free_sbufs:dma_free_coherent (vdev->dev.parent->parent, - vrp->num_sbufs * vrp->buf_size, + vrp->num_sbufs * vrp->sbuf_size, vrp->sbufs, vrp->sbufs_dma); free_rbufs:dma_free_coherent (vdev->dev.parent->parent, - vrp->num_rbufs * vrp->buf_size, + vrp->num_rbufs * vrp->rbuf_size, vrp->rbufs, vrp->rbufs_dma); vqs_del: vdev->config->del_vqs (vrp->vdev);@@ -1023,10 +1036,10 @@ static void rpmsg_remove(struct virtio_device *vdev) vdev->config->del_vqs (vrp->vdev);dma_free_coherent (vdev->dev.parent->parent, - vrp->num_sbufs * vrp->buf_size, + vrp->num_sbufs * vrp->sbuf_size, vrp->sbufs, vrp->sbufs_dma);dma_free_coherent (vdev->dev.parent->parent, - vrp->num_rbufs * vrp->buf_size, + vrp->num_rbufs * vrp->rbuf_size, vrp->rbufs, vrp->rbufs_dma);kfree (vrp);@@ -1039,6 +1052,7 @@ static struct virtio_device_id id_table[] = {static unsigned int features[] = {VIRTIO_RPMSG_F_NS , +VIRTIO_RPMSG_F_BUFSZ , };
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
include/uapi/linux/virtio_rpmsg.h
Quoted 6 lines
drivers/rpmsg/virtio_rpmsg_bus.c +0
-0static struct virtio_driver virtio_ipc_driver = {include/uapi/linux/virtio_rpmsg.h new file +0
-0Strange to define a user space API for kernel usage need. Could you elaborate?
I just follow the practice other virtio drivers(e.g. include/uapi/virtio_net.h) applied, but rpmsg driver don't need to talk with the host VM software like other virtio driver, yes this header file isn't really needed.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
include/uapi/linux/virtio_rpmsg.h new file +7
-0@@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +/* +* Copyright (C) Pinecone Inc. 2019 +* Copyright (C) Xiang Xiao <[email protected]> +*/ + +#ifndef _UAPI_LINUX_VIRTIO_RPMSG_H
Quoted 8 lines
Quoted 7 lines
Quoted 7 lines
include/uapi/linux/virtio_rpmsg.h line 14
Quoted 7 lines
include/uapi/linux/virtio_rpmsg.h new file +7
-0+#define _UAPI_LINUX_VIRTIO_RPMSG_H + +#include <linux/types.h> + +/* The feature bitmap for virtio rpmsg */ +#define VIRTIO_RPMSG_F_NS 0/* RP supports name service notifications */ +#define VIRTIO_RPMSG_F_BUFSZ 2/* RP get buffer size from config space */ Would be useful to document it in rpmsg.txt
Good point, but it is better to put them into this document:
https://docs.oasis-open.org/virtio/virtio/v1.1/csprd01/virtio-v1.1-csprd01.html
like other vritio driver spec.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
include/uapi/linux/virtio_rpmsg.h new file +4
-0+ +struct virtio_rpmsg_config { +/* The tx/rx individual buffer size(if VIRTIO_RPMSG_F_BUFSZ) */ +__u32 txbuf_size;
Quoted 9 lines
Quoted 5 lines
Quoted 5 lines
include/uapi/linux/virtio_rpmsg.h line 23
Quoted 5 lines
include/uapi/linux/virtio_rpmsg.h new file +5
-0+__u32 rxbuf_size; +__u32 reserved[14 ];/* Reserve for the future use */ +/* Put the customize config here */ +} __attribute__((packed)); +Wouldn't it be better to add an identifier and a version fields at the beginning of the structure? Idea would be to simplify a future extension In this case is
VIRTIO_RPMSG_F_BUFSZstill useful?
Yes, I consider this option before, but after review all include/uapi/virtio_*.h, I found that virito driver prefer feature bits than version number to handle the compability issue. For example, if we need introduce more options in the furture, we need:
1.Add new feature bit to notice the option exist
2.Allocate the field from reserved space
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
include/uapi/linux/virtio_rpmsg.h line 24
Quoted 2 lines
include/uapi/linux/virtio_rpmsg.h new file +1
-0+#endif /* _UAPI_LINUX_VIRTIO_RPMSG_H */ --
Thanks
Arnaud
Re: [PATCH 3/3] rpmsg: virtio_rpmsg_bus: get buffer size from config space
Quoted 154 lines · xiang xiao
Quoted 129 lines · Arnaud Pouliquen
Hello Xiang,
Quoted 16 lines · xiang xiao
Quoted 8 lines · Arnaud Pouliquen
Hello Xiang,
Similar mechanism has been proposed by Loic 2 years ago (link to the series here https://lkml.org/lkml/2017/3/28/349).
Did you see them? Regarding history, patches seem just on hold...
Just saw this patchset, so it's common problem hit by many vendor, rpmsg framework need to address it.:)
Quoted 2 lines
Main differences (except interesting RX/TX size split) seems that you
- don't use thevirtio_config_ops->getQuoted 7 lines
virtio_creadcallvirtio_config_ops->getinternally, the ideal is same for both patch, just the implementation detail is different.Quoted 1 line
- define a new feature
VIRTIO_RPMSG_F_NS.I add this flag to keep the compatibility with old remote peer, and also follow the common virito driver practice.
I discussed with Loic, he is ok to go further with your patch and abandon his one. Please find some remarks below in-line
Quoted 100 lines
Quoted 100 lines
Regards
ArnaudQuoted 93 lines · Xiang Xiao
512 bytes isn't always suitable for all case, let firmware maker decide the best value from resource table. enable by
VIRTIO_RPMSG_F_BUFSZfeature bit.Signed-off-by Xiang Xiao <[email protected]>--- drivers/rpmsg/virtio_rpmsg_bus.c | 50 +++++++++++++++++++++++++-------------- include/uapi/linux/virtio_rpmsg.h | 24 +++++++++++++++++++ 2 files changed, 56 insertions(+), 18 deletions(-) create mode 100644 include/uapi/linux/virtio_rpmsg.hdrivers/rpmsg/virtio_rpmsg_bus.c +9
-9@@ -16,6 +16,7 @@#include <linux/virtio.h> #include <linux/virtio_ids.h> #include <linux/virtio_config.h> +#include <linux/virtio_rpmsg.h> #include <linux/scatterlist.h> #include <linux/dma-mapping.h> #include <linux/slab.h> @@ -38,7 +39,8 @@* @sbufs: kernel address of tx buffers * @num_rbufs: total number of buffers for rx * @num_sbufs: total number of buffers for tx -* @buf_size: size of one rx or tx buffer +* @rbuf_size: size of one rx buffer +* @sbuf_size: size of one tx buffer * @last_sbuf: index of last tx buffer used * @rbufs_dma: dma base addr of rx buffers * @sbufs_dma: dma base addr of tx buffers @@ -61,7 +63,8 @@ struct virtproc_info {void *rbufs, *sbufs;unsigned int num_rbufs;unsigned int num_sbufs; -unsigned int buf_size; +unsigned int rbuf_size; +unsigned int sbuf_size;int last_sbuf;dma_addr_t rbufs_dma;dma_addr_t sbufs_dma;@@ -73,9 +76,6 @@ struct virtproc_info {struct rpmsg_endpoint *ns_ept; }; -/* The feature bitmap for virtio rpmsg */ -#define VIRTIO_RPMSG_F_NS 0/* RP supports name service notifications */ -/** * struct rpmsg_hdr - common header for all rpmsg messages * @src: source address @@ -452,7 +452,7 @@ static void *get_a_tx_buf(struct virtproc_info *vrp)/* either pick the next unused tx buffer */ if (vrp->last_sbuf < vrp->num_sbufs) - ret = vrp->sbufs + vrp->buf_size * vrp->last_sbuf++; + ret = vrp->sbufs + vrp->sbuf_size * vrp->last_sbuf++;/* or recycle a used one */ else ret =virtqueue_get_buf (vrp->svq, &len);@@ -578,7 +578,7 @@ static int rpmsg_send_offchannel_raw(struct rpmsg_device *rpdev,* messaging), or to improve the buffer allocator, to support * variable-length buffer sizes. */ -if (len > vrp->buf_size -sizeof (struct rpmsg_hdr )) { +if (len > vrp->sbuf_size -sizeof (struct rpmsg_hdr )) {dev_err (dev,"message is too big (%d)\n" , len);return -EMSGSIZE ; }@@ -718,7 +718,7 @@ static int rpmsg_recv_single(struct virtproc_info *vrp, struct device *dev,* We currently use fixed-sized buffers, so trivially sanitize * the reported payload length. */ -if (len > vrp->buf_size || +if (len > vrp->rbuf_size || msg->len > (len -sizeof (struct rpmsg_hdr ))) {dev_warn (dev,"inbound msg too big: (%d, %d)\n" , len, msg->len);return -EINVAL ;@@ -751,7 +751,7 @@ static int rpmsg_recv_single(struct virtproc_info *vrp, struct device *dev,dev_warn (dev,"msg received with no recipient\n" );/* publish the real size of the buffer */ -rpmsg_sg_init (&sg, msg, vrp->buf_size); +rpmsg_sg_init (&sg, msg, vrp->rbuf_size);/* add the buffer back to the remote processor's virtqueue */ err =virtqueue_add_inbuf (vrp->rvq, &sg,1 , msg,GFP_KERNEL );@@ -907,11 +907,24 @@ static int rpmsg_probe(struct virtio_device *vdev)else vrp->num_sbufs =MAX_RPMSG_NUM_BUFS ;Quoted 15 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
drivers/rpmsg/virtio_rpmsg_bus.c +4
-1- vrp->buf_size =MAX_RPMSG_BUF_SIZE ; +/* try to get buffer size from config space */ +if (virtio_has_feature (vdev,VIRTIO_RPMSG_F_BUFSZ )) { +/* note: virtio_rpmsg_config is defined from remote view */ + virtio_cread(vdev, struct virtio_rpmsg_config,Quoted 8 lines
Quoted 8 lines
drivers/rpmsg/virtio_rpmsg_bus.c static int rpmsg_probe(struct virtio_device *vdev) · line 921
Quoted 8 lines
drivers/rpmsg/virtio_rpmsg_bus.c +8
-0+ txbuf_size, &vrp->rbuf_size); + virtio_cread(vdev, struct virtio_rpmsg_config, + rxbuf_size, &vrp->sbuf_size); + } + +/* use the default if resource table doesn't provide one */ +if (!vrp->rbuf_size) + vrp->rbuf_size =MAX_RPMSG_BUF_SIZE ;In this case constant should be renamed
DEFAULT_RPMSG_BUF_SIZEas it is no more a max valueYes,
DEFAULT_RPMSG_BUF_SIZEis more reasonable now.Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
drivers/rpmsg/virtio_rpmsg_bus.c static int rpmsg_probe(struct virtio_device *vdev) · line 923
Quoted 2 lines
drivers/rpmsg/virtio_rpmsg_bus.c +2
-0+if (!vrp->sbuf_size) + vrp->sbuf_size =MAX_RPMSG_BUF_SIZE ;Here, if the config space exists you need to update it in consequence to ensure coherency with the remote processor config.
Quoted 9 lines
The update is already done in if (
virtio_has_feature(vdev, VIRTIO_RPMSG_F_BUFSZ)), here just handle the zero value in config space which mean the remote side want to use the default value evenVIRTIO_RPMSG_F_BUFSZset.
For example:
1.remote side want to change one direction buffer size, but keep another direction as default
2.or remote side want to change other config options(define in the furture) not the buffer size
In code above i can see a virtio_cread of the config structure, but no writing of it...
I mentioned the configs space in the resource table itself. Without an update, you must ensure that both have the same default value... In addition, it makes sense that the master can update the buffer size according to some other constraints.
Quoted 126 lines
Quoted 65 lines
Quoted 65 lines
Quoted 65 lines
Quoted 65 lines
drivers/rpmsg/virtio_rpmsg_bus.c +9
-8/* allocate coherent memory for the buffers */ vrp->rbufs =dma_alloc_coherent (vdev->dev.parent->parent, - vrp->num_rbufs * vrp->buf_size, + vrp->num_rbufs * vrp->rbuf_size, &vrp->rbufs_dma,GFP_KERNEL );if (!vrp->rbufs) { err = -ENOMEM ;@@ -922,7 +935,7 @@ static int rpmsg_probe(struct virtio_device *vdev) vrp->rbufs, &vrp->rbufs_dma); vrp->sbufs =dma_alloc_coherent (vdev->dev.parent->parent, - vrp->num_sbufs * vrp->buf_size, + vrp->num_sbufs * vrp->sbuf_size, &vrp->sbufs_dma,GFP_KERNEL );if (!vrp->sbufs) { err = -ENOMEM ;@@ -935,9 +948,9 @@ static int rpmsg_probe(struct virtio_device *vdev)/* set up the receive buffers */ for (i =0 ; i < vrp->num_rbufs; i++) {struct scatterlist sg; -void *cpu_addr = vrp->rbufs + i * vrp->buf_size; +void *cpu_addr = vrp->rbufs + i * vrp->rbuf_size; -rpmsg_sg_init (&sg, cpu_addr, vrp->buf_size); +rpmsg_sg_init (&sg, cpu_addr, vrp->rbuf_size); err =virtqueue_add_inbuf (vrp->rvq, &sg,1 , cpu_addr,GFP_KERNEL );@@ -984,11 +997,11 @@ static int rpmsg_probe(struct virtio_device *vdev) free_sbufs:dma_free_coherent (vdev->dev.parent->parent, - vrp->num_sbufs * vrp->buf_size, + vrp->num_sbufs * vrp->sbuf_size, vrp->sbufs, vrp->sbufs_dma); free_rbufs:dma_free_coherent (vdev->dev.parent->parent, - vrp->num_rbufs * vrp->buf_size, + vrp->num_rbufs * vrp->rbuf_size, vrp->rbufs, vrp->rbufs_dma); vqs_del: vdev->config->del_vqs (vrp->vdev);@@ -1023,10 +1036,10 @@ static void rpmsg_remove(struct virtio_device *vdev) vdev->config->del_vqs (vrp->vdev);dma_free_coherent (vdev->dev.parent->parent, - vrp->num_sbufs * vrp->buf_size, + vrp->num_sbufs * vrp->sbuf_size, vrp->sbufs, vrp->sbufs_dma);dma_free_coherent (vdev->dev.parent->parent, - vrp->num_rbufs * vrp->buf_size, + vrp->num_rbufs * vrp->rbuf_size, vrp->rbufs, vrp->rbufs_dma);kfree (vrp);@@ -1039,6 +1052,7 @@ static struct virtio_device_id id_table[] = {static unsigned int features[] = {VIRTIO_RPMSG_F_NS , +VIRTIO_RPMSG_F_BUFSZ , };Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
include/uapi/linux/virtio_rpmsg.h
Quoted 6 lines
drivers/rpmsg/virtio_rpmsg_bus.c +0
-0static struct virtio_driver virtio_ipc_driver = {include/uapi/linux/virtio_rpmsg.h new file +0
-0Strange to define a user space API for kernel usage need. Could you elaborate?
I just follow the practice other virtio drivers(e.g.
include/uapi/virtio_net.h) applied, but rpmsg driver don't need to talk with the host VM software like other virtio driver, yes this header file isn't really needed.Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
include/uapi/linux/virtio_rpmsg.h new file +7
-0@@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +/* +* Copyright (C) Pinecone Inc. 2019 +* Copyright (C) Xiang Xiao <[email protected]> +*/ + +#ifndef _UAPI_LINUX_VIRTIO_RPMSG_HQuoted 8 lines
Quoted 7 lines
Quoted 7 lines
include/uapi/linux/virtio_rpmsg.h line 14
Quoted 7 lines
include/uapi/linux/virtio_rpmsg.h new file +7
-0+#define _UAPI_LINUX_VIRTIO_RPMSG_H + +#include <linux/types.h> + +/* The feature bitmap for virtio rpmsg */ +#define VIRTIO_RPMSG_F_NS 0/* RP supports name service notifications */ +#define VIRTIO_RPMSG_F_BUFSZ 2/* RP get buffer size from config space */ Would be useful to document it in rpmsg.txt
Good point, but it is better to put them into this document:
https://docs.oasis-open.org/virtio/virtio/v1.1/csprd01/virtio-v1.1-csprd01.html
like other vritio driver spec.Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
include/uapi/linux/virtio_rpmsg.h new file +4
-0+ +struct virtio_rpmsg_config { +/* The tx/rx individual buffer size(if VIRTIO_RPMSG_F_BUFSZ) */ +__u32 txbuf_size;Quoted 9 lines
Quoted 5 lines
Quoted 5 lines
include/uapi/linux/virtio_rpmsg.h line 23
Quoted 5 lines
include/uapi/linux/virtio_rpmsg.h new file +5
-0+__u32 rxbuf_size; +__u32 reserved[14 ];/* Reserve for the future use */ +/* Put the customize config here */ +} __attribute__((packed)); +Wouldn't it be better to add an identifier and a version fields at the beginning of the structure? Idea would be to simplify a future extension In this case is
VIRTIO_RPMSG_F_BUFSZstill useful?Yes, I consider this option before, but after review all
include/uapi/virtio_*.h, I found that virito driver prefer feature bits than version number to handle the compability issue. For example, if we need introduce more options in the furture, we need:
1.Add new feature bit to notice the option exist
2.Allocate the field from reserved spaceQuoted 5 lines
Quoted 2 lines
Quoted 2 lines
include/uapi/linux/virtio_rpmsg.h line 24
Quoted 2 lines
include/uapi/linux/virtio_rpmsg.h new file +1
-0+#endif /* _UAPI_LINUX_VIRTIO_RPMSG_H */ --
Thanks
Arnaud
Re: [PATCH 3/3] rpmsg: virtio_rpmsg_bus: get buffer size from config space
Quoted 168 lines · Arnaud Pouliquen
Quoted 154 lines · xiang xiao
Quoted 129 lines · Arnaud Pouliquen
Hello Xiang,
Quoted 16 lines · xiang xiao
Quoted 8 lines · Arnaud Pouliquen
Hello Xiang,
Similar mechanism has been proposed by Loic 2 years ago (link to the series here https://lkml.org/lkml/2017/3/28/349).
Did you see them? Regarding history, patches seem just on hold...
Just saw this patchset, so it's common problem hit by many vendor, rpmsg framework need to address it.:)
Quoted 2 lines
Main differences (except interesting RX/TX size split) seems that you
- don't use thevirtio_config_ops->getQuoted 7 lines
virtio_creadcallvirtio_config_ops->getinternally, the ideal is same for both patch, just the implementation detail is different.Quoted 1 line
- define a new feature
VIRTIO_RPMSG_F_NS.I add this flag to keep the compatibility with old remote peer, and also follow the common virito driver practice.
I discussed with Loic, he is ok to go further with your patch and abandon his one. Please find some remarks below in-line
Quoted 100 lines
Quoted 100 lines
Regards
ArnaudQuoted 93 lines · Xiang Xiao
512 bytes isn't always suitable for all case, let firmware maker decide the best value from resource table. enable by
VIRTIO_RPMSG_F_BUFSZfeature bit.Signed-off-by Xiang Xiao <[email protected]>--- drivers/rpmsg/virtio_rpmsg_bus.c | 50 +++++++++++++++++++++++++-------------- include/uapi/linux/virtio_rpmsg.h | 24 +++++++++++++++++++ 2 files changed, 56 insertions(+), 18 deletions(-) create mode 100644 include/uapi/linux/virtio_rpmsg.hdrivers/rpmsg/virtio_rpmsg_bus.c +9
-9@@ -16,6 +16,7 @@#include <linux/virtio.h> #include <linux/virtio_ids.h> #include <linux/virtio_config.h> +#include <linux/virtio_rpmsg.h> #include <linux/scatterlist.h> #include <linux/dma-mapping.h> #include <linux/slab.h> @@ -38,7 +39,8 @@* @sbufs: kernel address of tx buffers * @num_rbufs: total number of buffers for rx * @num_sbufs: total number of buffers for tx -* @buf_size: size of one rx or tx buffer +* @rbuf_size: size of one rx buffer +* @sbuf_size: size of one tx buffer * @last_sbuf: index of last tx buffer used * @rbufs_dma: dma base addr of rx buffers * @sbufs_dma: dma base addr of tx buffers @@ -61,7 +63,8 @@ struct virtproc_info {void *rbufs, *sbufs;unsigned int num_rbufs;unsigned int num_sbufs; -unsigned int buf_size; +unsigned int rbuf_size; +unsigned int sbuf_size;int last_sbuf;dma_addr_t rbufs_dma;dma_addr_t sbufs_dma;@@ -73,9 +76,6 @@ struct virtproc_info {struct rpmsg_endpoint *ns_ept; }; -/* The feature bitmap for virtio rpmsg */ -#define VIRTIO_RPMSG_F_NS 0/* RP supports name service notifications */ -/** * struct rpmsg_hdr - common header for all rpmsg messages * @src: source address @@ -452,7 +452,7 @@ static void *get_a_tx_buf(struct virtproc_info *vrp)/* either pick the next unused tx buffer */ if (vrp->last_sbuf < vrp->num_sbufs) - ret = vrp->sbufs + vrp->buf_size * vrp->last_sbuf++; + ret = vrp->sbufs + vrp->sbuf_size * vrp->last_sbuf++;/* or recycle a used one */ else ret =virtqueue_get_buf (vrp->svq, &len);@@ -578,7 +578,7 @@ static int rpmsg_send_offchannel_raw(struct rpmsg_device *rpdev,* messaging), or to improve the buffer allocator, to support * variable-length buffer sizes. */ -if (len > vrp->buf_size -sizeof (struct rpmsg_hdr )) { +if (len > vrp->sbuf_size -sizeof (struct rpmsg_hdr )) {dev_err (dev,"message is too big (%d)\n" , len);return -EMSGSIZE ; }@@ -718,7 +718,7 @@ static int rpmsg_recv_single(struct virtproc_info *vrp, struct device *dev,* We currently use fixed-sized buffers, so trivially sanitize * the reported payload length. */ -if (len > vrp->buf_size || +if (len > vrp->rbuf_size || msg->len > (len -sizeof (struct rpmsg_hdr ))) {dev_warn (dev,"inbound msg too big: (%d, %d)\n" , len, msg->len);return -EINVAL ;@@ -751,7 +751,7 @@ static int rpmsg_recv_single(struct virtproc_info *vrp, struct device *dev,dev_warn (dev,"msg received with no recipient\n" );/* publish the real size of the buffer */ -rpmsg_sg_init (&sg, msg, vrp->buf_size); +rpmsg_sg_init (&sg, msg, vrp->rbuf_size);/* add the buffer back to the remote processor's virtqueue */ err =virtqueue_add_inbuf (vrp->rvq, &sg,1 , msg,GFP_KERNEL );@@ -907,11 +907,24 @@ static int rpmsg_probe(struct virtio_device *vdev)else vrp->num_sbufs =MAX_RPMSG_NUM_BUFS ;Quoted 15 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
drivers/rpmsg/virtio_rpmsg_bus.c +4
-1- vrp->buf_size =MAX_RPMSG_BUF_SIZE ; +/* try to get buffer size from config space */ +if (virtio_has_feature (vdev,VIRTIO_RPMSG_F_BUFSZ )) { +/* note: virtio_rpmsg_config is defined from remote view */ + virtio_cread(vdev, struct virtio_rpmsg_config,Quoted 8 lines
Quoted 8 lines
drivers/rpmsg/virtio_rpmsg_bus.c static int rpmsg_probe(struct virtio_device *vdev) · line 921
Quoted 8 lines
drivers/rpmsg/virtio_rpmsg_bus.c +8
-0+ txbuf_size, &vrp->rbuf_size); + virtio_cread(vdev, struct virtio_rpmsg_config, + rxbuf_size, &vrp->sbuf_size); + } + +/* use the default if resource table doesn't provide one */ +if (!vrp->rbuf_size) + vrp->rbuf_size =MAX_RPMSG_BUF_SIZE ;In this case constant should be renamed
DEFAULT_RPMSG_BUF_SIZEas it is no more a max valueYes,
DEFAULT_RPMSG_BUF_SIZEis more reasonable now.Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
drivers/rpmsg/virtio_rpmsg_bus.c static int rpmsg_probe(struct virtio_device *vdev) · line 923
Quoted 2 lines
drivers/rpmsg/virtio_rpmsg_bus.c +2
-0+if (!vrp->sbuf_size) + vrp->sbuf_size =MAX_RPMSG_BUF_SIZE ;Here, if the config space exists you need to update it in consequence to ensure coherency with the remote processor config.
Quoted 9 lines
The update is already done in if (
virtio_has_feature(vdev, VIRTIO_RPMSG_F_BUFSZ)), here just handle the zero value in config space which mean the remote side want to use the default value evenVIRTIO_RPMSG_F_BUFSZset.
For example:
1.remote side want to change one direction buffer size, but keep another direction as default
2.or remote side want to change other config options(define in the furture) not the buffer size
Quoted 6 lines
In code above i can see a
virtio_creadof the config structure, but no writing of it...
I mentioned the configs space in the resource table itself. Without an update, you must ensure that both have the same default value... In addition, it makes sense that the master can update the buffer size according to some other constraints.
Get your point, thanks.
Quoted 127 lines
Quoted 127 lines
Quoted 66 lines
Quoted 66 lines
Quoted 66 lines
Quoted 66 lines
drivers/rpmsg/virtio_rpmsg_bus.c +9
-8/* allocate coherent memory for the buffers */ vrp->rbufs =dma_alloc_coherent (vdev->dev.parent->parent, - vrp->num_rbufs * vrp->buf_size, + vrp->num_rbufs * vrp->rbuf_size, &vrp->rbufs_dma,GFP_KERNEL );if (!vrp->rbufs) { err = -ENOMEM ;@@ -922,7 +935,7 @@ static int rpmsg_probe(struct virtio_device *vdev) vrp->rbufs, &vrp->rbufs_dma); vrp->sbufs =dma_alloc_coherent (vdev->dev.parent->parent, - vrp->num_sbufs * vrp->buf_size, + vrp->num_sbufs * vrp->sbuf_size, &vrp->sbufs_dma,GFP_KERNEL );if (!vrp->sbufs) { err = -ENOMEM ;@@ -935,9 +948,9 @@ static int rpmsg_probe(struct virtio_device *vdev)/* set up the receive buffers */ for (i =0 ; i < vrp->num_rbufs; i++) {struct scatterlist sg; -void *cpu_addr = vrp->rbufs + i * vrp->buf_size; +void *cpu_addr = vrp->rbufs + i * vrp->rbuf_size; -rpmsg_sg_init (&sg, cpu_addr, vrp->buf_size); +rpmsg_sg_init (&sg, cpu_addr, vrp->rbuf_size); err =virtqueue_add_inbuf (vrp->rvq, &sg,1 , cpu_addr,GFP_KERNEL );@@ -984,11 +997,11 @@ static int rpmsg_probe(struct virtio_device *vdev) free_sbufs:dma_free_coherent (vdev->dev.parent->parent, - vrp->num_sbufs * vrp->buf_size, + vrp->num_sbufs * vrp->sbuf_size, vrp->sbufs, vrp->sbufs_dma); free_rbufs:dma_free_coherent (vdev->dev.parent->parent, - vrp->num_rbufs * vrp->buf_size, + vrp->num_rbufs * vrp->rbuf_size, vrp->rbufs, vrp->rbufs_dma); vqs_del: vdev->config->del_vqs (vrp->vdev);@@ -1023,10 +1036,10 @@ static void rpmsg_remove(struct virtio_device *vdev) vdev->config->del_vqs (vrp->vdev);dma_free_coherent (vdev->dev.parent->parent, - vrp->num_sbufs * vrp->buf_size, + vrp->num_sbufs * vrp->sbuf_size, vrp->sbufs, vrp->sbufs_dma);dma_free_coherent (vdev->dev.parent->parent, - vrp->num_rbufs * vrp->buf_size, + vrp->num_rbufs * vrp->rbuf_size, vrp->rbufs, vrp->rbufs_dma);kfree (vrp);@@ -1039,6 +1052,7 @@ static struct virtio_device_id id_table[] = {static unsigned int features[] = {VIRTIO_RPMSG_F_NS , +VIRTIO_RPMSG_F_BUFSZ , };Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
include/uapi/linux/virtio_rpmsg.h
Quoted 6 lines
drivers/rpmsg/virtio_rpmsg_bus.c +0
-0static struct virtio_driver virtio_ipc_driver = {include/uapi/linux/virtio_rpmsg.h new file +0
-0Strange to define a user space API for kernel usage need. Could you elaborate?
I just follow the practice other virtio drivers(e.g.
include/uapi/virtio_net.h) applied, but rpmsg driver don't need to talk with the host VM software like other virtio driver, yes this header file isn't really needed.Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
include/uapi/linux/virtio_rpmsg.h new file +7
-0@@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +/* +* Copyright (C) Pinecone Inc. 2019 +* Copyright (C) Xiang Xiao <[email protected]> +*/ + +#ifndef _UAPI_LINUX_VIRTIO_RPMSG_HQuoted 8 lines
Quoted 7 lines
Quoted 7 lines
include/uapi/linux/virtio_rpmsg.h line 14
Quoted 7 lines
include/uapi/linux/virtio_rpmsg.h new file +7
-0+#define _UAPI_LINUX_VIRTIO_RPMSG_H + +#include <linux/types.h> + +/* The feature bitmap for virtio rpmsg */ +#define VIRTIO_RPMSG_F_NS 0/* RP supports name service notifications */ +#define VIRTIO_RPMSG_F_BUFSZ 2/* RP get buffer size from config space */ Would be useful to document it in rpmsg.txt
Good point, but it is better to put them into this document:
https://docs.oasis-open.org/virtio/virtio/v1.1/csprd01/virtio-v1.1-csprd01.html
like other vritio driver spec.Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
include/uapi/linux/virtio_rpmsg.h new file +4
-0+ +struct virtio_rpmsg_config { +/* The tx/rx individual buffer size(if VIRTIO_RPMSG_F_BUFSZ) */ +__u32 txbuf_size;Quoted 9 lines
Quoted 5 lines
Quoted 5 lines
include/uapi/linux/virtio_rpmsg.h line 23
Quoted 5 lines
include/uapi/linux/virtio_rpmsg.h new file +5
-0+__u32 rxbuf_size; +__u32 reserved[14 ];/* Reserve for the future use */ +/* Put the customize config here */ +} __attribute__((packed)); +Wouldn't it be better to add an identifier and a version fields at the beginning of the structure? Idea would be to simplify a future extension In this case is
VIRTIO_RPMSG_F_BUFSZstill useful?Yes, I consider this option before, but after review all
include/uapi/virtio_*.h, I found that virito driver prefer feature bits than version number to handle the compability issue. For example, if we need introduce more options in the furture, we need:
1.Add new feature bit to notice the option exist
2.Allocate the field from reserved spaceQuoted 5 lines
Quoted 2 lines
Quoted 2 lines
include/uapi/linux/virtio_rpmsg.h line 24
Quoted 2 lines
include/uapi/linux/virtio_rpmsg.h new file +1
-0+#endif /* _UAPI_LINUX_VIRTIO_RPMSG_H */ --
Thanks
Arnaud
Quoted 8 lines · Xiang Xiao
Hi,
This series enhance the buffer allocation by:
1.Support the different buffer number in rx/tx direction 2.Get the individual rx/tx buffer size from config spaceHere is the related OpenAMP change:
https://github.com/OpenAMP/open-amp/pull/155
This looks pretty reasonable, but can you confirm that it's possible to use new firmware with an old Linux kernel when introducing this?
But ever since we discussed Loic's similar proposal earlier I've been questioning if the fixed buffer size isn't just an artifact of how we preallocate our buffers. The virtqueue seems to support arbitrary sizes of buffers and I see that the receive function in OpenAMP has been fixed to put back the buffer of the size that was received, rather than 512 bytes. So it seems like Linux would be able to send whatever size messages to OpenAMP it would handle it.
The question is if we could do the same in the other direction, perhaps by letting the OpenAMP side do it's message allocation when it's sending, rather than Linux pushing inbufs to be filled by the remote.
This would remove the problem of always having suboptimal buffer sizes.
Regards,
Bjorn
Quoted 13 lines
Xiang Xiao (3):
rpmsg:virtio_rpmsg_bus: allow the different vring size for send/recv
rpmsg:virtio_rpmsg_bus: allocate rx/tx buffer separately
rpmsg:virtio_rpmsg_bus: get buffer size from config spacedrivers/rpmsg/virtio_rpmsg_bus.c | 127 +++++++++++++++++++++++--------------- include/uapi/linux/virtio_rpmsg.h | 24 +++++++ 2 files changed, 100 insertions(+), 51 deletions(-) create mode 100644 include/uapi/linux/virtio_rpmsg.hSignature · 1 line
2.7.4
Hi Bjorn,
Quoted 23 lines · Bjorn Andersson
Quoted 8 lines · Xiang Xiao
Hi,
This series enhance the buffer allocation by:
1.Support the different buffer number in rx/tx direction 2.Get the individual rx/tx buffer size from config spaceHere is the related OpenAMP change:
https://github.com/OpenAMP/open-amp/pull/155This looks pretty reasonable, but can you confirm that it's possible to use new firmware with an old Linux kernel when introducing this?
But ever since we discussed Loic's similar proposal earlier I've been questioning if the fixed buffer size isn't just an artifact of how we preallocate our buffers. The virtqueue seems to support arbitrary sizes of buffers and I see that the receive function in OpenAMP has been fixed to put back the buffer of the size that was received, rather than 512 bytes. So it seems like Linux would be able to send whatever size messages to OpenAMP it would handle it.
Quoted 3 lines
The question is if we could do the same in the other direction, perhaps by letting the OpenAMP side do it's message allocation when it's sending, rather than Linux pushing inbufs to be filled by the remote.
IMHO, both could be useful and could be not correlated. On-the fly buffer allocation seems more efficient but needs an allocator.This can be a generic allocator (with a va to da) for system where large amount of memories are accessible from both side.
Now what about system with small shared memory? In this case you have to deal with a limited/optimized memory chunk. To avoid memory fragmentation the allocator should have a pre-reserved buffers pool(so similar to existing implementation). This serie seems useful to optimize the size of the pre-reserved pool.
Quoted 19 lines
This would remove the problem of always having suboptimal buffer sizes.
Regards,
BjornQuoted 13 lines
Xiang Xiao (3):
rpmsg:virtio_rpmsg_bus: allow the different vring size for send/recv
rpmsg:virtio_rpmsg_bus: allocate rx/tx buffer separately
rpmsg:virtio_rpmsg_bus: get buffer size from config spacedrivers/rpmsg/virtio_rpmsg_bus.c | 127 +++++++++++++++++++++++--------------- include/uapi/linux/virtio_rpmsg.h | 24 +++++++ 2 files changed, 100 insertions(+), 51 deletions(-) create mode 100644 include/uapi/linux/virtio_rpmsg.hSignature · 1 line
2.7.4
--
Regards,
Arnaud
Quoted 36 lines · Arnaud Pouliquen
Hi Bjorn,
Quoted 23 lines · Bjorn Andersson
Quoted 8 lines · Xiang Xiao
Hi,
This series enhance the buffer allocation by:
1.Support the different buffer number in rx/tx direction 2.Get the individual rx/tx buffer size from config spaceHere is the related OpenAMP change:
https://github.com/OpenAMP/open-amp/pull/155This looks pretty reasonable, but can you confirm that it's possible to use new firmware with an old Linux kernel when introducing this?
But ever since we discussed Loic's similar proposal earlier I've been questioning if the fixed buffer size isn't just an artifact of how we preallocate our buffers. The virtqueue seems to support arbitrary sizes of buffers and I see that the receive function in OpenAMP has been fixed to put back the buffer of the size that was received, rather than 512 bytes. So it seems like Linux would be able to send whatever size messages to OpenAMP it would handle it.
Quoted 3 lines
The question is if we could do the same in the other direction, perhaps by letting the OpenAMP side do it's message allocation when it's sending, rather than Linux pushing inbufs to be filled by the remote.
IMHO, both could be useful and could be not correlated. On-the fly buffer allocation seems more efficient but needs an allocator.This can be a generic allocator (with a va to da) for system where large amount of memories are accessible from both side.
Quoted 6 lines
Now what about system with small shared memory? In this case you have to deal with a limited/optimized memory chunk. To avoid memory fragmentation the allocator should have a pre-reserved buffers pool(so similar to existing implementation). This serie seems useful to optimize the size of the pre-reserved pool.
Maybe we can reuse rxbuf_size/txbuf_size to trigger the different allocation policy:
1.If buf_size equal 0xfffffff, turn on the dynamic allocator
2.If buf_size equall 0, turn on the fixed allocator with the default buffer size 3.otherwise, turn on the fixed allocator with the configed buffer size So, both requirement could be satisfied without breaking the compatibility.
Quoted 24 lines
Quoted 19 lines
This would remove the problem of always having suboptimal buffer sizes.
Regards,
BjornQuoted 13 lines
Xiang Xiao (3):
rpmsg:virtio_rpmsg_bus: allow the different vring size for send/recv
rpmsg:virtio_rpmsg_bus: allocate rx/tx buffer separately
rpmsg:virtio_rpmsg_bus: get buffer size from config spacedrivers/rpmsg/virtio_rpmsg_bus.c | 127 +++++++++++++++++++++++--------------- include/uapi/linux/virtio_rpmsg.h | 24 +++++++ 2 files changed, 100 insertions(+), 51 deletions(-) create mode 100644 include/uapi/linux/virtio_rpmsg.h--
2.7.4--
Regards,
Arnaud
Quoted 35 lines · Arnaud Pouliquen
Hi Bjorn,
Quoted 23 lines · Bjorn Andersson
Quoted 8 lines · Xiang Xiao
Hi,
This series enhance the buffer allocation by:
1.Support the different buffer number in rx/tx direction 2.Get the individual rx/tx buffer size from config spaceHere is the related OpenAMP change:
https://github.com/OpenAMP/open-amp/pull/155This looks pretty reasonable, but can you confirm that it's possible to use new firmware with an old Linux kernel when introducing this?
But ever since we discussed Loic's similar proposal earlier I've been questioning if the fixed buffer size isn't just an artifact of how we preallocate our buffers. The virtqueue seems to support arbitrary sizes of buffers and I see that the receive function in OpenAMP has been fixed to put back the buffer of the size that was received, rather than 512 bytes. So it seems like Linux would be able to send whatever size messages to OpenAMP it would handle it.
Quoted 3 lines
The question is if we could do the same in the other direction, perhaps by letting the OpenAMP side do it's message allocation when it's sending, rather than Linux pushing inbufs to be filled by the remote.
IMHO, both could be useful and could be not correlated. On-the fly buffer allocation seems more efficient but needs an allocator.This can be a generic allocator (with a va to da) for system where large amount of memories are accessible from both side.
Quoted 6 lines
Now what about system with small shared memory? In this case you have to deal with a limited/optimized memory chunk. To avoid memory fragmentation the allocator should have a pre-reserved buffers pool(so similar to existing implementation). This serie seems useful to optimize the size of the pre-reserved pool.
Having an implementation that uses small fixed size buffers seems very reasonable and I'm in favour of making the message size configurable.
I would however prefer to have this implemented in a way where the remote side should not be receiving messages in a way that's based on the remote side's allocation parameters.
I don't think this series prevents the introduction of such isolation, but it would render this code unnecessary.
Regards,
Bjorn