thunderbolt: USB4STREAM improvements
Subsystems (2)
cover thunderbolt: USB4STREAM improvements
Hi all,
This series has a couple of non-critical fixes for USB4STREAM driver and then makes it work better with io_uring. In addition to that we add a busy polling mode that can be used by applications that need lower latency than what is possible with the interrupt mode and worker scheduling.
Mika Westerberg (5):
thunderbolt: stream: Restore consumer if copying from iter fails
thunderbolt: stream: Fix possible short reads/writes
thunderbolt: stream: Support IOCB_NOWAIT in non-blocking I/O as well
thunderbolt: Make interrupt optional for rings
thunderbolt: stream: Add support for busy polling
.../ABI/testing/configfs-thunderbolt_stream | 15 + drivers/thunderbolt/nhi.c | 27 +- drivers/thunderbolt/stream.c | 256 ++++++++++++++---- include/linux/thunderbolt.h | 2 + 4 files changed, 243 insertions(+), 57 deletions(-)
Signature · 1 line
2.50.1
PATCH 1/5 thunderbolt: stream: Restore consumer if copying from iter fails
In tbstream_dev_alloc_tx() if copying data from iterator fails we leave the consumer pointer as is wasting one entry in the ring. Fix this by restoring the consumer back in case of failure.
--- drivers/thunderbolt/stream.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/thunderbolt/stream.c +3 -1
@@ -512,8 +512,10 @@ tbstream_dev_alloc_tx(struct tbstream_dev *sdev, enum tbstream_frame_pdf pdf,dma_sync_single_for_cpu (dma_dev, sf->frame.buffer_phy, size,DMA_TO_DEVICE );if (pdf ==TBSTREAM_DATA ) { -if (copy_page_from_iter (sf->page,0 , size, from) != size) +if (copy_page_from_iter (sf->page,0 , size, from) != size) { + sdev->tx_ring.cons--;return ERR_PTR (-EFAULT ); + } }else {memset (page_address (sf->page),0 , size); }
Signature · 1 line
2.50.1
PATCH 2/5 thunderbolt: stream: Fix possible short reads/writes
In reply to Mika WesterbergSince copy_page_{to|from}_iter() advances the iterator and makes iov_iter_count() reflect the remaining bytes, subtracting nbytes from it makes it count it twice resulting in possible short reads/writes on a read/write spanning multiple frames.
Fix this by using iov_iter_count() directly.
--- drivers/thunderbolt/stream.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
drivers/thunderbolt/stream.c +4 -4
@@ -673,7 +673,7 @@ tbstream_dev_fops_read_iter(struct kiocb *kiocb, struct iov_iter *to) } nbytes =0 ; -while (nbytes <iov_iter_count (to)) { +while (iov_iter_count (to)) {struct tbstream_frame *sf;size_t size, sf_size;
@@ -695,7 +695,7 @@ tbstream_dev_fops_read_iter(struct kiocb *kiocb, struct iov_iter *to) } sf_size =tb_ring_frame_size (&sf->frame); - size =min (iov_iter_count (to) - nbytes, sf_size); + size =min (iov_iter_count (to), sf_size);if (copy_page_to_iter (sf->page, sf->offset, size, to) != size) { ret = -EFAULT ;
@@ -765,10 +765,10 @@ tbstream_dev_fops_write_iter(struct kiocb *kiocb, struct iov_iter *from) } nbytes =0 ; -while (nbytes <iov_iter_count (from)) { +while (iov_iter_count (from)) {size_t size; - size =min (iov_iter_count (from) - nbytes,TB_MAX_FRAME_SIZE ); + size =min (iov_iter_count (from),TB_MAX_FRAME_SIZE ); ret =tbstream_dev_send_data (sdev, from, size);if (ret) {/*
Signature · 1 line
2.50.1
PATCH 3/5 thunderbolt: stream: Support IOCB_NOWAIT in non-blocking I/O as well
In reply to Mika WesterbergFor read_iter/write_iter() it is also possible to pass IOCB_NOWAIT with the kiocb to indicate non-blocking read/write. For instance io_uring does this. So take this into account on read and write paths.
--- drivers/thunderbolt/stream.c | 40 +++++++++++++++++++++++++++--------- 1 file changed, 30 insertions(+), 10 deletions(-)
drivers/thunderbolt/stream.c +30 -10
@@ -633,10 +633,23 @@ static void tbstream_dev_stop(struct tbstream_dev *sdev) sdev->tx_ring.ring =NULL ; } +static int tbstream_dev_lock (struct tbstream_dev *sdev,bool nowait) +{ +if (nowait) { +if (!mutex_trylock (&sdev->lock)) +return -EAGAIN ; + }else { +if (mutex_lock_interruptible (&sdev->lock)) +return -ERESTARTSYS ; + } +return 0 ; +} +static ssize_t tbstream_dev_fops_read_iter (struct kiocb *kiocb,struct iov_iter *to) {struct file *file = kiocb->ki_filp; +bool nowait = file->f_flags &O_NONBLOCK || kiocb->ki_flags &IOCB_NOWAIT ;struct tbstream_dev *sdev =to_tbstream_dev (file->private_data);size_t nbytes;int ret;
@@ -645,14 +658,16 @@ tbstream_dev_fops_read_iter(struct kiocb *kiocb, struct iov_iter *to)if (ret)return ret; -if (mutex_lock_interruptible (&sdev->lock)) -return -ERESTARTSYS ; + ret =tbstream_dev_lock (sdev, nowait); +if (ret) +return ret;while (!tbstream_ring_available (&sdev->rx_ring)) {mutex_unlock (&sdev->lock); -if (file->f_flags &O_NONBLOCK ) +if (nowait)return -EAGAIN ; + ret = wait_event_interruptible(sdev->wait, tbstream_ring_available(&sdev->rx_ring) || tbstream_dev_valid(sdev) != 0 ||
@@ -668,8 +683,9 @@ tbstream_dev_fops_read_iter(struct kiocb *kiocb, struct iov_iter *to)if (tbstream_dev_closed (sdev) ||tbstream_dev_removed (sdev))return 0 ; -if (mutex_lock_interruptible (&sdev->lock)) -return -ERESTARTSYS ; + ret =tbstream_dev_lock (sdev, nowait); +if (ret) +return ret; } nbytes =0 ;
@@ -729,6 +745,7 @@ static ssize_t tbstream_dev_fops_write_iter(struct kiocb *kiocb, struct iov_iter *from) { struct file *file = kiocb->ki_filp; +bool nowait = file->f_flags &O_NONBLOCK || kiocb->ki_flags &IOCB_NOWAIT ;struct tbstream_dev *sdev =to_tbstream_dev (file->private_data);size_t nbytes;int ret;
@@ -737,14 +754,16 @@ tbstream_dev_fops_write_iter(struct kiocb *kiocb, struct iov_iter *from)if (ret)return ret; -if (mutex_lock_interruptible (&sdev->lock)) -return -ERESTARTSYS ; + ret =tbstream_dev_lock (sdev, nowait); +if (ret) +return ret;while (!tbstream_ring_available (&sdev->tx_ring)) {mutex_unlock (&sdev->lock); -if (file->f_flags &O_NONBLOCK ) +if (nowait)return -EAGAIN ; + ret = wait_event_interruptible(sdev->wait, tbstream_ring_available(&sdev->tx_ring) || tbstream_dev_valid(sdev) != 0 ||
@@ -760,8 +779,9 @@ tbstream_dev_fops_write_iter(struct kiocb *kiocb, struct iov_iter *from)if (tbstream_dev_closed (sdev) ||tbstream_dev_removed (sdev))return -ENXIO ; -if (mutex_lock_interruptible (&sdev->lock)) -return -ERESTARTSYS ; + ret =tbstream_dev_lock (sdev, nowait); +if (ret) +return ret; } nbytes =0 ;
Signature · 1 line
2.50.1
PATCH 4/5 thunderbolt: Make interrupt optional for rings
In reply to Mika WesterbergFor some use-cases it does make sense to poll the rings directly instead of relying on the interrupt. For this reason add a new flag RING_FLAG_NO_INTERRUPT that can be used to allocate ring in polled mode.
--- drivers/thunderbolt/nhi.c | 27 +++++++++++++++++++++------ include/linux/thunderbolt.h | 2 ++ 2 files changed, 23 insertions(+), 6 deletions(-)
drivers/thunderbolt/nhi.c +21 -6
@@ -235,6 +235,12 @@ static void ring_write_descriptors(struct tb_ring *ring) {struct ring_frame *frame, *n;struct ring_desc *descriptor; +u32 flags; + + flags =RING_DESC_POSTED ; +if (!(ring->flags &RING_FLAG_NO_INTERRUPT )) + flags |=RING_DESC_INTERRUPT ; +list_for_each_entry_safe (frame, n, &ring->queue, list) {if (ring_full (ring))break ;
@@ -242,7 +248,7 @@ static void ring_write_descriptors(struct tb_ring *ring) descriptor = &ring->descriptors[ring->head]; descriptor->phys = frame->buffer_phy; descriptor->time =0 ; - descriptor->flags =RING_DESC_POSTED |RING_DESC_INTERRUPT ; + descriptor->flags = flags;if (ring->is_tx) { descriptor->length = frame->size; descriptor->eof = frame->eof;
@@ -339,8 +345,9 @@ EXPORT_SYMBOL_GPL(__tb_ring_enqueue);* @ring: Ring to poll * * This function can be called when @start_poll callback of the @ring -* has been called. It will read one completed frame from the ring and -* return it to the caller. +* has been called or the ring is created with %RING_FLAG_NO_INTERRUPT. +* It will read one completed frame from the ring and return it to the +* caller. * * Return: Pointer to &struct ring_frame, %NULL if there is no more * completed frames.
@@ -538,6 +545,12 @@ static struct tb_ring *tb_ring_alloc(struct tb_nhi *nhi, u32 hop, int size,dev_dbg (nhi->dev,"allocating %s ring %d of size %d\n" , transmit ?"TX" :"RX" , hop, size); +if ((flags &RING_FLAG_NO_INTERRUPT ) && start_poll) { +dev_WARN (nhi->dev, +"start_poll() and NO_INTERRUPT cannot be used at the same time\n" ); +return NULL ; + } + ring =kzalloc_obj (*ring);if (!ring)return NULL ;
@@ -568,7 +581,7 @@ static struct tb_ring *tb_ring_alloc(struct tb_nhi *nhi, u32 hop, int size,if (!ring->descriptors) goto err_free_ring; -if (nhi->ops->request_ring_irq) { +if (!(flags &RING_FLAG_NO_INTERRUPT ) && nhi->ops->request_ring_irq) {if (nhi->ops->request_ring_irq (ring, flags &RING_FLAG_NO_SUSPEND )) goto err_free_descs; }
@@ -701,7 +714,8 @@ void tb_ring_start(struct tb_ring *ring)ring_iowrite32options (ring, flags,0 ); } -ring_interrupt_active (ring, true); +if (!(ring->flags &RING_FLAG_NO_INTERRUPT )) +ring_interrupt_active (ring, true); ring->running = true; err:spin_unlock (&ring->lock);
@@ -761,7 +775,8 @@ void tb_ring_stop(struct tb_ring *ring) RING_TYPE(ring), ring->hop); goto err; } -ring_interrupt_active (ring, false); +if (!(ring->flags &RING_FLAG_NO_INTERRUPT )) +ring_interrupt_active (ring, false);ring_iowrite32options (ring,0 ,0 );ring_iowrite64desc (ring,0 ,0 );
include/linux/thunderbolt.h +2 -0
@@ -592,6 +592,8 @@ struct tb_ring {#define RING_FLAG_FRAME BIT(1)/* Enable end-to-end flow control */ #define RING_FLAG_E2E BIT(2) +/* Do not enable interrupt for the ring */ +#define RING_FLAG_NO_INTERRUPT BIT(3)struct ring_frame ;typedef void (*ring_cb )(struct tb_ring *,struct ring_frame *,bool canceled);
Signature · 1 line
2.50.1
PATCH 5/5 thunderbolt: stream: Add support for busy polling
In reply to Mika WesterbergUsing interrupts and scheduling workers increase latency so latency critical applications may want to avoid that. Make this possible in USB4STREAM by adding a new ConfigFS attribute: busy_poll that, when activated switches the rings to polling mode. The cost for lower latency is that this burns more CPU cycles and things like poll(2) cannot be used.
--- .../ABI/testing/configfs-thunderbolt_stream | 15 ++ drivers/thunderbolt/stream.c | 212 ++++++++++++++---- 2 files changed, 187 insertions(+), 40 deletions(-)
Documentation/ABI/testing/configfs-thunderbolt_stream +15 -0
@@ -27,6 +27,21 @@ Description: default values. If there is an advertised remote stream with the same name, uses its values as the default. +What: /sys/kernel/config/thunderbolt/stream/<xdomain>.<service>/$name/busy_poll +Date: Nov 2026 +KernelVersion: v7.3 +Contact: Mika Westerberg <[email protected]> +Description: + Instead of using interrupts for completing the frames in + the TX/RX rings, busy poll them directly from the + read(2) and write(2) calls. This burns more CPU cycles + but provides lower latency for applications that need it. + + This also makes poll(2) return EPOLLERR because + interrupts do not provide wakeup anymore. Likewise a + blocking read(2) without available data busy-spins until + data arrives or a signal is received. + What: /sys/kernel/config/thunderbolt/stream/<xdomain>.<service>/$name/index Date: Sep 2026 KernelVersion: v7.2
drivers/thunderbolt/stream.c +172 -40
316 lines not on this page · raw message
Signature · 1 line
2.50.1
Re: [PATCH 5/5] thunderbolt: stream: Add support for busy polling
Quoted 6 lines · Mika Westerberg
Using interrupts and scheduling workers increase latency so latency critical applications may want to avoid that. Make this possible in USB4STREAM by adding a new ConfigFS attribute:
busy_pollthat, when activated switches the rings to polling mode. The cost for lower latency is that this burns more CPU cycles and things like poll(2) cannot be used.
Hm, would it be possible to auto-detect whether polling is more performant for the current workload and automatically enable and disable it? E.g. by counting the number of interrupts in a given interval and enabling/disabling polling based on a high water mark / low water mark? Those marks could then be finetuned from user space, but moving in and out of polling automatically would be more desirable I think. Forcing polling could be achieved by setting the water marks to a special value, e.g. 0 or UINT_MAX.
Thanks,
Lukas
Re: [PATCH 5/5] thunderbolt: stream: Add support for busy polling
Hi,
Quoted 8 lines · Lukas Wunner
Quoted 6 lines · Mika Westerberg
Using interrupts and scheduling workers increase latency so latency critical applications may want to avoid that. Make this possible in USB4STREAM by adding a new ConfigFS attribute:
busy_pollthat, when activated switches the rings to polling mode. The cost for lower latency is that this burns more CPU cycles and things like poll(2) cannot be used.
Quoted 8 lines
Hm, would it be possible to auto-detect whether polling is more performant for the current workload and automatically enable and disable it? E.g. by counting the number of interrupts in a given interval and enabling/disabling polling based on a high water mark / low water mark? Those marks could then be finetuned from user space, but moving in and out of polling automatically would be more desirable I think. Forcing polling could be achieved by setting the water marks to a special value, e.g. 0 or
UINT_MAX.
I think the applications that want the minimal latency and end up setting this will put it to the "minimal" anyway (because they don't tolerate any additional latencies caused by interrupts and the worker scheduling) and for the rest they can already tune this by setting non-default values to "throttling" attribute (eg. 0 to avoid it completely, makes latency lower while still keeping it from hogging the CPU fully).
We can introduce auto-tuning mode afterwards as you suggest if it turns out to be wanted feature.
Quoted 5 lines · Mika Westerberg
thunderbolt: stream: Restore consumer if copying from iter fails
thunderbolt: stream: Fix possible short reads/writes
thunderbolt: stream: SupportIOCB_NOWAITin non-blocking I/O as well
thunderbolt: Make interrupt optional for rings
thunderbolt: stream: Add support for busy polling
All applied to thunderbolt.git/next.