usb: musb: gadget: don't flush a waiting RX packet when queueing a request
Subsystems (2)
PATCH v2 1/1 usb: musb: gadget: don't flush a waiting RX packet when queueing a request
Since commit 3fc137386c46 ("usb: musb: Fix hardware lockup on first Rx endpoint request"), musb_ep_restart() flushes the RX FIFO instead of calling rxstate(), with the premise that in case of Ethernet, dropping a couple of packets during gadget initialization is not an issue, because it will only happen when the software is not yet ready.
But when using other functions with musb, for example, f_mass_storage or f_acm, this code path will be used not only when the software is not yet ready, but also mid-communication, and it will cause data loss. Data loss happens on gadgets that don't always have requests queued for an ep.
The timeline of a data loss:
1. A function that does not have a lot of requests pre-queued completes an OUT request. The queue is empty.
2. The next packet comes. MUSB auto-acks it, places it into the FIFO, sets RXPKTRDY, raises an interrupt.
3. musb_g_rx() is called, sees the request queue is empty, early returns.
4. The function queues the next request (f_mass_storage does it from the main fsg thread), musb_ep_restart does FLUSHFIFO, packet is lost.
5. The packet was auto-acked by MUSB, so it is not resent by the host.
Next data chunk arrives with an IRQ like so:
musb_g_rx returns, queue empty
(drivers/usb/musb/musb_gadget.c:810)
musb_g_rx from musb_interrupt
(drivers/usb/musb/musb_core.c:1786)
musb_interrupt from dsps_interrupt
(drivers/usb/musb/musb_dsps.c:389)and then it is dropped from the FIFO when usb_ep_queue() is called:
MUSB_RXCSR_FLUSHFIFO from musb_ep_restart
(drivers/usb/musb/musb_gadget.c:1174)
musb_ep_restart from musb_ep_restart_resume_work
(drivers/usb/musb/musb_gadget.c:1184)
musb_ep_restart_resume_work from musb_queue_resume_work
(drivers/usb/musb/musb_core.c:2277)
musb_queue_resume_work from musb_gadget_queue
(drivers/usb/musb/musb_gadget.c:1249)
musb_gadget_queue from usb_ep_queue
(drivers/usb/gadget/udc/core.c:306)In case of f_mass_storage, it keeps waiting for the lost data; the host then times out and resets the device. Since this is a race condition, this can be seen as data transfers taking too long, or as drives getting mounted but disappearing when data is read/written, and so on.
usb 1-8.4: reset high-speed USB device number 31 using xhci_hcd
I/O error, dev sdb, sector 240 op 0x1:(WRITE) flags 0x804800 phys_seg 30 prio class 2 Buffer I/O error on dev sdb, logical block 30, lost async page write
When testing f_acm, this is seen as byte loss on the tty. f_hid, f_printer and f_fs lose data similarly.
Testing with commit 3fc137386c46 ("usb: musb: Fix hardware lockup on first Rx endpoint request") reverted fixed these symptoms.
Move the data processing to system_bh_wq. Purpose of doing so is to honor the usb_ep_queue docstring and to run in an interrupt context, avoiding the lockup Hubert described in his commit message. Alternative way of looking at this is it "re-triggers" the interrupt, running the musb_g_rx that had to be skipped.
Such a change makes a different race condition more visible. On SMP or PREEMPT_RT, the RX interrupt can come while musb_g_giveback() is running on the same endpoint and has already dropped musb->lock; before this patch, that would be possible only when usb_ep_dequeue() runs in parallel with an interrupt, but after this patch,
it could be the workqueue item running when an interrupt comes. Handling it at that moment could lead to packets getting stuck never matched with their requests (locking up the device essentially) or requests completing but out of order.
So in musb_g_rx() and musb_gadget_queue(), also defer the RX'd data to a work item if ep->busy is set. The deferred work items are scheduled when the last musb_g_giveback() finishes and busy drops to 0.
Do not do any deferral of DMA completion. When musb_g_rx() is called with DMAENAB set, it means it's called from musb_dma_completion(). A DMA completion could not be happening at the same time as a workqueue non-DMA giveback (because this DMA needed to be programmed first, and programming of a DMA is only possible when all non-DMA requests ahead in the request queue already got their givebacks). And work items of DMA requests don't do givebacks. Deferring DMA completion is more work, so just don't change the behavior there.
DMA completion can still race with usb_ep_dequeue(), that's the same as without this patch. So DMA situation should not become any worse.
The issue was reproduced and the patch was tested on a BeagleBone Black (AM335x, one core, no RX DMA) with f_mass_storage, f_acm, f_hid, f_printer and f_fs. It was also verified OK before and after this patch with f_ecm and f_ncm.
This issue was also reproduced and the patch was tested on a Pine A64+ (Allwinner A64, four cores, no DMA) with both PREEMPT_DYNAMIC and PREEMPT_RT, with lockdep enabled, with f_mass_storage, f_acm, f_hid, f_printer and f_fs.
This patch was also tested with suspend to RAM on the Pine A64+, both idle and during a transfer.
---
v1 -> v2:
- musb_g_rx_kick() does not re-queue itself when busy is set. Instead, musb_g_giveback() re-queues it when rx_kick_deferred. This new flag is set in three cases:
1) busy is set inside musb_g_rx_kick()
2) musb_gadget_queue() adds an RX request to an empty request queue and busy is set
3) busy is set in the beginning of musb_g_rx() (addresses sashiko's "RX endpoint lockup after giveback in musb_g_rx_kick()", "softirq busy-loop and livelock in musb_g_rx_kick()", and "race and ordering bug in musb_gadget_disable() for rx_kick")
- musb_g_rx_kick checks musb->is_runtime_suspended now and exits early if the controller is suspended, and the work items are disabled during system suspend (addresses sashiko's
"unushed rx_kick work in musb_suspend() accesses disabled MMIO")
- This time tested also on a Pine A64+ (Allwinner A64), which has both MUSB and 4 cores.
v1: https://lore.kernel.org/linux-usb/[email protected]/
drivers/usb/musb/musb_core.c | 3 + drivers/usb/musb/musb_gadget.c | 107 ++++++++++++++++++++++++++++++++- drivers/usb/musb/musb_gadget.h | 9 +++ 3 files changed, 116 insertions(+), 3 deletions(-)
drivers/usb/musb/musb_core.c +3 -0
@@ -2818,6 +2818,8 @@ static int musb_suspend(struct device *dev) ; musb->flush_irq_work = false; +musb_gadget_disable_rx_kicks (musb); +if (!(musb->ops->quirks &MUSB_PRESERVE_SESSION ))musb_writeb (musb->mregs,MUSB_DEVCTL ,0 );
@@ -2866,6 +2868,7 @@ static int musb_resume(struct device *dev)if ((devctl & mask) != (musb->context.devctl & mask)) musb->port1_status =0 ; +musb_gadget_enable_rx_kicks (musb);musb_enable_interrupts (musb);musb_platform_enable (musb);
drivers/usb/musb/musb_gadget.c +104 -3
150 lines not on this page · raw message
drivers/usb/musb/musb_gadget.h +9 -0
@@ -11,6 +11,7 @@#define __MUSB_GADGET_H#include <linux/list.h> +#include <linux/workqueue.h> #if IS_ENABLED (CONFIG_USB_MUSB_GADGET ) ||IS_ENABLED (CONFIG_USB_MUSB_DUAL_ROLE )extern irqreturn_t musb_g_ep0_irq (struct musb *);
@@ -23,6 +24,8 @@ extern void musb_g_wakeup(struct musb *);extern void musb_g_disconnect (struct musb *);extern void musb_gadget_cleanup (struct musb *);extern int musb_gadget_setup (struct musb *); +void musb_gadget_disable_rx_kicks (struct musb *musb); +void musb_gadget_enable_rx_kicks (struct musb *musb);#else static inline irqreturn_t musb_g_ep0_irq (struct musb *musb)
@@ -42,6 +45,9 @@ static inline int musb_gadget_setup(struct musb *musb) {return 0 ; } + +static inline void musb_gadget_disable_rx_kicks (struct musb *musb) {} +static inline void musb_gadget_enable_rx_kicks (struct musb *musb) {}#endif enum buffer_map_state {
@@ -93,6 +99,9 @@ struct musb_ep {/* true if lock must be dropped but req_list may not be advanced */ u8 busy; +struct work_struct rx_kick; +bool rx_kick_deferred; +u8 hb_mult; };
base-commit: c339f4010dea8b0d754f84ace70cd1b089878ef8
Signature · 1 line
2.53.0
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 3 potential issues to consider: - [High] usb: musb: gadget: overlapping musb_g_giveback() calls wedge ep->busy - [High] usb: musb: gadget…
Thank you for your contribution! Sashiko AI review found 3 potential issues to consider:
- [High] usb: musb: gadget: overlapping musb_g_giveback() calls wedge ep->busy
- [High] usb: musb: gadget: race and ordering bug in musb_gadget_disable() for rx_kick
- [Medium] usb: musb: gadget: lost rx_kick across system suspend and resume
--
commit cef9b9e2278bf856fd53af421c899c0ae67866b7
Author: Elvira Khabirova <[email protected]>
usb: musb: gadget: don't flush a waiting RX packet when queueing a request
This commit replaces the RX FIFO flush in musb_ep_restart() with a system_bh_wq work item that invokes musb_g_rx(), and defers RX processing via ep->rx_kick_deferred while ep->busy is set.
Quoted 5 lines
drivers/usb/musb/musb_gadget.c +0
-0
drivers/usb/musb/musb_gadget.c __acquires(ep->musb->lock) · line 157
Quoted 10 lines
drivers/usb/musb/musb_gadget.c +7
-0@@ -148,6 +148,21 @@ __acquires(ep->musb->lock)usb_gadget_giveback_request (&req->ep->end_point, &req->request);spin_lock (&musb->lock); ep->busy = busy; + +if (!ep->busy && ep->rx_kick_deferred) { + ep->rx_kick_deferred = false; +if (!queue_work (system_bh_wq, &ep->rx_kick)) +pm_runtime_put_noidle (musb->controller); + } +}
[Severity: High]
Can overlapping musb_g_giveback() calls finish out of LIFO order and leave ep->busy permanently stuck at 1?
musb_g_giveback() saves ep->busy in a stack variable, sets ep->busy = 1, and drops musb->lock around usb_gadget_giveback_request(). With RX givebacks running in system_bh_wq, a second giveback on the same endpoint (for example via musb_gadget_dequeue(), which enters musb_g_giveback() when musb_ep->busy is set) can overlap on another CPU:
CPU0 CPU1musb_g_rx_kick()
musb_g_rx()
musb_g_giveback()
busy = ep->busy; // 0
ep->busy = 1;
spin_unlock(&musb->lock)
usb_gadget_giveback_request()
musb_gadget_dequeue()
spin_lock_irqsave(&musb->lock, ...)
musb_g_giveback()
busy = ep->busy; // 1
ep->busy = 1;
spin_unlock(&musb->lock)
usb_gadget_giveback_request()
spin_lock(&musb->lock)
ep->busy = 0;
queue_work(..., &ep->rx_kick)
spin_unlock(&musb->lock)
spin_lock(&musb->lock)
ep->busy = 1; <-- stuck at 1First, CPU0 clears ep->busy to 0 and schedules ep->rx_kick while CPU1 is still in usb_gadget_giveback_request(). Second, when CPU1 finishes, it writes its saved busy = 1 back to ep->busy. After that, every subsequent musb_g_rx(), musb_g_rx_kick(), and musb_gadget_queue() call sees ep->busy == 1 and defers work forever, wedging the endpoint and leaking the runtime PM reference taken by musb_g_defer_rx_kick().
Making ep->busy an active giveback counter (incremented before dropping musb->lock and decremented after re-acquiring it) would ensure ep->busy only reaches 0 when the last giveback completes.
[ ... ]
drivers/usb/musb/musb_gadget.c static int musb_gadget_disable(struct usb_ep *ep) · line 1148
Quoted 13 lines
drivers/usb/musb/musb_gadget.c +6
-0@@ -1111,6 +1137,12 @@ static int musb_gadget_disable(struct usb_ep *ep)/* abort all pending DMA and requests */ nuke (musb_ep, -ESHUTDOWN ); +if (cancel_work (&musb_ep->rx_kick)) +pm_runtime_put_autosuspend (musb->controller); +if (musb_ep->rx_kick_deferred) { + musb_ep->rx_kick_deferred = false; +pm_runtime_put_autosuspend (musb->controller); + } musb_ep->desc =NULL ; musb_ep->end_point.desc =NULL ;
[Severity: High]
Can musb_g_rx_kick() run while nuke() drops musb->lock or after the endpoint is re-enabled?
Because musb_ep->desc is cleared after nuke(), musb->lock is dropped inside nuke() -> musb_g_giveback() while musb_ep->desc is still non-NULL. If an earlier giveback on another CPU finishes during that window, it restores ep->busy = 0 and queues ep->rx_kick, allowing musb_g_rx_kick() -> musb_g_rx() -> rxstate() to run after nuke() has already freed musb_ep->dma.
In addition, cancel_work() is non-synchronous:
CPU0 CPU1
<softirq>
musb_g_rx_kick()
// waiting for musb->lockmusb_gadget_disable()
spin_lock_irqsave(&musb->lock, ...)
cancel_work(&musb_ep->rx_kick) // false
musb_ep->desc = NULL;
spin_unlock_irqrestore(&musb->lock, ...)musb_gadget_enable()
spin_lock_irqsave(&musb->lock, ...)
musb_ep->desc = desc;
musb_ep->busy = 0;
spin_unlock_irqrestore(&musb->lock, ...)
spin_lock_irqsave(&musb->lock, ...)
if (!musb_ep->desc || ...) // false
musb_g_rx() <-- stale kick runsClearing musb_ep->desc and musb_ep->end_point.desc before calling nuke() prevents musb_g_rx_kick() and musb_gadget_queue() from acting on the endpoint while nuke() drops musb->lock, and prevented stale kicks will see the updated state if musb_gadget_enable() also cancels any pending rx_kick.
[ ... ]
Quoted 25 lines
drivers/usb/musb/musb_gadget.c +21
-0@@ -1810,6 +1881,36 @@ void musb_gadget_cleanup(struct musb *musb)cancel_delayed_work_sync (&musb->gadget_work);usb_del_gadget_udc (&musb->g); + +musb_gadget_disable_rx_kicks (musb); +} + +static void musb_g_disable_rx_kick (struct musb *musb,struct musb_ep *musb_ep) +{ +if (musb_ep->rx_kick.func &&disable_work_sync (&musb_ep->rx_kick)) +pm_runtime_put_noidle (musb->controller); +} + +void musb_gadget_disable_rx_kicks (struct musb *musb) +{ +int i; + +for (i =0 ; i < musb->nr_endpoints; i++) { +musb_g_disable_rx_kick (musb, &musb->endpoints[i].ep_in); +musb_g_disable_rx_kick (musb, &musb->endpoints[i].ep_out); + } +} + +void musb_gadget_enable_rx_kicks (struct musb *musb)
drivers/usb/musb/musb_gadget.c void musb_gadget_cleanup(struct musb *musb) · line 1914
Quoted 10 lines
drivers/usb/musb/musb_gadget.c +9
-0+{ +int i; + +for (i =0 ; i < musb->nr_endpoints; i++) { +if (musb->endpoints[i].ep_in.rx_kick.func) +enable_work (&musb->endpoints[i].ep_in.rx_kick); +if (musb->endpoints[i].ep_out.rx_kick.func) +enable_work (&musb->endpoints[i].ep_out.rx_kick); + } }
[Severity: Medium]
Does musb_gadget_disable_rx_kicks() lose a pending or deferred rx_kick across system suspend and resume?
If ep->rx_kick is pending when musb_suspend() calls disable_work_sync(), the work item is cancelled and its runtime PM reference is dropped, or if musb_g_rx_kick() is currently in musb_g_giveback() with ep->rx_kick_deferred set, disable_work_sync() marks ep->rx_kick disabled before waiting for completion so queue_work() in musb_g_giveback() fails. On resume, musb_gadget_enable_rx_kicks() only calls enable_work(), which does not re-queue the cancelled work. When the USB session is preserved across suspend/resume (such as with MUSB_PRESERVE_SESSION or standby), MUSB_RXCSR_RXPKTRDY remains set in hardware with no interrupt to re-trigger it, stalling RX on the endpoint.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Quoted 74 lines · sashiko-bot
Thank you for your contribution! Sashiko AI review found 3 potential issues to consider:
- [High] usb: musb: gadget: overlappingmusb_g_giveback()calls wedgeep->busy
- [High] usb: musb: gadget: race and ordering bug inmusb_gadget_disable()forrx_kick
- [Medium] usb: musb: gadget: lostrx_kickacross system suspend and resume
--commit
cef9b9e2278bf856fd53af421c899c0ae67866b7
Author: Elvira Khabirova <[email protected]>usb: musb: gadget: don't flush a waiting RX packet when queueing a request
This commit replaces the RX FIFO flush in
musb_ep_restart()with asystem_bh_wqwork item that invokesmusb_g_rx(), and defers RX processing viaep->rx_kick_deferredwhileep->busyis set.Quoted 5 lines
drivers/usb/musb/musb_gadget.c +0
-0drivers/usb/musb/musb_gadget.c __acquires(ep->musb->lock) · line 157
Quoted 10 lines
drivers/usb/musb/musb_gadget.c +7
-0@@ -148,6 +148,21 @@ __acquires(ep->musb->lock)usb_gadget_giveback_request (&req->ep->end_point, &req->request);spin_lock (&musb->lock); ep->busy = busy; + +if (!ep->busy && ep->rx_kick_deferred) { + ep->rx_kick_deferred = false; +if (!queue_work (system_bh_wq, &ep->rx_kick)) +pm_runtime_put_noidle (musb->controller); + } +}[Severity: High]
Can overlappingmusb_g_giveback()calls finish out of LIFO order and leaveep->busypermanently stuck at 1?
musb_g_giveback()savesep->busyin a stack variable, setsep->busy= 1, and dropsmusb->lockaroundusb_gadget_giveback_request(). With RX givebacks running insystem_bh_wq, a second giveback on the same endpoint (for example viamusb_gadget_dequeue(), which entersmusb_g_giveback()whenmusb_ep->busyis set) can overlap on another CPU:CPU0 CPU1
musb_g_rx_kick()
musb_g_rx()musb_g_giveback() busy = ep->busy; // 0 ep->busy = 1; spin_unlock(&musb->lock) usb_gadget_giveback_request() musb_gadget_dequeue() spin_lock_irqsave(&musb->lock, ...) musb_g_giveback() busy = ep->busy; // 1 ep->busy = 1; spin_unlock(&musb->lock) usb_gadget_giveback_request() spin_lock(&musb->lock) ep->busy = 0; queue_work(..., &ep->rx_kick) spin_unlock(&musb->lock) spin_lock(&musb->lock) ep->busy = 1; <-- stuck at 1First, CPU0 clears
ep->busyto 0 and schedulesep->rx_kickwhile CPU1 is still inusb_gadget_giveback_request(). Second, when CPU1 finishes, it writes its saved busy = 1 back toep->busy. After that, every subsequentmusb_g_rx(),musb_g_rx_kick(), andmusb_gadget_queue()call seesep->busy== 1 and defers work forever, wedging the endpoint and leaking the runtime PM reference taken bymusb_g_defer_rx_kick().
Quoted 3 lines
Making
ep->busyan active giveback counter (incremented before droppingmusb->lockand decremented after re-acquiring it) would ensureep->busyonly reaches 0 when the last giveback completes.
Yes, that is possible, but this is a pre-existing bug. I even have a separate patch prepared to make busy a counter instead of a flag. This patch does not aggravate this problem.
The same race is possible between a dequeue and a HW IRQ.
Quoted 18 lines
drivers/usb/musb/musb_gadget.c static int musb_gadget_disable(struct usb_ep *ep) · line 1148
Quoted 13 lines
drivers/usb/musb/musb_gadget.c +6
-0@@ -1111,6 +1137,12 @@ static int musb_gadget_disable(struct usb_ep *ep)/* abort all pending DMA and requests */ nuke (musb_ep, -ESHUTDOWN ); +if (cancel_work (&musb_ep->rx_kick)) +pm_runtime_put_autosuspend (musb->controller); +if (musb_ep->rx_kick_deferred) { + musb_ep->rx_kick_deferred = false; +pm_runtime_put_autosuspend (musb->controller); + } musb_ep->desc =NULL ; musb_ep->end_point.desc =NULL ;[Severity: High]
Canmusb_g_rx_kick()run whilenuke()dropsmusb->lockor after the endpoint is re-enabled?
Quoted 6 lines
Because
musb_ep->descis cleared afternuke(),musb->lockis dropped insidenuke()->musb_g_giveback()whilemusb_ep->descis still non-NULL. If an earlier giveback on another CPU finishes during that window, it restoresep->busy= 0 and queuesep->rx_kick, allowingmusb_g_rx_kick()->musb_g_rx()->rxstate()to run afternuke()has already freedmusb_ep->dma.
Good catch, and actually I think this can be resolved with making ep->busy a counter instead.
I will send them together as a series for v3.
Quoted 20 lines
In addition,
cancel_work()is non-synchronous:CPU0 CPU1
<softirq> musb_g_rx_kick() // waiting for musb->lock
musb_gadget_disable()
spin_lock_irqsave(&musb->lock, ...)
cancel_work(&musb_ep->rx_kick)// false
musb_ep->desc= NULL;
spin_unlock_irqrestore(&musb->lock, ...)musb_gadget_enable()
spin_lock_irqsave(&musb->lock, ...)
musb_ep->desc= desc;
musb_ep->busy= 0;
spin_unlock_irqrestore(&musb->lock, ...)spin_lock_irqsave(&musb->lock, ...) if (!musb_ep->desc || ...) // false musb_g_rx() <-- stale kick runs
Quoted 5 lines
Clearing
musb_ep->descandmusb_ep->end_point.descbefore callingnuke()preventsmusb_g_rx_kick()andmusb_gadget_queue()from acting on the endpoint whilenuke()dropsmusb->lock, and prevented stale kicks will see the updated state ifmusb_gadget_enable()also cancels any pendingrx_kick.
Is this any different from before the patch, with a HW IRQ? I think a stale kick handles everything well, if there is no work, musb_g_rx() just returns.
Quoted 40 lines
Quoted 25 lines
drivers/usb/musb/musb_gadget.c +21
-0@@ -1810,6 +1881,36 @@ void musb_gadget_cleanup(struct musb *musb)cancel_delayed_work_sync (&musb->gadget_work);usb_del_gadget_udc (&musb->g); + +musb_gadget_disable_rx_kicks (musb); +} + +static void musb_g_disable_rx_kick (struct musb *musb,struct musb_ep *musb_ep) +{ +if (musb_ep->rx_kick.func &&disable_work_sync (&musb_ep->rx_kick)) +pm_runtime_put_noidle (musb->controller); +} + +void musb_gadget_disable_rx_kicks (struct musb *musb) +{ +int i; + +for (i =0 ; i < musb->nr_endpoints; i++) { +musb_g_disable_rx_kick (musb, &musb->endpoints[i].ep_in); +musb_g_disable_rx_kick (musb, &musb->endpoints[i].ep_out); + } +} + +void musb_gadget_enable_rx_kicks (struct musb *musb)drivers/usb/musb/musb_gadget.c void musb_gadget_cleanup(struct musb *musb) · line 1914
Quoted 10 lines
drivers/usb/musb/musb_gadget.c +9
-0+{ +int i; + +for (i =0 ; i < musb->nr_endpoints; i++) { +if (musb->endpoints[i].ep_in.rx_kick.func) +enable_work (&musb->endpoints[i].ep_in.rx_kick); +if (musb->endpoints[i].ep_out.rx_kick.func) +enable_work (&musb->endpoints[i].ep_out.rx_kick); + } }[Severity: Medium]
Doesmusb_gadget_disable_rx_kicks()lose a pending or deferredrx_kickacross system suspend and resume?
Quoted 10 lines
If
ep->rx_kickis pending whenmusb_suspend()callsdisable_work_sync(), the work item is cancelled and its runtime PM reference is dropped, or ifmusb_g_rx_kick()is currently inmusb_g_giveback()withep->rx_kick_deferredset,disable_work_sync()marksep->rx_kickdisabled before waiting for completion soqueue_work()inmusb_g_giveback()fails. On resume,musb_gadget_enable_rx_kicks()only callsenable_work(), which does not re-queue the cancelled work. When the USB session is preserved across suspend/resume (such as withMUSB_PRESERVE_SESSIONor standby),MUSB_RXCSR_RXPKTRDYremains set in hardware with no interrupt to re-trigger it, stalling RX on the endpoint.
I see, I wish I had more hardware to test on.
This seems to be possible on da8xx.
I will find a way to re-queue the cancelled work items for v3.