changeset 9029 (http://linuxtv.org/hg/v4l-dvb/rev/aa3e5cc1d833)
Hi,
this change set is wrong. The affected functions cannot be called from an interrupt context, because they may process large buffers. In this case, interrupts are disabled for a long time. Functions, like dvb_dmx_swfilter_packets(), could be called only from a tasklet. This change set does hide some strong design bugs in dm1105.c and au0828-dvb.c.
Please revert this change set and do fix the bugs in dm1105.c and au0828-dvb.c (and other files).
Regards,
Hartmut
Quoted 6 lines · On Mon, 2009-02-02 at 02:46 +0100, e9hack wrote:
Hi,
this change set is wrong. The affected functions cannot be called from an interrupt context, because they may process large buffers. In this case, interrupts are disabled for a long time. Functions, like
dvb_dmx_swfilter_packets(), could be called only from a tasklet.
I agree with Hartmut that these functions should not be called from an interrupt context.
Although for deferrable work, I thought tasklets were deprecated and that work handlers were the preferred mechanism:
http://lwn.net/Articles/23634/
Quoted 4 lines
This change set does hide some strong design bugs in
dm1105.candau0828-dvb.c.Please revert this change set and do fix the bugs in
dm1105.candau0828-dvb.c(and other files).
BTW dm1105.c does use a tasklet for IR keypresses. However, since it is only imlemented with one "struct infrared", and hence only one "ir_command", per device and not a queue, it is possible to lose button presses that happen very close together. That probably doesn't matter practically for IR button presses, but the same strategy cannot be used for TS packets.
Regards,
Andy
Quoted 2 lines
Regards,
Hartmut
Quoted 9 lines · e9hack wrote:
Hi,
this change set is wrong. The affected functions cannot be called from an interrupt context, because they may process large buffers. In this case, interrupts are disabled for a long time. Functions, like
dvb_dmx_swfilter_packets(), could be called only from a tasklet. This change set does hide some strong design bugs indm1105.candau0828-dvb.c.Please revert this change set and do fix the bugs in
dm1105.candau0828-dvb.c(and other files).
@Mauro:
This changeset _must_ be reverted! It breaks all kernels since 2.6.27 for applications which use DVB and require a low interrupt latency.
It is a very bad idea to call the demuxer to process data buffers with interrupts disabled!
FYI, a LIRC problem was reported here:
http://vdrportal.de/board/thread.php?postid=786366#post786366
and it has been verified that changeset
http://linuxtv.org/hg/v4l-dvb/rev/aa3e5cc1d833
causes the problem:
http://vdrportal.de/board/thread.php?postid=791813#post791813
Please revert this changeset immediately and submit a fix to the stable kernels >= 2.6.27.
CU
Oliver
Signature · 3 lines
---------------------------------------------------------------- VDR Remote Plugin 0.4.0: http://www.escape-edv.de/endriss/vdr/ ----------------------------------------------------------------
Quoted 10 lines · Oliver Endriss
Quoted 9 lines · e9hack wrote:
Hi,
this change set is wrong. The affected functions cannot be called from an interrupt context, because they may process large buffers. In this case, interrupts are disabled for a long time. Functions, like
dvb_dmx_swfilter_packets(), could be called only from a tasklet. This change set does hide some strong design bugs indm1105.candau0828-dvb.c.Please revert this change set and do fix the bugs in
dm1105.candau0828-dvb.c(and other files).
Does anyone have a complete list of the drivers that are the bad actors? It would be nice to have some sense of the scope of the work to fix them, and to know who might have hardware for testing.
Hartmut mentioned 2 current problem drivers. I know cx18 was a problem, but I fixed it months ago. I can agree in every driver that does things wrong, it is a design error. Fixing design errors will not be a quick fix for most drivers
Regards,
Andy
Quoted 22 lines
@Mauro:
This changeset _must_ be reverted! It breaks all kernels since 2.6.27 for applications which use DVB and require a low interrupt latency.
It is a very bad idea to call the demuxer to process data buffers with interrupts disabled!
FYI, a LIRC problem was reported here:
http://vdrportal.de/board/thread.php?postid=786366#post786366and it has been verified that changeset
http://linuxtv.org/hg/v4l-dvb/rev/aa3e5cc1d833
causes the problem:
http://vdrportal.de/board/thread.php?postid=791813#post791813Please revert this changeset immediately and submit a fix to the stable kernels >= 2.6.27.
CU
Oliver
Quoted 27 lines · Oliver Endriss
Quoted 9 lines · e9hack wrote:
Hi,
this change set is wrong. The affected functions cannot be called from an interrupt context, because they may process large buffers. In this case, interrupts are disabled for a long time. Functions, like
dvb_dmx_swfilter_packets(), could be called only from a tasklet. This change set does hide some strong design bugs indm1105.candau0828-dvb.c.Please revert this change set and do fix the bugs in
dm1105.candau0828-dvb.c(and other files).@Mauro:
This changeset _must_ be reverted! It breaks all kernels since 2.6.27 for applications which use DVB and require a low interrupt latency.
It is a very bad idea to call the demuxer to process data buffers with interrupts disabled!
FYI, a LIRC problem was reported here:
http://vdrportal.de/board/thread.php?postid=786366#post786366and it has been verified that changeset
http://linuxtv.org/hg/v4l-dvb/rev/aa3e5cc1d833
causes the problem:
http://vdrportal.de/board/thread.php?postid=791813#post791813
Quoted 5 lines
Please revert this changeset immediately and submit a fix to the stable kernels >= 2.6.27.
CU
Oliver
The patch below is an idea for a fix that uses a module parameter to give back right away the original behavior to those who need it, while buying time to fix the drivers that are doing things wrong.
I don't know if this patch will be acceptable to anyone, and I suspect there will be disagreement on the default behavior.
It compiles and comes through checkpatch.pl with only one warning about an extern declaration I didn't know where to place. This patch still needs to be checked for correctness and tested.
Regards,
Andy
linux/drivers/media/dvb/dvb-core/dmxdev.c +41 -9
@@ -35,6 +35,17 @@module_param (debug, int,0644 );MODULE_PARM_DESC (debug,"Turn on/off debugging (default:off)." ); + +/* +* FIXME - remove this conservative lock kludge, when the offending drivers +* that make some calls improperly from an interrupt context are fixed. +*/ +int dmxdev_conservative_locks; + +module_param_named (conservative_locks, dmxdev_conservative_locks, int,0644 ); +MODULE_PARM_DESC (conservative_locks, +"Work around for drivers that make calls\n" +"\t\twith interrupts disabled (default:0/off)." );#define dprintk if (debug) printk
@@ -364,16 +375,22 @@ enum dmx_success success) {struct dmxdev_filter *dmxdevfilter = filter->priv; -unsigned long flags; +unsigned long flags =0 ;int ret;if (dmxdevfilter->buffer.error) {wake_up (&dmxdevfilter->buffer.queue);return 0 ; } -spin_lock_irqsave (&dmxdevfilter->dev->lock, flags); +if (dmxdev_conservative_locks) +spin_lock_irqsave (&dmxdevfilter->dev->lock, flags); +else +spin_lock (&dmxdevfilter->dev->lock);if (dmxdevfilter->state !=DMXDEV_STATE_GO ) { -spin_unlock_irqrestore (&dmxdevfilter->dev->lock, flags); +if (dmxdev_conservative_locks) +spin_unlock_irqrestore (&dmxdevfilter->dev->lock, flags); +else +spin_unlock (&dmxdevfilter->dev->lock);return 0 ; }del_timer (&dmxdevfilter->timer);
@@ -392,7 +409,10 @@ }if (dmxdevfilter->params.sec.flags &DMX_ONESHOT ) dmxdevfilter->state =DMXDEV_STATE_DONE ; -spin_unlock_irqrestore (&dmxdevfilter->dev->lock, flags); +if (dmxdev_conservative_locks) +spin_unlock_irqrestore (&dmxdevfilter->dev->lock, flags); +else +spin_unlock (&dmxdevfilter->dev->lock);wake_up (&dmxdevfilter->buffer.queue);return 0 ; }
@@ -404,12 +424,18 @@ {struct dmxdev_filter *dmxdevfilter = feed->priv;struct dvb_ringbuffer *buffer; -unsigned long flags; +unsigned long flags =0 ;int ret; -spin_lock_irqsave (&dmxdevfilter->dev->lock, flags); +if (dmxdev_conservative_locks) +spin_lock_irqsave (&dmxdevfilter->dev->lock, flags); +else +spin_lock (&dmxdevfilter->dev->lock);if (dmxdevfilter->params.pes.output ==DMX_OUT_DECODER ) { -spin_unlock_irqrestore (&dmxdevfilter->dev->lock, flags); +if (dmxdev_conservative_locks) +spin_unlock_irqrestore (&dmxdevfilter->dev->lock, flags); +else +spin_unlock (&dmxdevfilter->dev->lock);return 0 ; }
@@ -419,7 +445,10 @@else buffer = &dmxdevfilter->dev->dvr_buffer;if (buffer->error) { -spin_unlock_irqrestore (&dmxdevfilter->dev->lock, flags); +if (dmxdev_conservative_locks) +spin_unlock_irqrestore (&dmxdevfilter->dev->lock, flags); +else +spin_unlock (&dmxdevfilter->dev->lock);wake_up (&buffer->queue);return 0 ; }
@@ -430,7 +459,10 @@dvb_ringbuffer_flush (buffer); buffer->error = ret; } -spin_unlock_irqrestore (&dmxdevfilter->dev->lock, flags); +if (dmxdev_conservative_locks) +spin_unlock_irqrestore (&dmxdevfilter->dev->lock, flags); +else +spin_unlock (&dmxdevfilter->dev->lock);wake_up (&buffer->queue);return 0 ; }
linux/drivers/media/dvb/dvb-core/dvb_demux.c +33 -9
90 lines not on this page · raw message
Quoted 9 lines · Oliver Endriss
Quoted 7 lines · e9hack wrote:
this change set is wrong. The affected functions cannot be called from an interrupt context, because they may process large buffers. In this case, interrupts are disabled for a long time. Functions, like
dvb_dmx_swfilter_packets(), could be called only from a tasklet. This change set does hide some strong design bugs indm1105.candau0828-dvb.c.Please revert this change set and do fix the bugs in
dm1105.candau0828-dvb.c(and other files).
Quoted 7 lines
@Mauro:
This changeset _must_ be reverted! It breaks all kernels since 2.6.27 for applications which use DVB and require a low interrupt latency.
It is a very bad idea to call the demuxer to process data buffers with interrupts disabled!
I agree, this is bad. The demuxer is far too much work to be done with IRQs off. IMHO, even doing it under a spin-lock is excessive. It should be a mutex. Drivers should use a work-queue to feed the demuxer.
Why was this even committed in the first place being that it has such a horrible result??
Quoted 15 lines · On Mon, 16 Feb 2009 08:19:06 -0800 (PST) Trent Piepho <[email protected]>…
Quoted 14 lines · Oliver Endriss
Quoted 7 lines · e9hack wrote:
this change set is wrong. The affected functions cannot be called from an interrupt context, because they may process large buffers. In this case, interrupts are disabled for a long time. Functions, like
dvb_dmx_swfilter_packets(), could be called only from a tasklet. This change set does hide some strong design bugs indm1105.candau0828-dvb.c.Please revert this change set and do fix the bugs in
dm1105.candau0828-dvb.c(and other files).@Mauro:
This changeset _must_ be reverted! It breaks all kernels since 2.6.27 for applications which use DVB and require a low interrupt latency.
Quoted 6 lines
Quoted 2 lines
It is a very bad idea to call the demuxer to process data buffers with interrupts disabled!
I agree, this is bad. The demuxer is far too much work to be done with IRQs off. IMHO, even doing it under a spin-lock is excessive. It should be a mutex. Drivers should use a work-queue to feed the demuxer.
Hartmut, Oliver and Trent: Thanks for helping with this issue. I've just reverted the changeset. We still need a fix at dm1105, au0828-dvb and maybe other drivers that call the filtering routines inside IRQ's.
Cheers,
Mauro
Quoted 3 lines
Hartmut, Oliver and Trent: Thanks for helping with this issue. I've just reverted the changeset. We still need a fix at dm1105, au0828-dvb and maybe other drivers that call the filtering routines inside IRQ's.
Fix the demux, add a worker thread and allow drivers to call it directly.
I'm not a big fan of videobuf_dvb or having each driver do it's own thing as an alternative.
Fixing the demux... Would this require and extra buffer copy? probably, but it's a trade-off between the amount of spent during code management on a driver by driver basis vs wrestling with videobuf_dvb and all of problems highlighted on the ML over the last 2 years.
demux->register_driver()demux->deliver_payload()demux->unregister_driver()
Then deprecate sw_filter....N() methods.
That would simplify drivers significantly, at the expense of another buffer copy while deliver-payload() clones the buffer into its internal state to be more timely.
- Steve
Quoted 10 lines · Steven Toth
Quoted 3 lines
Hartmut, Oliver and Trent: Thanks for helping with this issue. I've just reverted the changeset. We still need a fix at dm1105, au0828-dvb and maybe other drivers that call the filtering routines inside IRQ's.
Fix the demux, add a worker thread and allow drivers to call it directly.
I'm not a big fan of
videobuf_dvbor having each driver do it's own thing as an alternative.Fixing the demux... Would this require and extra buffer copy? probably, but it's a trade-off between the amount of spent during code management on a driver by driver basis vs wrestling with
videobuf_dvband all of problems highlighted on the ML over the last 2 years.
Quoted 7 lines
demux->register_driver()demux->deliver_payload()demux->unregister_driver()Then deprecate sw_filter....N() methods.
That would simplify drivers significantly, at the expense of another buffer copy while deliver-payload() clones the buffer into its internal state to be more timely.
I meant to add...
The cx18 and a few other smaller drivers (flexcop?) dvb drivers also call directly. cx23885/cx88 does not.
- Steve
Quoted 22 lines · Steven Toth
Quoted 21 lines · Steven Toth
Quoted 4 lines
Hartmut, Oliver and Trent: Thanks for helping with this issue. I've just reverted the changeset. We still need a fix at dm1105, au0828-dvb and maybe
other drivers that call the filtering routines inside IRQ's.Fix the demux, add a worker thread and allow drivers to call it directly.
I'm not a big fan of
videobuf_dvbor having each driver do it's own thing as an alternative.Fixing the demux... Would this require and extra buffer copy? probably, but it's a trade-off between the amount of spent during code management on a driver by driver basis vs wrestling with
videobuf_dvband all of problems highlighted on the ML over the last 2 years.
demux->register_driver()demux->deliver_payload()demux->unregister_driver()Then deprecate sw_filter....N() methods.
Quoted 8 lines
Quoted 3 lines
That would simplify drivers significantly, at the expense of another buffer copy while deliver-payload() clones the buffer into its internal state to be more timely.
I meant to add...
The cx18 and a few other smaller drivers (flexcop?) dvb drivers also call directly. cx23885/cx88 does not.
cx18 calls it from it's own worker thread. To keep up with the rate at which the CX23418 firmware hands over buffers (and times out the cpu2epu mailbox!) during a dual analog DTV capture, there's no avoiding having a worker thread in the cx18 driver.
Regards,
Andy
Quoted 6 lines · Steven Toth
Quoted 3 lines
Hartmut, Oliver and Trent: Thanks for helping with this issue. I've just reverted the changeset. We still need a fix at dm1105, au0828-dvb and maybe other drivers that call the filtering routines inside IRQ's.
Fix the demux, add a worker thread and allow drivers to call it directly.
Quoted 7 lines
I'm not a big fan of
videobuf_dvbor having each driver do it's own thing as an alternative.Fixing the demux... Would this require and extra buffer copy? probably, but it's a trade-off between the amount of spent during code management on a driver by driver basis vs wrestling with
videobuf_dvband all of problems highlighted on the ML over the last 2 years.
Have the driver copy the data into the demuxer from the interrupt handler with irqs disabled? That's still too much.
The right way to do it is to have a queue of DMA buffers. In the interrupt handler the driver takes the completed DMA buffer off the "to DMA" queue and puts it in the "to process" queue. The driver should not copy and cetainly not demux the data from the interrupt handler.
Quoted 14 lines · Trent Piepho
Quoted 5 lines · Steven Toth
Quoted 3 lines
Hartmut, Oliver and Trent: Thanks for helping with this issue. I've just reverted the changeset. We still need a fix at dm1105, au0828-dvb and maybe other drivers that call the filtering routines inside IRQ's.
Fix the demux, add a worker thread and allow drivers to call it directly.
Quoted 7 lines
I'm not a big fan of
videobuf_dvbor having each driver do it's own thing as an alternative.Fixing the demux... Would this require and extra buffer copy? probably, but it's a trade-off between the amount of spent during code management on a driver by driver basis vs wrestling with
videobuf_dvband all of problems highlighted on the ML over the last 2 years.
Quoted 7 lines
Have the driver copy the data into the demuxer from the interrupt handler with irqs disabled? That's still too much.
The right way to do it is to have a queue of DMA buffers. In the interrupt handler the driver takes the completed DMA buffer off the "to DMA" queue and puts it in the "to process" queue. The driver should not copy and cetainly not demux the data from the interrupt handler.
I know what the right way is Trent (see cx23885) although thank you for reminding me. videobuf_dvb hasn't convinced people like me to bury myself in its mess or complexity during retro fits cases. Retro fitting videobuf_dvb into cx18 (at the time) was way too much effort.
Retro fitting it into existing drivers can be painful and I haven't seen any volunteers stand up over the last 24 months to get this done.
My suggestion? For the most part we're talking about very low data rates for DVB, coupled with fast memory buses for memcpy's. If the group doesn't want calls to the sw_filter methods then implement a half-way-house replacement for those drivers - as I mentioned above - with the memcpy. Either this approach, or make videobuf_dvb mandatory and deprecate sw_filter_...n() to intensionally break drivers and force maintainers to comply. This will upset people.
Realistically, this thread will probably go on for a couple of days then trail off into nothing. The subject will rear it's head again in a few months, like it's done in the past.
A general question to the group: Who wants to volunteer to retro fit videobuf_dvb into the current drivers so we can avoid calls to sw_filter_...n() directly?
I'm willing to loan any hardware I have to the volunteer assuming we can work out something on shipping.
- Steve
Quoted 3 lines · Steven Toth
A general question to the group: Who wants to volunteer to retro fit
videobuf_dvbinto the current drivers so we can avoid calls to sw_filter_...n() directly?
Can videobuf_dvb be used by hardware with MPEG decoders, too? My (probably not up-to-date) impression was that it's been designed specifically for "budget" cards with no demux or decoder hardware. But the sw_filter functions are useful for those devices, too, i.e. to feed a PID to multiple userspace clients or to provide section filters where some kind of hardware only provides basic PID filtering.
Regards,
Andreas
Quoted 28 lines · Steven Toth
Quoted 14 lines · Trent Piepho
Quoted 5 lines · Steven Toth
Quoted 3 lines
Hartmut, Oliver and Trent: Thanks for helping with this issue. I've just reverted the changeset. We still need a fix at dm1105, au0828-dvb and maybe other drivers that call the filtering routines inside IRQ's.
Fix the demux, add a worker thread and allow drivers to call it directly.
Quoted 7 lines
I'm not a big fan of
videobuf_dvbor having each driver do it's own thing as an alternative.Fixing the demux... Would this require and extra buffer copy? probably, but it's a trade-off between the amount of spent during code management on a driver by driver basis vs wrestling with
videobuf_dvband all of problems highlighted on the ML over the last 2 years.Quoted 7 lines
Have the driver copy the data into the demuxer from the interrupt handler with irqs disabled? That's still too much.
The right way to do it is to have a queue of DMA buffers. In the interrupt handler the driver takes the completed DMA buffer off the "to DMA" queue and puts it in the "to process" queue. The driver should not copy and cetainly not demux the data from the interrupt handler.
I know what the right way is Trent (see cx23885) although thank you for reminding me.
videobuf_dvbhasn't convinced people like me to bury myself in its mess or complexity during retro fits cases. Retro fittingvideobuf_dvbinto cx18 (at the time) was way too much effort.
Quoted 7 lines
Retro fitting it into existing drivers can be painful and I haven't seen any volunteers stand up over the last 24 months to get this done.
My suggestion? For the most part we're talking about very low data rates for DVB, coupled with fast memory buses for memcpy's. If the group doesn't want calls to the
sw_filtermethods then implement a half-way-house replacement for those drivers - as I mentioned above - with the memcpy. Either this approach, or
The problem is holding a spin lock with irqs disabled for a long amount of time. What exactly is your plan that will remove this, yet allow drivers to process their buffers from an irq handler?
Quoted 3 lines
A general question to the group: Who wants to volunteer to retro fit
videobuf_dvbinto the current drivers so we can avoid calls to sw_filter_...n() directly?
I don't see why videobuf_dvb is needed.
Quoted 30 lines · Trent Piepho
Quoted 29 lines · Steven Toth
Quoted 13 lines · Trent Piepho
Quoted 5 lines · Steven Toth
Quoted 3 lines
Hartmut, Oliver and Trent: Thanks for helping with this issue. I've just reverted the changeset. We still need a fix at dm1105, au0828-dvb and maybe other drivers that call the filtering routines inside IRQ's.
Fix the demux, add a worker thread and allow drivers to call it directly.
Quoted 7 lines
I'm not a big fan of
videobuf_dvbor having each driver do it's own thing as an alternative.Fixing the demux... Would this require and extra buffer copy? probably, but it's a trade-off between the amount of spent during code management on a driver by driver basis vs wrestling with
videobuf_dvband all of problems highlighted on the ML over the last 2 years.Quoted 7 lines
Have the driver copy the data into the demuxer from the interrupt handler with irqs disabled? That's still too much.
The right way to do it is to have a queue of DMA buffers. In the interrupt handler the driver takes the completed DMA buffer off the "to DMA" queue and puts it in the "to process" queue. The driver should not copy and cetainly not demux the data from the interrupt handler.
I know what the right way is Trent (see cx23885) although thank you for reminding me.
videobuf_dvbhasn't convinced people like me to bury myself in its mess or complexity during retro fits cases. Retro fittingvideobuf_dvbinto cx18 (at the time) was way too much effort.Retro fitting it into existing drivers can be painful and I haven't seen any volunteers stand up over the last 24 months to get this done.
Quoted 8 lines
Quoted 4 lines
My suggestion? For the most part we're talking about very low data rates for DVB, coupled with fast memory buses for memcpy's. If the group doesn't want calls to the
sw_filtermethods then implement a half-way-house replacement for those drivers - as I mentioned above - with the memcpy. Either this approach, orThe problem is holding a spin lock with irqs disabled for a long amount of time. What exactly is your plan that will remove this, yet allow drivers to process their buffers from an irq handler?
That's not what I was suggesting. I was suggesting adding some ring buffer code and a worker thread for each driver context (done in a mythical demux->register func). This means that each driver get's it's own worker and ringbuffer. Driver mutex on your own ring buffer is localized to your instance of the driver. It would be up to your drivers worker thread (instantiated and managed incidentally by the demux core, not at irq level), to acquire the long term spinlock via sw_filter_n (already in demux core) underdiscussion and NOT block a driver calling demux->feedMyPersonalRingBuffer().
Quoted 6 lines
Quoted 4 lines
A general question to the group: Who wants to volunteer to retro fit
videobuf_dvbinto the current drivers so we can avoid calls to sw_filter_...n() directly?I don't see why
videobuf_dvbis needed.
That was the point I was trying to make. IE. Push videobuf_dvb like behavior into the demux core, having drivers register, give each driver it's own worker thread and have that thread, running not in the interrupt context, feed the existing sw_filter_n() functions. The price is the cost of doing a memcpy of a low bitrate low frequency buffer into your demux's personal ring buffer. That has to be more efficient than the current drivers that feed sw_filter_n() directly, but not ideal. It's a half-way-house solution that consolidates complicated code into code, and keeps the drivers clean and easier to manage.
Trade off an infrequent memcpy on a low volume stream in < 50% of the drivers for a simplified approach. You don't have to do this for every driver, especially drivers that already implement videobuf_dvb, do it for the current problematic drivers...... or, have a volunteer add videobuf_dvb to all of the existing drivers.
- Steve
Quoted 28 lines · Steven Toth
Quoted 27 lines · Trent Piepho
Quoted 25 lines · Steven Toth
Quoted 5 lines · Trent Piepho
Quoted 4 lines · Steven Toth
Fixing the demux... Would this require and extra buffer copy? probably, but it's a trade-off between the amount of spent during code management on a driver by driver basis vs wrestling with
videobuf_dvband all of problems highlighted on the ML over the last 2 years.Quoted 7 lines
Have the driver copy the data into the demuxer from the interrupt handler with irqs disabled? That's still too much.
The right way to do it is to have a queue of DMA buffers. In the interrupt handler the driver takes the completed DMA buffer off the "to DMA" queue and puts it in the "to process" queue. The driver should not copy and cetainly not demux the data from the interrupt handler.
I know what the right way is Trent (see cx23885) although thank you for reminding me.
videobuf_dvbhasn't convinced people like me to bury myself in its mess or complexity during retro fits cases. Retro fittingvideobuf_dvbinto cx18 (at the time) was way too much effort.Retro fitting it into existing drivers can be painful and I haven't seen any volunteers stand up over the last 24 months to get this done.
My suggestion? For the most part we're talking about very low data rates for DVB, coupled with fast memory buses for memcpy's. If the group doesn't want calls to the
sw_filtermethods then implement a half-way-house replacement for those drivers - as I mentioned above - with the memcpy. Either this approach, or
Quoted 7 lines
Quoted 3 lines
The problem is holding a spin lock with irqs disabled for a long amount of time. What exactly is your plan that will remove this, yet allow drivers to process their buffers from an irq handler?
That's not what I was suggesting. I was suggesting adding some ring buffer code and a worker thread for each driver context (done in a mythical
demux->registerfunc). This means that each driver get's it's own worker and ringbuffer. Driver
But won't this new ringbuffer be fed from interrupt context? So instead of feeding the demuxer from an irq, you are feeding the pre-demuxer ringbuffer from an irq. Isn't that going to have the same long term lock holding with irqs disabled problem?
Quoted 10 lines
Quoted 5 lines
Quoted 3 lines
A general question to the group: Who wants to volunteer to retro fit
videobuf_dvbinto the current drivers so we can avoid calls to sw_filter_...n() directly?I don't see why
videobuf_dvbis needed.That was the point I was trying to make. IE. Push
videobuf_dvblike behavior into the demux core, having drivers register, give each driver it's own worker thread and have that thread, running not in the interrupt context, feed the existingsw_filter_n()functions. The price is the cost of doing a memcpy of a
If you look at the pluto2 fix I did, the code to create a work queue is very simple. I don't think moving that to the demuxer would make the patch to the driver much simpler.
The difficulty comes from not using a single buffer so the driver doesn't need to hold a spin lock while it copies data out.
Quoted 15 lines · Trent Piepho
Quoted 14 lines · Oliver Endriss
Quoted 7 lines · e9hack wrote:
this change set is wrong. The affected functions cannot be called from an interrupt context, because they may process large buffers. In this case, interrupts are disabled for a long time. Functions, like
dvb_dmx_swfilter_packets(), could be called only from a tasklet. This change set does hide some strong design bugs indm1105.candau0828-dvb.c.Please revert this change set and do fix the bugs in
dm1105.candau0828-dvb.c(and other files).@Mauro:
This changeset _must_ be reverted! It breaks all kernels since 2.6.27 for applications which use DVB and require a low interrupt latency.
Quoted 6 lines
Quoted 2 lines
It is a very bad idea to call the demuxer to process data buffers with interrupts disabled!
I agree, this is bad. The demuxer is far too much work to be done with IRQs off. IMHO, even doing it under a spin-lock is excessive. It should be a mutex. Drivers should use a work-queue to feed the demuxer.
Agreed, this would be the best solution.
On the other hand, a workqueue handler would be scheduled later, so you need larger buffers in the driver. Some chipsets have very small buffers...
Anway, this would be a major change. All drivers must be carefully modified and tested for an extended period.
Meanwhile I had a look at the changeset, and I do not understand why spin_lock_irq... should be required everywhere.
Afaics a driver may safely call dvb_dmx_swfilter_packets, dvb_dmx_swfilter_204 or dvb_dmx_swfilter from process context, tasklet or interrupt handler 'as is'.
@Andreas:
Could you please explain in more detail what bad things might happen?
CU
Oliver
Signature · 3 lines
---------------------------------------------------------------- VDR Remote Plugin 0.4.0: http://www.escape-edv.de/endriss/vdr/ ----------------------------------------------------------------
Quoted 33 lines · Oliver Endriss
Quoted 11 lines · Trent Piepho
Quoted 10 lines · Oliver Endriss
Quoted 7 lines · e9hack wrote:
this change set is wrong. The affected functions cannot be called from an interrupt context, because they may process large buffers. In this case, interrupts are disabled for a long time. Functions, like
dvb_dmx_swfilter_packets(), could be called only from a tasklet. This change set does hide some strong design bugs indm1105.candau0828-dvb.c.Please revert this change set and do fix the bugs in
dm1105.candau0828-dvb.c(and other files).@Mauro:
Quoted 8 lines
Quoted 5 lines
This changeset _must_ be reverted! It breaks all kernels since 2.6.27 for applications which use DVB and require a low interrupt latency.
It is a very bad idea to call the demuxer to process data buffers with interrupts disabled!
I agree, this is bad. The demuxer is far too much work to be done with IRQs off. IMHO, even doing it under a spin-lock is excessive. It should be a mutex. Drivers should use a work-queue to feed the demuxer.
Agreed, this would be the best solution.
On the other hand, a workqueue handler would be scheduled later, so you need larger buffers in the driver. Some chipsets have very small buffers...
Anway, this would be a major change. All drivers must be carefully modified and tested for an extended period.
Meanwhile I had a look at the changeset, and I do not understand why
spin_lock_irq... should be required everywhere.
Quoted 6 lines
Afaics a driver may safely call
dvb_dmx_swfilter_packets,dvb_dmx_swfilter_204ordvb_dmx_swfilterfrom process context, tasklet or interrupt handler 'as is'.@Andreas:
Could you please explain in more detail what bad things might happen?
To quote myself from the changelog: This fixes a deadlock discovered by lockdep.
The lock is used in process context (e.g. DMX_START) and might also be used from interrupt context (e.g. dvb_dmx_swfilter).
Quoted 1 line
From http://osdir.com/ml/kernel.janitors/2002-08/msg00022.html:
"spin_lock_irq disables local interrupts and then takes the spin_lock. If you know you're in process context and other users may be in interrupt context, this is the correct call to make.
spin_lock_irqsave saves local interrupt state into the flags variable, disables interrupts, then takes the spin_lock. spin_unlock_irqrestore restores the local state saved in the flags. Use this variant if you don't know whether you're in interrupt or process context."
So, if the assumtions above are correct, then spin_lock_irq must be used by all functions called from process context and spin_lock_irqsave must be used by all functions called from an unknown context.
Regards,
Andreas
Quoted 7 lines · Andreas Oberritter
Quoted 3 lines · Oliver Endriss
...
@Andreas:
Could you please explain in more detail what bad things might happen?To quote myself from the changelog: This fixes a deadlock discovered by lockdep.
I re-read the commit message, but it still does not ring any bells. Could you please post the lockdep output?
Quoted 2 lines
The lock is used in process context (e.g.
DMX_START) and might also be used from interrupt context (e.g.dvb_dmx_swfilter).
Correct:
- dvb_dmx_swfilter uses spin_lock (called from irq, tasklet, whatever)
- DMX_START uses spin_lock_irq (called from process)
-> ok (see below).
Quoted 11 lines
Quoted 1 line
From http://osdir.com/ml/kernel.janitors/2002-08/msg00022.html:
"
spin_lock_irqdisables local interrupts and then takes thespin_lock. If you know you're in process context and other users may be in interrupt context, this is the correct call to make.
spin_lock_irqsavesaves local interrupt state into the flags variable, disables interrupts, then takes thespin_lock.spin_unlock_irqrestorerestores the local state saved in the flags. Use this variant if you don't know whether you're in interrupt or process context."
Quoted 4 lines
So, if the assumtions above are correct, then
spin_lock_irqmust be used by all functions called from process context andspin_lock_irqsavemust be used by all functions called from an unknown context.
Correct.
[1] If you want to lock a process against an interrupt handler,
- the process must use spin_lock_irq()
- the interrupt can use spin_lock()
A routine has to use spin_lock_irqsave if (and only if) process and irq call the routine concurrently. I do not see yet how this might happen.
(Basically, the same happens when locking between tasklet and process context, except that it is sufficient to use spin_lock_bh instead of spin_lock_irq.)
Regards
Oliver
Signature · 3 lines
---------------------------------------------------------------- VDR Remote Plugin 0.4.0: http://www.escape-edv.de/endriss/vdr/ ----------------------------------------------------------------
Quoted 6 lines · Oliver Endriss
[1] If you want to lock a process against an interrupt handler,
- the process must usespin_lock_irq()
- the interrupt can usespin_lock()A routine has to use
spin_lock_irqsaveif (and only if) process and irq call the routine concurrently. I do not see yet how this might happen.
Some code calls the swfilter functions from process context and some drivers call them from interrupt context.
Quoted 10 lines · Trent Piepho
Quoted 6 lines · Oliver Endriss
[1] If you want to lock a process against an interrupt handler,
- the process must usespin_lock_irq()
- the interrupt can usespin_lock()A routine has to use
spin_lock_irqsaveif (and only if) process and irq call the routine concurrently. I do not see yet how this might happen.Some code calls the swfilter functions from process context and some drivers call them from interrupt context.
There would be a problem if (and only if) it could happen concurrently within a given driver. A driver may call the functions either from process context or from a tasklet/irq.
User space access will occur only if demux_source == DMX_MEMORY_FE. In this case the driver must not call the routine.
If demux_source == DMX_FRONTEND, the driver may call the routine, but userspace won't.
Sorry, I need more information to identify the problem.
CU
Oliver
Signature · 3 lines
---------------------------------------------------------------- VDR Remote Plugin 0.4.0: http://www.escape-edv.de/endriss/vdr/ ----------------------------------------------------------------
Quoted 2 lines · Oliver Endriss
I re-read the commit message, but it still does not ring any bells. Could you please post the lockdep output?
No. I didn't save it.
Regards,
Andreas
Quoted 6 lines · Andreas Oberritter
Quoted 5 lines · Oliver Endriss
Quoted 3 lines · Trent Piepho
I agree, this is bad. The demuxer is far too much work to be done with IRQs off. IMHO, even doing it under a spin-lock is excessive. It should be a mutex. Drivers should use a work-queue to feed the demuxer.
Quoted 8 lines
Quoted 8 lines
Agreed, this would be the best solution.
On the other hand, a workqueue handler would be scheduled later, so you need larger buffers in the driver. Some chipsets have very small buffers...
Anway, this would be a major change. All drivers must be carefully modified and tested for an extended period.
Don't most drivers already feed the demuxer from process context and not the irq handler? What drivers _do_ have a problem? I see pluto2 is one. Anyone have documentation for this chip?
Quoted 4 lines
So, if the assumtions above are correct, then
spin_lock_irqmust be used by all functions called from process context andspin_lock_irqsavemust be used by all functions called from an unknown context.
I agree, to be correct that's what's necessary. Some drivers call the demuxer functions from interrupt context, so we have to use the irq disabling functions everywhere.
But disabling irqs for demuxing causes too much latency. The proper fix is to not call the demuxer from interrupt context.