High CPU load in "top" due to budget_av slot polling
Hello,
on 2.6.24 I run into a small issue with the budget_av driver:
Hardware: Athlon 64X2 3800+, Satelco Easywatch DVB-C (1894:002c), no CI/CAM
Software: Linux kernel 2.6.24 (gentoo-r4) SMP x86_64, budget_av module
Error description:
After the budget_av driver module is loaded (even without any DVB application), the CPU load average in 'top' rises to ~1, but in top no active tasks are found. After unloading the driver, the load decreases again to ~0.
Displaying the blocked tasks during high load with Alt-SysRq-W shows that the kdvb-ca kernel thread seems to be accounted as blocked when it polls for the CI slot status:
---------------------------------------------------
SysRq : Show Blocked State
task PC stack pid father
kdvb-ca-0:0 D 0000000100d78984 0 2046 2ffff81007ee73c70 0000000000000046 0000000000000000 0000000000000008 0000000000000800 0000000000000000 0000000000000286 ffff81007bebe080 ffff81007f8d47e0 ffff81007ee73c80 ffff81007ed91a00 0000000100d78984
Call Trace: [<ffffffff80241da6>] __mod_timer+0xb6/0xd0 [<ffffffff804e52ff>] schedule_timeout+0x5f/0xd0 [<ffffffff80241890>] process_timeout+0x0/0x10 [<ffffffff80241dd6>] msleep+0x16/0x30 [<ffffffff880a0189>] :saa7146:saa7146_wait_for_debi_done+0x159/0x260 [<ffffffff880bba3a>] :budget_core:ttpci_budget_debiread+0x6a/0x140 [<ffffffff880d1cfb>] :budget_av:ciintf_poll_slot_status+0xbb/0x1c0 [<ffffffff880aa800>] :dvb_core:dvb_ca_en50221_thread+0x0/0xa10 [<ffffffff880a9339>] :dvb_core:dvb_ca_en50221_check_camstatus+0x59/0x100 [<ffffffff880aa8ba>] :dvb_core:dvb_ca_en50221_thread+0xba/0xa10 [<ffffffff8022f221>] update_curr+0x61/0xb0 [<ffffffff8022f221>] update_curr+0x61/0xb0 [<ffffffff802341e6>] dequeue_task_fair+0x46/0x80 [<ffffffff8022f79d>] __dequeue_entity+0x3d/0x50 [<ffffffff804e4a7e>] thread_return+0x3d/0x52f [<ffffffff80288da4>] filp_close+0x54/0x90 [<ffffffff880aa800>] :dvb_core:dvb_ca_en50221_thread+0x0/0xa10 [<ffffffff8024d48b>] kthread+0x4b/0x80 [<ffffffff8020ca48>] child_rip+0xa/0x12 [<ffffffff8024d440>] kthread+0x0/0x80 [<ffffffff8020ca3e>] child_rip+0x0/0x12 ---------------------------------------------------
Enabling debug traces shows that polling for the PSR in function 'saa7146_wait_for_debi_done_sleep' constantly times out after 250x1ms sleeps:
Quoted 1 line
saa7146:
saa7146_wait_for_debi_done_sleep(): saa7146 (0):
saa7146_wait_for_debi_done_sleep timed out while waiting for transfer completion
Increasing the 250ms did not avoid the timeout. And as I understood from previous list mails, this timeout is normal when no CI/CAM is connected to the DEBI. However, for me the high frequency polling does not make sense if someone does not plan to plug in a CI/CAM.
When commenting out two lines in 'dvb_ca_en50221_thread_update_delay' to increase the polling timer for slotstate NONE from 100ms (!) to 60s, the CPU load went down to 0. So this is some kind of workaround for me.
Please note: The high CPU load is not only a statistical issue due to the kernel accounting. It would put at least some unnecessary stress on PCI and CPU and e.g. also has the effect that cron tasks with idle guard condition would not start, so on midterm this should be fixed.
Finally, my questions:
1. Did I understand this right, that the behaviour above is expected when no CI is connected?
2. Are all budget_av cards unable to check CAM slot state via interrupt for HW reasons (as budget_ci does)?
3. Would it be possible to substitute the current PSR DEBI_S polling with an interrupt based solution via IER/ISR? (driver av7110 alreadys seems to do this for its DEBI DMA)? Or was this not considered worthy, due to the expected short waiting period and the tricky IER handling? The code does not state further thoughts about this.
4. Are the high timeout periods in debi_done (50ms/250ms) in relation to the 100ms poll timer intended? (I found the recent patch to this code in the mailing list end of last year)
5. If we would be restricted to poll with high frequency: Why not at least allow users without CI to disable polling for slots or change the interval, e.g. via module options?
Regards,
Robert Schedel
_______________________________________________
linux-dvb mailing list
[email protected]
http://www.linuxtv.org/cgi-bin/mailman/listinfo/linux-dvb
Hello all,
meanwhile I also found old threads from August 2007 and earlier, named "System load raises when budget_av is loaded" and "System load raises when budget_av is loaded", which describe exactly the same issue. However, both were written before the latest code changes.
The latest debi_done code change from November 2007 does not use any DEBI_E checks anymore. To my understanding this SAA7146 event indicates
DEBI errors like timeout via PCI and should be received by the driver much earlier than those 250ms (=end of cycle). Seems like I have to
play around with this myself.
Time for a bugzilla ticket?
Regards,
Robert Schedel
_______________________________________________
linux-dvb mailing list
[email protected]
http://www.linuxtv.org/cgi-bin/mailman/listinfo/linux-dvb
Quoted 7 lines · Robert Schedel
Hello all,
meanwhile I also found old threads from August 2007 and earlier, named "System load raises when
budget_avis loaded" and "System load raises whenbudget_avis loaded", which describe exactly the same issue. However, both were written before the latest code changes.
Quoted 5 lines
The latest
debi_donecode change from November 2007 does not use anyDEBI_Echecks anymore. To my understanding this SAA7146 event indicates
DEBI errors like timeout via PCI and should be received by the driver much earlier than those 250ms (=end of cycle). Seems like I have to
play around with this myself.
IIRC there was a problem with the DEBI_E flag but I don't remember what it was exactly.
Quoted 1 line
Time for a bugzilla ticket?
You can do this if you prefer (but this will neither fix the bug nor have any other positive effect).
CU
Oliver
Signature · 8 lines
---------------------------------------------------------------- VDR Remote Plugin 0.4.0: http://www.escape-edv.de/endriss/vdr/ ---------------------------------------------------------------- _______________________________________________ linux-dvb mailing list [email protected] http://www.linuxtv.org/cgi-bin/mailman/listinfo/linux-dvb
Quoted 55 lines · Robert Schedel
Hello,
on 2.6.24 I run into a small issue with the
budget_avdriver:Hardware: Athlon 64X2 3800+, Satelco Easywatch DVB-C (1894:002c), no CI/CAM
Software: Linux kernel 2.6.24 (gentoo-r4) SMPx86_64,budget_avmoduleError description:
After thebudget_avdriver module is loaded (even without any DVB application), the CPU load average in 'top' rises to ~1, but in top no active tasks are found. After unloading the driver, the load decreases again to ~0.Displaying the blocked tasks during high load with Alt-SysRq-W shows that the kdvb-ca kernel thread seems to be accounted as blocked when it polls for the CI slot status:
---------------------------------------------------
SysRq : Show Blocked State
task PC stack pid father
kdvb-ca-0:0 D0000000100d789840 2046 2ffff81007ee73c700000000000000046 0000000000000000 0000000000000008 0000000000000800 0000000000000000 0000000000000286ffff81007bebe080ffff81007f8d47e0ffff81007ee73c80ffff81007ed91a000000000100d78984Call Trace: [<ffffffff80241da6>] __mod_timer+0xb6/0xd0 [<ffffffff804e52ff>] schedule_timeout+0x5f/0xd0 [<ffffffff80241890>] process_timeout+0x0/0x10 [<ffffffff80241dd6>] msleep+0x16/0x30 [<ffffffff880a0189>] :saa7146:saa7146_wait_for_debi_done+0x159/0x260 [<ffffffff880bba3a>] :budget_core:ttpci_budget_debiread+0x6a/0x140 [<ffffffff880d1cfb>] :budget_av:ciintf_poll_slot_status+0xbb/0x1c0 [<ffffffff880aa800>] :dvb_core:dvb_ca_en50221_thread+0x0/0xa10 [<ffffffff880a9339>] :dvb_core:dvb_ca_en50221_check_camstatus+0x59/0x100 [<ffffffff880aa8ba>] :dvb_core:dvb_ca_en50221_thread+0xba/0xa10 [<ffffffff8022f221>] update_curr+0x61/0xb0 [<ffffffff8022f221>] update_curr+0x61/0xb0 [<ffffffff802341e6>] dequeue_task_fair+0x46/0x80 [<ffffffff8022f79d>] __dequeue_entity+0x3d/0x50 [<ffffffff804e4a7e>] thread_return+0x3d/0x52f [<ffffffff80288da4>] filp_close+0x54/0x90 [<ffffffff880aa800>] :dvb_core:dvb_ca_en50221_thread+0x0/0xa10 [<ffffffff8024d48b>] kthread+0x4b/0x80 [<ffffffff8020ca48>] child_rip+0xa/0x12 [<ffffffff8024d440>] kthread+0x0/0x80 [<ffffffff8020ca3e>] child_rip+0x0/0x12 ---------------------------------------------------Enabling debug traces shows that polling for the PSR in function '
saa7146_wait_for_debi_done_sleep' constantly times out after 250x1ms sleeps:Quoted 1 line
saa7146:
saa7146_wait_for_debi_done_sleep(): saa7146 (0):
saa7146_wait_for_debi_done_sleeptimed out while waiting for transfer completion
Quoted 8 lines
Increasing the 250ms did not avoid the timeout. And as I understood from previous list mails, this timeout is normal when no CI/CAM is connected to the DEBI. However, for me the high frequency polling does not make sense if someone does not plan to plug in a CI/CAM.
When commenting out two lines in '
dvb_ca_en50221_thread_update_delay' to increase the polling timer for slotstate NONE from 100ms (!) to 60s, the CPU load went down to 0. So this is some kind of workaround for me.
Afaics the polling interval could be increased to something like 5s or 10s if (and only if) the slot is empty. Could you provide a patch?
Quoted 4 lines
Please note: The high CPU load is not only a statistical issue due to the kernel accounting. It would put at least some unnecessary stress on PCI and CPU and e.g. also has the effect that cron tasks with idle guard condition would not start, so on midterm this should be fixed.
Agreed.
Quoted 3 lines
Finally, my questions:
1. Did I understand this right, that the behaviour above is expected when no CI is connected?
Yes, but afaics the polling interval is way too short.
Quoted 2 lines
2. Are all
budget_avcards unable to check CAM slot state via interrupt for HW reasons (asbudget_cidoes)?
I don't have any budget-av hardware, so I don't know.
But I think that Andrew(?) had a good reason to implement it this way. (In contrast the budget-ci driver was always interrupt-driven.)
If someone finds out that a given card can operate in interrupt mode, it should be changed for this card. Patches are welcome. ;-)
Quoted 5 lines
3. Would it be possible to substitute the current PSR
DEBI_Spolling with an interrupt based solution via IER/ISR? (driver av7110 alreadys seems to do this for its DEBI DMA)? Or was this not considered worthy, due to the expected short waiting period and the tricky IER handling? The code does not state further thoughts about this.
The av7110 driver uses interrupt mode for buffer transfers in dma mode. It does not make much sense to use interrupt mode for single-word transfers, because the single-word transfers are very fast. But I understand that the timeout causes a problem in this case.
Quoted 3 lines
4. Are the high timeout periods in
debi_done(50ms/250ms) in relation to the 100ms poll timer intended? (I found the recent patch to this code in the mailing list end of last year)
That patch was applied to reduce the load on the pci bus in busy-wait mode. Basically it did not change anything for cam polling. (In fact I was not aware that the CAM was polled every 100ms. Imho this should be fixed.)
Quoted 3 lines
5. If we would be restricted to poll with high frequency: Why not at least allow users without CI to disable polling for slots or change the interval, e.g. via module options?
Module options are evil. ;-)
We should do this only if everything else fails.
CU
Oliver
Signature · 8 lines
---------------------------------------------------------------- VDR Remote Plugin 0.4.0: http://www.escape-edv.de/endriss/vdr/ ---------------------------------------------------------------- _______________________________________________ linux-dvb mailing list [email protected] http://www.linuxtv.org/cgi-bin/mailman/listinfo/linux-dvb
Quoted 21 lines · Oliver Endriss
Quoted 20 lines · Robert Schedel
Hello,
on 2.6.24 I run into a small issue with the
budget_avdriver:Hardware: Athlon 64X2 3800+, Satelco Easywatch DVB-C (1894:002c), no CI/CAM
Software: Linux kernel 2.6.24 (gentoo-r4) SMPx86_64,budget_avmoduleError description:
After thebudget_avdriver module is loaded (even without any DVB application), the CPU load average in 'top' rises to ~1, but in top no active tasks are found. After unloading the driver, the load decreases again to ~0.Displaying the blocked tasks during high load with Alt-SysRq-W shows that the kdvb-ca kernel thread seems to be accounted as blocked when it polls for the CI slot status:
---------------------------------------------------
[...]
---------------------------------------------------Enabling debug traces shows that polling for the PSR in function '
saa7146_wait_for_debi_done_sleep' constantly times out after 250x1ms sleeps:Quoted 1 line
saa7146:
saa7146_wait_for_debi_done_sleep(): saa7146 (0):
saa7146_wait_for_debi_done_sleeptimed out while waiting for transfer completion
Quoted 6 lines
Quoted 3 lines
Increasing the 250ms did not avoid the timeout. And as I understood from previous list mails, this timeout is normal when no CI/CAM is connected to the DEBI. However, for me the high frequency polling does not make sense if someone does not plan to plug in a CI/CAM.
When commenting out two lines in '
dvb_ca_en50221_thread_update_delay' to increase the polling timer for slotstate NONE from 100ms (!) to 60s, the CPU load went down to 0. So this is some kind of workaround for me.Afaics the polling interval could be increased to something like 5s or 10s if (and only if) the slot is empty. Could you provide a patch?
Attached a patch for 2.6.24.4. Opinions?
Unfortunately, a 5s poll interval for empty slots still results in a load average of about 0,06 (~ 250ms/5s).
Increasing to 10s would decrease the load and be fine for people without CAM, but increase the delay for those users inserting CAMs during runtime. 5s sounds like a reasonable tradeoff.
Is the 250ms timeout an approved limit? Decreasing it would push the load further down. Probably it still has to cover slow CAMs as well as a stressed PCI bus. Unfortunately, without CAM/CI I cannot make any statements myself.
Quoted 8 lines
Quoted 2 lines
Finally, my questions:
1. Did I understand this right, that the behaviour above is expected when no CI is connected?Yes, but afaics the polling interval is way too short.
Quoted 2 lines
2. Are all
budget_avcards unable to check CAM slot state via interrupt for HW reasons (asbudget_cidoes)?
Quoted 6 lines
I don't have any budget-av hardware, so I don't know.
But I think that Andrew(?) had a good reason to implement it this way. (In contrast the budget-ci driver was always interrupt-driven.)If someone finds out that a given card can operate in interrupt mode, it should be changed for this card. Patches are welcome. ;-)
My impression is, due to the different GPIO layout there is no way to get a CAM change IRQ. But it seems difficult to get information about the HW architecture of cards at all.
Regarding DEBI_E: Just found a av7110 code comment which also reflects what my recent tests showed:
/* Note: Don't try to handle the DEBI error irq (MASK_18), in
* intel mode the timeout is asserted all the time...
*/So really only DEBI_S would be left, see below.
Quoted 6 lines
Quoted 1 line
3. Would it be possible to substitute the current PSR
DEBI_Spolling with an interrupt based solution via IER/ISR? (driver av7110 alreadys seems to do this for its DEBI DMA)? Or was this not considered worthy, due to the expected short waiting period and the tricky IER handling? The code does not state further thoughts about this.The av7110 driver uses interrupt mode for buffer transfers in dma mode. It does not make much sense to use interrupt mode for single-word transfers, because the single-word transfers are very fast. But I understand that the timeout causes a problem in this case.
OK, interrupts of course decrease performance in the "sunny day" cases (=communication with inserted card in slot state READY). Having both approaches (interrupt when slotstate empty, later polling) would combine all benefits but also be somewhat crazy.
Quoted 6 lines
Quoted 1 line
4. Are the high timeout periods in
debi_done(50ms/250ms) in relation to the 100ms poll timer intended? (I found the recent patch to this code in the mailing list end of last year)That patch was applied to reduce the load on the pci bus in busy-wait mode. Basically it did not change anything for cam polling. (In fact I was not aware that the CAM was polled every 100ms. Imho this should be fixed.)
Only wondered whether the 250ms might have been smaller in former driver versions.
Quoted 5 lines
Quoted 2 lines
5. If we would be restricted to poll with high frequency: Why not at least allow users without CI to disable polling for slots or change the interval, e.g. via module options?
Module options are evil. ;-)
We should do this only if everything else fails.
Correct.
P.S. Had already created bugzilla #10459 for tracking. Sorry for late response, previous mail was stuck in spam filter.
Regards,
Robert
From: Robert Schedel <[email protected]>
This change addresses kernel bug #10459: In kernel 2.6.24 the budget_av driver polls for an CI slot in 100ms intervals (because no interrupt solution for budget_av cards is feasible due to HW reasons). If no CI/CAM is connected to the DVB card, polling times out only after 250ms. This periodic polling leads to high CPU load.
The change increases the polling interval for empty slots from 100ms to 5s. Intervals for remaining slot states (invalid, in progress, ready) are unchanged, as they are either temporary conditions or no timeout should occur.
--- drivers/media/dvb/dvb-core/dvb_ca_en50221.c | 20 ++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-)
drivers/media/dvb/dvb-core/dvb_ca_en50221.c +14 -6
@@ -910,14 +910,22 @@ static void dvb_ca_en50221_thread_updateint curdelay =100000000 ;int slot; +/* Beware of too high polling frequency, because one polling +* call might take several hundred milliseconds until timeout! +*/ for (slot =0 ; slot < ca->slot_count; slot++) {switch (ca->slot_info[slot].slot_state) {default :case DVB_CA_SLOTSTATE_NONE : + delay =HZ *60 ;// 60s +if (!(ca->flags &DVB_CA_EN50221_FLAG_IRQ_CAMCHANGE )) { + delay =HZ *5 ;// 5s + } +break ;case DVB_CA_SLOTSTATE_INVALID : - delay =HZ *60 ; + delay =HZ *60 ;// 60s if (!(ca->flags &DVB_CA_EN50221_FLAG_IRQ_CAMCHANGE )) { - delay =HZ /10 ; + delay =HZ /10 ;// 100ms }break ;
@@ -926,18 +934,18 @@ static void dvb_ca_en50221_thread_updatecase DVB_CA_SLOTSTATE_VALIDATE :case DVB_CA_SLOTSTATE_WAITFR :case DVB_CA_SLOTSTATE_LINKINIT : - delay =HZ /10 ; + delay =HZ /10 ;// 100ms break ;case DVB_CA_SLOTSTATE_RUNNING : - delay =HZ *60 ; + delay =HZ *60 ;// 60s if (!(ca->flags &DVB_CA_EN50221_FLAG_IRQ_CAMCHANGE )) { - delay =HZ /10 ; + delay =HZ /10 ;// 100ms }if (ca->open) {if ((!ca->slot_info[slot].da_irq_supported) || (!(ca->flags &DVB_CA_EN50221_FLAG_IRQ_DA ))) { - delay =HZ /10 ; + delay =HZ /10 ;// 100ms } }break ;
_______________________________________________
linux-dvb mailing list
[email protected]
http://www.linuxtv.org/cgi-bin/mailman/listinfo/linux-dvb
Hi Robert,
This patch looks great. I tested it with Easywatch DVB-C (1011:0019) without CI/CAM and load is 0 now while it was ~0.6 constantly before.
uname -a : Linux devbox 2.6.25 #1 Thu Apr 17 12:18:32 CEST 2008 i686 AMD Athlon(tm) Processor AuthenticAMD GNU/Linux
Cheers,
Guy
Quoted 3 lines · On Wed, 16 Apr 2008 23:28:02 +0200 Robert Schedel <[email protected]> wrote:
Attached a patch for 2.6.24.4. Opinions?
Signature · 8 lines
Guy Martin Gentoo Linux - HPPA port lead _______________________________________________ linux-dvb mailing list [email protected] http://www.linuxtv.org/cgi-bin/mailman/listinfo/linux-dvb
Quoted 4 lines · Robert Schedel
Is the 250ms timeout an approved limit? Decreasing it would push the load further down. Probably it still has to cover slow CAMs as well as a stressed PCI bus. Unfortunately, without CAM/CI I cannot make any statements myself.
Just got another idea to improve the code: Function "saa7146_wait_for_debi_done_sleep" could be reworked to use what is known as "truncated binary exponential backoff" algorithm. IOW, on each sleep duplicate the period from 1ms until a fixed maximum, e.g. 32ms. This way polling ends fast for those users with fast bus/CAM, and those requiring 200ms due to slow bus/CAM should not worry about e.g. 216ms response time.
My first tests look promising (load goes down to 0). However, is not the simple BEB algorithm already patented?
Regards,
Robert
_______________________________________________
linux-dvb mailing list
[email protected]
http://www.linuxtv.org/cgi-bin/mailman/listinfo/linux-dvb
Quoted 15 lines · Robert Schedel
Quoted 4 lines · Robert Schedel
Is the 250ms timeout an approved limit? Decreasing it would push the load further down. Probably it still has to cover slow CAMs as well as a stressed PCI bus. Unfortunately, without CAM/CI I cannot make any statements myself.
Just got another idea to improve the code: Function "
saa7146_wait_for_debi_done_sleep" could be reworked to use what is known as "truncated binary exponential backoff" algorithm. IOW, on each sleep duplicate the period from 1ms until a fixed maximum, e.g. 32ms. This way polling ends fast for those users with fast bus/CAM, and those requiring 200ms due to slow bus/CAM should not worry about e.g. 216ms response time.
Quoted 2 lines
My first tests look promising (load goes down to 0). However, is not the simple BEB algorithm already patented?
Load should go down to 0 if the sleep call does not busy-wait.
Please test whether the attached code fixes the problem. Btw, I will not claim a patent for that. :D
CU
Oliver
Signature · 3 lines
---------------------------------------------------------------- VDR Remote Plugin 0.4.0: http://www.escape-edv.de/endriss/vdr/ ----------------------------------------------------------------
linux/drivers/media/common/saa7146_core.c +7 -2
@@ -64,8 +64,10 @@ static inline int saa7146_wait_for_debi_ {unsigned long timeout;int err; +bool slow;/* wait for registers to be programmed */ + slow = false; timeout = jiffies +usecs_to_jiffies (us1);while (1 ) { err =time_after (jiffies, timeout);
@@ -77,10 +79,12 @@ static inline int saa7146_wait_for_debi_ dev->name, __func__);return -ETIMEDOUT ; } -msleep (1 ); +msleep (slow ?10 :1 ); + slow = true; }/* wait for transfer to complete */ + slow = false; timeout = jiffies +usecs_to_jiffies (us2);while (1 ) { err =time_after (jiffies, timeout);
@@ -92,7 +96,8 @@ static inline int saa7146_wait_for_debi_ "completion\n", dev->name, __func__));return -ETIMEDOUT ; } -msleep (1 ); +msleep (slow ?10 :1 ); + slow = true; }return 0 ;
_______________________________________________
linux-dvb mailing list
[email protected]
http://www.linuxtv.org/cgi-bin/mailman/listinfo/linux-dvb
Quoted 15 lines · Oliver Endriss
Quoted 14 lines · Robert Schedel
Quoted 4 lines · Robert Schedel
Is the 250ms timeout an approved limit? Decreasing it would push the load further down. Probably it still has to cover slow CAMs as well as a stressed PCI bus. Unfortunately, without CAM/CI I cannot make any statements myself.
Just got another idea to improve the code: Function "
saa7146_wait_for_debi_done_sleep" could be reworked to use what is known as "truncated binary exponential backoff" algorithm. IOW, on each sleep duplicate the period from 1ms until a fixed maximum, e.g. 32ms. This way polling ends fast for those users with fast bus/CAM, and those requiring 200ms due to slow bus/CAM should not worry about e.g. 216ms response time.
Quoted 7 lines
Quoted 2 lines
My first tests look promising (load goes down to 0). However, is not the simple BEB algorithm already patented?
Load should go down to 0 if the sleep call does not busy-wait.
Please test whether the attached code fixes the problem. Btw, I will not claim a patent for that. :D
OK, I just took the time to make a more reliable test series (because load measurements varied). All nonessential system processes and modules were terminated before the test. Basically, only the login shell and the budget_av module were left. 1 minute uptime was used for measurements.
Kernel: Linux 2.6.25
HW: Athlon 64 X2 3800+, Satelco EasyWatch DVB-C (as before)
1. Original module budget_av is loaded:
Load: ~0,6-0,8
2. Module + Patch "saa7146_sleep.diff" (1ms/10ms polling intervals in debi_done function):
Load: ~0,6-0,8 (same as in 1., no difference visible)
3. Module + Patch "incr-empty-ca-slot-poll-2.6.24.4.patch" (5s polling timer on slot state EMPTY):
Load: Decays to 0,02, but after about 105s always a spike to 0,10, then again decays to 0,02, and so on
4. Module + Patch "incr-empty-ca-slot-poll" + "saa7146_sleep.diff":
Same as 3.
5. Module + Patch "incr-empty-ca-slot-poll" + "binary exponential backoff":
Same as 3.
6. Module budget_av is unloaded:
Load constantly stays at 0, no spikes
Bottomline for me:
- Increasing the poll timer from 100ms, e.g. to 5s, makes sense. Changing the polling intervals in the debi_done function, however, makes no difference (unlike my previous assumption which was caused by the ugly variations).
- There seems to be a spike in the CPU load, each ~105s, but only when budget_av is loaded. I cannot explain it (maybe some frontend background functions), but it is no issue for me.
Regards,
Robert
_______________________________________________
linux-dvb mailing list
[email protected]
http://www.linuxtv.org/cgi-bin/mailman/listinfo/linux-dvb
Quoted 35 lines · Robert Schedel
Quoted 34 lines · Oliver Endriss
Quoted 33 lines · Robert Schedel
Hello,
on 2.6.24 I run into a small issue with the
budget_avdriver:Hardware: Athlon 64X2 3800+, Satelco Easywatch DVB-C (1894:002c), no CI/CAM
Software: Linux kernel 2.6.24 (gentoo-r4) SMPx86_64,budget_avmoduleError description:
After thebudget_avdriver module is loaded (even without any DVB application), the CPU load average in 'top' rises to ~1, but in top no active tasks are found. After unloading the driver, the load decreases again to ~0.Displaying the blocked tasks during high load with Alt-SysRq-W shows that the kdvb-ca kernel thread seems to be accounted as blocked when it polls for the CI slot status:
---------------------------------------------------
[...]
---------------------------------------------------Enabling debug traces shows that polling for the PSR in function '
saa7146_wait_for_debi_done_sleep' constantly times out after 250x1ms sleeps:Quoted 1 line
saa7146:
saa7146_wait_for_debi_done_sleep(): saa7146 (0):
saa7146_wait_for_debi_done_sleeptimed out while waiting for transfer completionIncreasing the 250ms did not avoid the timeout. And as I understood from previous list mails, this timeout is normal when no CI/CAM is connected to the DEBI. However, for me the high frequency polling does not make sense if someone does not plan to plug in a CI/CAM.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
When commenting out two lines in '
dvb_ca_en50221_thread_update_delay' to increase the polling timer for slotstate NONE from 100ms (!) to 60s, the CPU load went down to 0. So this is some kind of workaround for me.Afaics the polling interval could be increased to something like 5s or 10s if (and only if) the slot is empty. Could you provide a patch?
Attached a patch for 2.6.24.4. Opinions?
Basically it should work but it has to be tested with CI/CAM, too. Furthermore it is not sufficient to test with budget-av because many other drivers will be affected.
So I would prefer a patch which does not touch behaviour for other card drivers (if possible).
Please note for 'final' patches:
Always run 'checkpatch.pl' and fix the errors.
Sorry for that. :-(
Quoted 25 lines
Unfortunately, a 5s poll interval for empty slots still results in a load average of about 0,06 (~ 250ms/5s).
Increasing to 10s would decrease the load and be fine for people without CAM, but increase the delay for those users inserting CAMs during runtime. 5s sounds like a reasonable tradeoff.
Is the 250ms timeout an approved limit? Decreasing it would push the load further down. Probably it still has to cover slow CAMs as well as a stressed PCI bus. Unfortunately, without CAM/CI I cannot make any statements myself.
Quoted 13 lines
Quoted 3 lines
Finally, my questions:
1. Did I understand this right, that the behaviour above is expected when no CI is connected?Yes, but afaics the polling interval is way too short.
Quoted 2 lines
2. Are all
budget_avcards unable to check CAM slot state via interrupt for HW reasons (asbudget_cidoes)?I don't have any budget-av hardware, so I don't know.
But I think that Andrew(?) had a good reason to implement it this way. (In contrast the budget-ci driver was always interrupt-driven.)
Quoted 6 lines
Quoted 2 lines
If someone finds out that a given card can operate in interrupt mode, it should be changed for this card. Patches are welcome. ;-)
My impression is, due to the different GPIO layout there is no way to get a CAM change IRQ. But it seems difficult to get information about the HW architecture of cards at all.
For most cards we have no hw info. :-(
Quoted 7 lines
Regarding
DEBI_E: Just found a av7110 code comment which also reflects what my recent tests showed:/* Note: Don't try to handle the DEBI error irq (MASK_18), in * intel mode the timeout is asserted all the time... */So really only
DEBI_Swould be left, see below.
Did you check whether DEBI_S and/or DEBI_E are ever asserted with your setup? If not, an interrupt would never occur anyway...
Quoted 20 lines
Quoted 10 lines
Quoted 5 lines
3. Would it be possible to substitute the current PSR
DEBI_Spolling with an interrupt based solution via IER/ISR? (driver av7110 alreadys seems to do this for its DEBI DMA)? Or was this not considered worthy, due to the expected short waiting period and the tricky IER handling? The code does not state further thoughts about this.The av7110 driver uses interrupt mode for buffer transfers in dma mode. It does not make much sense to use interrupt mode for single-word transfers, because the single-word transfers are very fast. But I understand that the timeout causes a problem in this case.
OK, interrupts of course decrease performance in the "sunny day" cases (=communication with inserted card in slot state READY). Having both approaches (interrupt when slotstate empty, later polling) would combine all benefits but also be somewhat crazy.
Quoted 4 lines
Quoted 3 lines
4. Are the high timeout periods in
debi_done(50ms/250ms) in relation to the 100ms poll timer intended? (I found the recent patch to this code in the mailing list end of last year)
Quoted 7 lines
Quoted 4 lines
That patch was applied to reduce the load on the pci bus in busy-wait mode. Basically it did not change anything for cam polling. (In fact I was not aware that the CAM was polled every 100ms. Imho this should be fixed.)
Only wondered whether the 250ms might have been smaller in former driver versions.
Iirc it should be even worse with older drivers.
Basically the 250ms timeout is just a last resort to escape from the loop, if the debi transfer hangs for some reason. We might try to reduce the timeout but I don't know how far we can go. (Touching 'magic' values might be dangeous.)
CU
Oliver
Signature · 8 lines
---------------------------------------------------------------- VDR Remote Plugin 0.4.0: http://www.escape-edv.de/endriss/vdr/ ---------------------------------------------------------------- _______________________________________________ linux-dvb mailing list [email protected] http://www.linuxtv.org/cgi-bin/mailman/listinfo/linux-dvb
Am Freitag, den 18.04.2008, 03:21 +0200 schrieb Oliver Endriss:
Quoted 131 lines
Quoted 35 lines · Robert Schedel
Quoted 34 lines · Oliver Endriss
Quoted 33 lines · Robert Schedel
Hello,
on 2.6.24 I run into a small issue with the
budget_avdriver:Hardware: Athlon 64X2 3800+, Satelco Easywatch DVB-C (1894:002c), no CI/CAM
Software: Linux kernel 2.6.24 (gentoo-r4) SMPx86_64,budget_avmoduleError description:
After thebudget_avdriver module is loaded (even without any DVB application), the CPU load average in 'top' rises to ~1, but in top no active tasks are found. After unloading the driver, the load decreases again to ~0.Displaying the blocked tasks during high load with Alt-SysRq-W shows that the kdvb-ca kernel thread seems to be accounted as blocked when it polls for the CI slot status:
---------------------------------------------------
[...]
---------------------------------------------------Enabling debug traces shows that polling for the PSR in function '
saa7146_wait_for_debi_done_sleep' constantly times out after 250x1ms sleeps:Quoted 1 line
saa7146:
saa7146_wait_for_debi_done_sleep(): saa7146 (0):
saa7146_wait_for_debi_done_sleeptimed out while waiting for transfer completionIncreasing the 250ms did not avoid the timeout. And as I understood from previous list mails, this timeout is normal when no CI/CAM is connected to the DEBI. However, for me the high frequency polling does not make sense if someone does not plan to plug in a CI/CAM.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
When commenting out two lines in '
dvb_ca_en50221_thread_update_delay' to increase the polling timer for slotstate NONE from 100ms (!) to 60s, the CPU load went down to 0. So this is some kind of workaround for me.Afaics the polling interval could be increased to something like 5s or 10s if (and only if) the slot is empty. Could you provide a patch?
Attached a patch for 2.6.24.4. Opinions?
Basically it should work but it has to be tested with CI/CAM, too. Furthermore it is not sufficient to test with budget-av because many other drivers will be affected.
So I would prefer a patch which does not touch behaviour for other card drivers (if possible).
Please note for 'final' patches:
Always run 'checkpatch.pl' and fix the errors.
Sorry for that. :-(Quoted 25 lines
Unfortunately, a 5s poll interval for empty slots still results in a load average of about 0,06 (~ 250ms/5s).
Increasing to 10s would decrease the load and be fine for people without CAM, but increase the delay for those users inserting CAMs during runtime. 5s sounds like a reasonable tradeoff.
Is the 250ms timeout an approved limit? Decreasing it would push the load further down. Probably it still has to cover slow CAMs as well as a stressed PCI bus. Unfortunately, without CAM/CI I cannot make any statements myself.
Quoted 13 lines
Quoted 3 lines
Finally, my questions:
1. Did I understand this right, that the behaviour above is expected when no CI is connected?Yes, but afaics the polling interval is way too short.
Quoted 2 lines
2. Are all
budget_avcards unable to check CAM slot state via interrupt for HW reasons (asbudget_cidoes)?I don't have any budget-av hardware, so I don't know.
But I think that Andrew(?) had a good reason to implement it this way. (In contrast the budget-ci driver was always interrupt-driven.)Quoted 6 lines
Quoted 2 lines
If someone finds out that a given card can operate in interrupt mode, it should be changed for this card. Patches are welcome. ;-)
My impression is, due to the different GPIO layout there is no way to get a CAM change IRQ. But it seems difficult to get information about the HW architecture of cards at all.
For most cards we have no hw info. :-(
Quoted 7 lines
Regarding
DEBI_E: Just found a av7110 code comment which also reflects what my recent tests showed:/* Note: Don't try to handle the DEBI error irq (MASK_18), in * intel mode the timeout is asserted all the time... */So really only
DEBI_Swould be left, see below.Did you check whether
DEBI_Sand/orDEBI_Eare ever asserted with your setup? If not, an interrupt would never occur anyway...Quoted 20 lines
Quoted 10 lines
Quoted 5 lines
3. Would it be possible to substitute the current PSR
DEBI_Spolling with an interrupt based solution via IER/ISR? (driver av7110 alreadys seems to do this for its DEBI DMA)? Or was this not considered worthy, due to the expected short waiting period and the tricky IER handling? The code does not state further thoughts about this.The av7110 driver uses interrupt mode for buffer transfers in dma mode. It does not make much sense to use interrupt mode for single-word transfers, because the single-word transfers are very fast. But I understand that the timeout causes a problem in this case.
OK, interrupts of course decrease performance in the "sunny day" cases (=communication with inserted card in slot state READY). Having both approaches (interrupt when slotstate empty, later polling) would combine all benefits but also be somewhat crazy.
Quoted 4 lines
Quoted 3 lines
4. Are the high timeout periods in
debi_done(50ms/250ms) in relation to the 100ms poll timer intended? (I found the recent patch to this code in the mailing list end of last year)Quoted 7 lines
Quoted 4 lines
That patch was applied to reduce the load on the pci bus in busy-wait mode. Basically it did not change anything for cam polling. (In fact I was not aware that the CAM was polled every 100ms. Imho this should be fixed.)
Only wondered whether the 250ms might have been smaller in former driver versions.
Iirc it should be even worse with older drivers.
Quoted 8 lines
Basically the 250ms timeout is just a last resort to escape from the loop, if the debi transfer hangs for some reason. We might try to reduce the timeout but I don't know how far we can go. (Touching 'magic' values might be dangeous.)
CU
Oliver
Hi,
what are magic values?
Simply something retrieved under NDA with even that hardware not in established/stable conditions yet, most likely.
If the commercial research facilities are not established for that, because too expensive, it likely goes out to some university or major research project around the globe, which recently collected some external money for something related, to stay alive, and then please could solve that minor issue on the run for free.
We had lots of high quality contributions in the past, seemingly coming out of nothing ...
If it really counts, Open Source is a must and everybody knows it.
Cheers,
Hermann
_______________________________________________
linux-dvb mailing list
[email protected]
http://www.linuxtv.org/cgi-bin/mailman/listinfo/linux-dvb
Quoted 9 lines · Oliver Endriss
Quoted 8 lines · Robert Schedel
Quoted 7 lines · Oliver Endriss
Quoted 6 lines · Robert Schedel
Enabling debug traces shows that polling for the PSR in function '
saa7146_wait_for_debi_done_sleep' constantly times out after 250x1ms sleeps:Quoted 1 line
saa7146:
saa7146_wait_for_debi_done_sleep(): saa7146 (0):
saa7146_wait_for_debi_done_sleeptimed out while waiting for transfer completion
Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
Increasing the 250ms did not avoid the timeout. And as I understood from previous list mails, this timeout is normal when no CI/CAM is connected to the DEBI. However, for me the high frequency polling does not make sense if someone does not plan to plug in a CI/CAM.
When commenting out two lines in '
dvb_ca_en50221_thread_update_delay' to increase the polling timer for slotstate NONE from 100ms (!) to 60s, the CPU load went down to 0. So this is some kind of workaround for me.Afaics the polling interval could be increased to something like 5s or 10s if (and only if) the slot is empty. Could you provide a patch?
Attached a patch for 2.6.24.4. Opinions?
Basically it should work but it has to be tested with CI/CAM, too.
Correct, unfortunately I cannot test it against a CI.
Quoted 5 lines
Furthermore it is not sufficient to test with budget-av because many other drivers will be affected.
So I would prefer a patch which does not touch behaviour for other card drivers (if possible).
To my understanding of the DVB code dvb_ca_en50221 is only referenced by budget_av and budget_ci, at least in the vanilla kernel 2.6.25. The patch only changes the timer for slot state EMPTY if DVB_CA_EN50221_FLAG_IRQ_CAMCHANGE is not set, which is for
1) budget_av
2) budget_ci if the CI firmware version is 0xa2 (because IRQs for CAM change are not supported in this version)
And those two cases are probably affected by the load issue and should be fixed.
Of course, we could add another DVB_CA_EN50221 flag solely for budget_av to exclude case 2), but does this make sense? Who is available to test against a budget_ci with FW=0xa2 whether it is affected by the load issue?
Quoted 3 lines
Please note for 'final' patches:
Always run 'checkpatch.pl' and fix the errors.
Sorry for that. :-(
Sorry, forgot to call the check script. Attached a revised patch (against 2.6.25).
Quoted 10 lines
Quoted 7 lines
Regarding
DEBI_E: Just found a av7110 code comment which also reflects what my recent tests showed:/* Note: Don't try to handle the DEBI error irq (MASK_18), in * intel mode the timeout is asserted all the time... */So really only
DEBI_Swould be left, see below.Did you check whether
DEBI_Sand/orDEBI_Eare ever asserted with your setup? If not, an interrupt would never occur anyway...
DEBI_E was always asserted (as described in the av7110 code comment), so it was worthless. DEBI_S was never asserted without CI (therefore the 250ms timeout), so it would probably only be received when a CI is used. But as described in my other email with measurements, it seems that there is no need to optimize the debi_done function further.
Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
4. Are the high timeout periods in
debi_done(50ms/250ms) in relation to the 100ms poll timer intended? (I found the recent patch to this code in the mailing list end of last year)That patch was applied to reduce the load on the pci bus in busy-wait mode. Basically it did not change anything for cam polling. (In fact I was not aware that the CAM was polled every 100ms. Imho this should be fixed.)
Only wondered whether the 250ms might have been smaller in former driver versions.
Quoted 6 lines
Iirc it should be even worse with older drivers.
Basically the 250ms timeout is just a last resort to escape from the loop, if the debi transfer hangs for some reason. We might try to reduce the timeout but I don't know how far we can go. (Touching 'magic' values might be dangeous.)
As above, according to my measurements we would not need to change the 250ms timeout.
Regards,
Robert
From: Robert Schedel <[email protected]>
This change addresses kernel bug #10459: In kernel 2.6.25 the budget_av driver polls for an CI slot in 100ms intervals (because no interrupt solution for budget_av cards is feasible due to HW reasons). If no CI/CAM is connected to the DVB card, polling times out only after 250ms. This periodic polling leads to high CPU load.
The change increases the polling interval for empty slots from 100ms to 5s. Intervals for remaining slot states (invalid, in progress, ready) are unchanged, as they are either temporary conditions or no timeout should occur.
--- drivers/media/dvb/dvb-core/dvb_ca_en50221.c | 28 ++++++++++-------- 1 file changed, 16 insertions(+), 12 deletions(-)
drivers/media/dvb/dvb-core/dvb_ca_en50221.c +16 -12
@@ -910,15 +910,21 @@ static void dvb_ca_en50221_thread_updateint curdelay =100000000 ;int slot; +/* Beware of too high polling frequency, because one polling +* call might take several hundred milliseconds until timeout! +*/ for (slot =0 ; slot < ca->slot_count; slot++) {switch (ca->slot_info[slot].slot_state) {default :case DVB_CA_SLOTSTATE_NONE : + delay =HZ *60 ;/* 60s */ +if (!(ca->flags &DVB_CA_EN50221_FLAG_IRQ_CAMCHANGE )) + delay =HZ *5 ;/* 5s */ +break ;case DVB_CA_SLOTSTATE_INVALID : - delay =HZ *60 ; -if (!(ca->flags &DVB_CA_EN50221_FLAG_IRQ_CAMCHANGE )) { - delay =HZ /10 ; - } + delay =HZ *60 ;/* 60s */ +if (!(ca->flags &DVB_CA_EN50221_FLAG_IRQ_CAMCHANGE )) + delay =HZ /10 ;/* 100ms */ break ;case DVB_CA_SLOTSTATE_UNINITIALISED :
@@ -926,19 +932,17 @@ static void dvb_ca_en50221_thread_updatecase DVB_CA_SLOTSTATE_VALIDATE :case DVB_CA_SLOTSTATE_WAITFR :case DVB_CA_SLOTSTATE_LINKINIT : - delay =HZ /10 ; + delay =HZ /10 ;/* 100ms */ break ;case DVB_CA_SLOTSTATE_RUNNING : - delay =HZ *60 ; -if (!(ca->flags &DVB_CA_EN50221_FLAG_IRQ_CAMCHANGE )) { - delay =HZ /10 ; - } + delay =HZ *60 ;/* 60s */ +if (!(ca->flags &DVB_CA_EN50221_FLAG_IRQ_CAMCHANGE )) + delay =HZ /10 ;/* 100ms */ if (ca->open) {if ((!ca->slot_info[slot].da_irq_supported) || - (!(ca->flags &DVB_CA_EN50221_FLAG_IRQ_DA ))) { - delay =HZ /10 ; - } + (!(ca->flags &DVB_CA_EN50221_FLAG_IRQ_DA ))) + delay =HZ /10 ;/* 100ms */ }break ; }
_______________________________________________
linux-dvb mailing list
[email protected]
http://www.linuxtv.org/cgi-bin/mailman/listinfo/linux-dvb
Quoted 21 lines · Robert Schedel
Quoted 20 lines · Oliver Endriss
Quoted 19 lines · Robert Schedel
Quoted 18 lines · Oliver Endriss
Quoted 17 lines · Robert Schedel
Enabling debug traces shows that polling for the PSR in function '
saa7146_wait_for_debi_done_sleep' constantly times out after 250x1ms sleeps:Quoted 1 line
saa7146:
saa7146_wait_for_debi_done_sleep(): saa7146 (0):
saa7146_wait_for_debi_done_sleeptimed out while waiting for transfer completionIncreasing the 250ms did not avoid the timeout. And as I understood from previous list mails, this timeout is normal when no CI/CAM is connected to the DEBI. However, for me the high frequency polling does not make sense if someone does not plan to plug in a CI/CAM.
When commenting out two lines in '
dvb_ca_en50221_thread_update_delay' to increase the polling timer for slotstate NONE from 100ms (!) to 60s, the CPU load went down to 0. So this is some kind of workaround for me.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
Afaics the polling interval could be increased to something like 5s or 10s if (and only if) the slot is empty. Could you provide a patch?
Attached a patch for 2.6.24.4. Opinions?
Basically it should work but it has to be tested with CI/CAM, too.
Correct, unfortunately I cannot test it against a CI.
@all:
Could someone who runs budget-av with a CAM please test Robert's patch? Thanks!
Quoted 8 lines
Quoted 5 lines
Furthermore it is not sufficient to test with budget-av because many other drivers will be affected.
So I would prefer a patch which does not touch behaviour for other card drivers (if possible).
To my understanding of the DVB code
dvb_ca_en50221is only referenced bybudget_avandbudget_ci, at least in the vanilla kernel 2.6.25.
You are right (and I am a bit surprised).
Quoted 6 lines
The patch only changes the timer for slot state EMPTY if
DVB_CA_EN50221_FLAG_IRQ_CAMCHANGEis not set, which is for
1)budget_av
2)budget_ciif the CI firmware version is0xa2(because IRQs for CAM change are not supported in this version)
Quoted 6 lines
And those two cases are probably affected by the load issue and should be fixed.
Of course, we could add another
DVB_CA_EN50221flag solely forbudget_avto exclude case 2), but does this make sense? Who is available to test against abudget_ciwith FW=0xa2 whether it is affected by the load issue?
After looking closer at your patch I saw that most changes are comments or coding-style related. Afaics the patch should be safe.
So I will commit your patch next weekend, unless someone spots a problem.
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Regarding
DEBI_E: Just found a av7110 code comment which also reflects what my recent tests showed:/* Note: Don't try to handle the DEBI error irq (MASK_18), in * intel mode the timeout is asserted all the time... */So really only
DEBI_Swould be left, see below.
Quoted 8 lines
Quoted 2 lines
Did you check whether
DEBI_Sand/orDEBI_Eare ever asserted with your setup? If not, an interrupt would never occur anyway...
DEBI_Ewas always asserted (as described in the av7110 code comment), so it was worthless.DEBI_Swas never asserted without CI (therefore the 250ms timeout), so it would probably only be received when a CI is used. But as described in my other email with measurements, it seems that there is no need to optimize thedebi_donefunction further.
Ok, thanks for checking.
Quoted 12 lines
Quoted 12 lines
Quoted 9 lines
Quoted 7 lines
Quoted 3 lines
4. Are the high timeout periods in
debi_done(50ms/250ms) in relation to the 100ms poll timer intended? (I found the recent patch to this code in the mailing list end of last year)That patch was applied to reduce the load on the pci bus in busy-wait mode. Basically it did not change anything for cam polling. (In fact I was not aware that the CAM was polled every 100ms. Imho this should be fixed.)
Only wondered whether the 250ms might have been smaller in former driver versions.
Iirc it should be even worse with older drivers.
Quoted 7 lines
Quoted 4 lines
Basically the 250ms timeout is just a last resort to escape from the loop, if the debi transfer hangs for some reason. We might try to reduce the timeout but I don't know how far we can go. (Touching 'magic' values might be dangeous.)
As above, according to my measurements we would not need to change the 250ms timeout.
Agreed.
CU
Oliver
Signature · 8 lines
---------------------------------------------------------------- VDR Remote Plugin 0.4.0: http://www.escape-edv.de/endriss/vdr/ ---------------------------------------------------------------- _______________________________________________ linux-dvb mailing list [email protected] http://www.linuxtv.org/cgi-bin/mailman/listinfo/linux-dvb