Skip to contentSkip to conversation

tvtime and the Linux 2.6 scheduler

DISCUSSION Billy Biggs 10 participants 16 messages May 23, 2004 – May 27, 2004

lore
Share & tools
lore
#0 lore raw

I am the author of tvtime, a TV application with advanced image
processing algorithms. Some users are complaining about poor performance under Linux 2.6, and I would like more information about how tvtime will be treated by the scheduler. Here is an example of the intended usage:

- Program running as root and SCHED_FIFO
- NTSC, input ~30 fps, each field processed for an output of ~60 fps
- CPU intensive processing, say 9 ms per field on my P3-733
- with a typical AGP card, the X driver takes 4 ms to draw
- Wait using /dev/rtc set to 1024 Hz

for(;;)
9 ms : process frame
4 ms : draw frame
3 ms : wait until next field time using /dev/rtc 9 ms : process frame
4 ms : draw frame
3 ms : block on /dev/video0 for next frame
-----
33 ms : time per NTSC frame

The theory is that Linux classifies this as a CPU hog regardless of
its priority, and preempts tvtime with other processes. Oswald Buddenhagen describes the effect as this:

"[...] it starts up fine, but after a few seconds (when the scheduler
gathered some stats) ... well, it looks funny: the scene goes roughly exponentially into slow motion, then there is a frame drop and the process starts over. this behaviour can be observed at any priority, which is clearly against the claim "no normally priorized interactive process will preempt a highly priorized cpu-hog" that i've read somewhere. the xserver priority does not change anything, either;"

Avoiding root/SCHED_FIFO and using usleep() instead of /dev/rtc seems
to exhibit the same behavior.

Thoughts?

-Billy

#1 lore raw
Quoted 8 lines · Billy Biggs

"[...] it starts up fine, but after a few seconds (when the scheduler
gathered some stats) ... well, it looks funny: the scene goes roughly exponentially into slow motion, then there is a frame drop and the process starts over. this behaviour can be observed at any priority, which is clearly against the claim "no normally priorized interactive process will preempt a highly priorized cpu-hog" that i've read somewhere. the xserver priority does not change anything, either;"

I am currently using tvtime 0.9.12-2 from Debian SID on a Linux kernel version 2.6.6 compiled with Sid's gcc 3.3.3 and haven't seen such problems so far. I use tvtime often to watch TV on my monitor, and it works OK, no artifacts and no slowdown (even when I download my email and spamassassin starts to eat CPU cicles like mad).

With my current configuration tvtime at full screen (1024x768) takes about 40% CPU from a AMD XP 1477 MHz (1700 rating). I have just tried launching three copies of "yes" on several aterm's and tvtime still looks as smooth as before, althoug CPU is always at 100%

Please ask for more information should you need it.

Signature · 2 lines
Jose Luis Domingo Lopez
Linux Registered User #189436     Debian Linux Sid (Linux 2.6.6)
#2 lore raw
In reply to Billy Biggs

Hi

Quoted 7 lines · Billy Biggs

I am the author of tvtime, a TV application with advanced image
processing algorithms. Some users are complaining about poor performance under Linux 2.6, and I would like more information about how tvtime will be treated by the scheduler. Here is an example of the intended usage:

- Program running as root and SCHED_FIFO

snip

Quoted 1 line

33 ms : time per NTSC frame

snip

The followup email from someone describing good performance may help us understand what's going on. Your example of poor performance is one when the cpu performance is marginal to get exactly 30 fps processed and on the screen. The cpu overhead in 2.6 is slightly higher than 2.4 so a borderline case may be just pushed over.

A program running as sched_fifo it will preempt absolutely everything regardless of how it behaves. It sounds like it's giving X less cpu time to draw the frame each time until eventually the processing fails to capture the frame and then X smooths out again. I cant pretend to understand how your application blocks (as you say) between X and tvtime, but does tvtime not try to schedule until X has finished using up cpu or will it just run off the timer and preempt X away? You say changing priorities doesnt help, but I cant tell if you tried this: run the processing sched_normal at lower priority than X.

Con

#3 lore raw

Hi Con, thanks for your reply.

Con Kolivas ([email protected]):

Quoted 9 lines

A program running as sched_fifo it will preempt absolutely everything regardless of how it behaves. It sounds like it's giving X less cpu time to draw the frame each time until eventually the processing fails to capture the frame and then X smooths out again. I cant pretend to understand how your application blocks (as you say) between X and tvtime, but does tvtime not try to schedule until X has finished using up cpu or will it just run off the timer and preempt X away? You say changing priorities doesnt help, but I cant tell if you tried this:
run the processing sched_normal at lower priority than X.

We call XvShmPutImage and then do an XSync, which will wait until X
sends a message indicating it has processed all outstanding requests. The XvShmPutImage uploads the image to video memory immediately, so the call to XSync blocks us until it is complete. We are not preempting the X server.

I will get accurate information about the priorities of all processes
involved here.

-Billy

#4 lore raw
In reply to Con Kolivas
Quoted 5 lines · Con Kolivas

Your example of poor performance is one when the cpu performance is marginal to get exactly 30 fps processed and on the screen. The cpu overhead in 2.6 is slightly higher than 2.4 so a borderline case may be just pushed over.

"his" example is in fact mine. 2.4 ghz chew 25 fps. the total cpu load is at 30-40% (depending on the selected deinterlacer algorithm).

Quoted 3 lines

A program running as sched_fifo it will preempt absolutely everything regardless of how it behaves.

errm ... right. i paid too little attention to this "tiny" detail. *blush*

now i hacked tvtime to simply ignore v4l2 ... and guess what? it works
...
so the new theory is, that tvtime has some problem with v4l2 input buffering ...

greetings

Signature · 3 lines
Hi! I'm a .signature virus! Copy me into your ~/.signature, please!
--
Chaos, panic, and disorder - my work here is done.
#10 lore raw
In reply to Billy Biggs

--- Billy Biggs <[email protected]> a écrit : > I am the
author of tvtime, a TV application with advanced

Quoted 46 lines

image
processing algorithms. Some users are complaining about poor performance under Linux 2.6, and I would like more information about how
tvtime will be treated by the scheduler. Here is an example of the
intended usage:

- Program running as root and SCHED_FIFO
- NTSC, input ~30 fps, each field processed for an output of
~60 fps
- CPU intensive processing, say 9 ms per field on my P3-733
- with a typical AGP card, the X driver takes 4 ms to draw
- Wait using /dev/rtc set to 1024 Hz

for(;;)
9 ms : process frame
4 ms : draw frame
3 ms : wait until next field time using /dev/rtc 9 ms : process frame
4 ms : draw frame
3 ms : block on /dev/video0 for next frame
-----
33 ms : time per NTSC frame

The theory is that Linux classifies this as a CPU hog
regardless of
its priority, and preempts tvtime with other processes. Oswald
Buddenhagen describes the effect as this:

"[...] it starts up fine, but after a few seconds (when the
scheduler
gathered some stats) ... well, it looks funny: the scene goes roughly
exponentially into slow motion, then there is a frame drop and the
process starts over. this behaviour can be observed at any priority,
which is clearly against the claim "no normally priorized interactive
process will preempt a highly priorized cpu-hog" that i've read
somewhere. the xserver priority does not change anything, either;"

Quoted 6 lines

Avoiding root/SCHED_FIFO and using usleep() instead of
/dev/rtc seems
to exhibit the same behavior.

Thoughts?

For me it works ok
Running tvtime with 2.6.6-mm5 load average 26 (twenty six) it works
almost perfect (no frame skipped -- at least not reported by tvtime)
at full rate (50 frames/sec). Computer: AMD Duron 700 MHz, 256MB Ram,
VIA KT133 chipset, AGP 4x, Ati Radeon 7200

Maybe your users are renicing tvtime ?
AFAIK it is not recommended to renice a process which uses X server and
wants realtime processing

Ohhhhhhhhhh
Take a look at this. Is tvtime's fault (X server does'n have the chance to display what tvtime processed because tvtime gets almost all the cpu)

sony@grinch -01:40:31- 0 jobs, ver 2.05b.0 3
/~ $ tvtime -s -v
Running tvtime 0.9.12.
Reading configuration from /usr/local/etc/tvtime/tvtime.xml Reading configuration from /home/sony/.tvtime/tvtime.xml
cpuinfo: CPU AMD Duron(tm) Processor, family 6, model 3, stepping 1.
cpuinfo: CPU measured at 700,579MHz.
tvtime: Cannot set priority to -19: Permission denied. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

xcommon: Display :0.0, vendor The XFree86 Project, Inc, XFree86 4.4.0
xfullscreen: Single-head detected, pixel aspect will be calculated.
xfullscreen: Pixel aspect ratio on the primary head is: 75/76 == 0,99.
xfullscreen: Using the XFree86-VidModeExtension to calculate fullscreen size.
xfullscreen: Fullscreen to 0,0 with size 1024x768.
xcommon: Have XTest, will use it to ping the screensaver.
xcommon: Pixel aspect ratio 75:76.
xcommon: No window properties found for EWMH.
xcommon: Window manager is not EWMH compliant.
xcommon: Pixel aspect ratio 75:76.
xcommon: Displaying in a 778x576 window inside 778x576 space.
xvoutput: Using XVIDEO adaptor 69: ATI Radeon Video Overlay.
speedycode: Using MMXEXT optimized functions.
station: Reading stationlist from /home/sony/.tvtime/stationlist.xml
videoinput: Using video4linux2 driver 'bttv', card 'BT878 video (AVerMedia TVCaptur' (bus PCI:0000:00:09.0).
videoinput: Version is 2318, capabilities 5010015.
tvtime: Sampling input at 720 pixels per scanline.
xcommon: Pixel aspect ratio 75:76.
xcommon: Displaying in a 778x576 window inside 778x576 space.
xcommon: Received a map, marking window as visible (50)

I hope the line above answers your question

Quoted 2 lines

-Billy

Calin

P.S. Load average is that high because i run freenet
http://freenet.sourceforge.net -- lots of java processes

=====
--
A mouse is a device used to point at
the xterm you want to type in.
Kim Alm on a.s.r.

Yahoo! Mail : votre e-mail personnel et gratuit qui vous suit partout ! Créez votre Yahoo! Mail sur http://fr.benefits.yahoo.com/

Dialoguez en direct avec vos amis grâce à Yahoo! Messenger !Téléchargez Yahoo! Messenger sur http://fr.messenger.yahoo.com

#5 lore raw
In reply to Con Kolivas
Quoted 9 lines · Con Kolivas
Quoted 1 line

33 ms : time per NTSC frame

snip

The followup email from someone describing good performance may help us understand what's going on. Your example of poor performance is one when the cpu performance is marginal to get exactly 30 fps processed and on the screen. The cpu overhead in 2.6 is slightly higher than 2.4 so a borderline case may be just pushed over.

most of the cpu overhead comes from HZ=1000. Especial with SCHED_FIFO there should be minimal (if any) impact from the scheduler changes - SCHED_FIFO tasks get all CPU time, no ifs and whens.

could people who experience tvtime performance problems apply the patch below to change HZ back to 100? Does it have any impact?

Ingo

include/asm-i386/param.h renamed from include/asm-i386/param.h.orig +1 -1
@@ -2,7 +2,7 @@
 #define _ASMi386_PARAM_H
 
 #ifdef __KERNEL__
-# define HZ		1000		/* Internal kernel timer frequency */
+# define HZ		100		/* Internal kernel timer frequency */
 # define USER_HZ	100		/* .. some user interfaces are in "ticks" */
 # define CLOCKS_PER_SEC		(USER_HZ)	/* like times() */
 #endif
#6 lore raw
Quoted 14 lines · Ingo Molnar
Quoted 9 lines · Con Kolivas
Quoted 1 line

33 ms : time per NTSC frame

snip

The followup email from someone describing good performance may help us understand what's going on. Your example of poor performance is one when the cpu performance is marginal to get exactly 30 fps processed and on the screen. The cpu overhead in 2.6 is slightly higher than 2.4 so a borderline case may be just pushed over.

Quoted 7 lines

most of the cpu overhead comes from HZ=1000. Especial with SCHED_FIFO there should be minimal (if any) impact from the scheduler changes - SCHED_FIFO tasks get all CPU time, no ifs and whens.

could people who experience tvtime performance problems apply the patch below to change HZ back to 100? Does it have any impact?

Just one other thing - realtime scheduling was basically broken up until around 2.6.5. Before starting any tests, please ensure first that you are using at least the 2.6.5 kernel. Thanks.

#7 lore raw
Quoted 3 lines · Nick Piggin

Just one other thing - realtime scheduling was basically broken up until around 2.6.5. Before starting any tests, please ensure first that you are using at least the 2.6.5 kernel. Thanks.

you mean the spurious 'queue to end of prio-queue' bug noticed by Joe Korty? tvtime should not be affected by this one. This bug only hits if there are multiple SCHED_FIFO tasks on the same priority level - tvtime is a single-process application.

Ingo

#8 lore raw
Quoted 10 lines · Ingo Molnar
Quoted 3 lines · Nick Piggin

Just one other thing - realtime scheduling was basically broken up until around 2.6.5. Before starting any tests, please ensure first that you are using at least the 2.6.5 kernel. Thanks.

you mean the spurious 'queue to end of prio-queue' bug noticed by Joe Korty?

Yes I did mean that one.

tvtime should not be affected by this one. This bug only hits if

Quoted 3 lines

there are multiple SCHED_FIFO tasks on the same priority level - tvtime is a single-process application.

Yes I see.

#13 lore raw
In reply to Billy Biggs
Quoted 10 lines · Billy Biggs

for(;;)
9 ms : process frame
4 ms : draw frame
3 ms : wait until next field time using /dev/rtc 9 ms : process frame
4 ms : draw frame
3 ms : block on /dev/video0 for next frame
-----
33 ms : time per NTSC frame

Quoted 2 lines

The theory is that Linux classifies this as a CPU hog regardless of
its priority, and preempts tvtime with other processes. [...]

this would indicate a pretty broken scheduler. To prove (or exclude) this possibility, could you apply the attached debugging patch? The patch checks whether we ever switch away from a still running (and hence, on-runqueue) RT task to a non-RT task.

(NOTE: dont run this patch on an SMP kernel, the debugging message will deadlock there due to spinlock recursion.)

Ingo

kernel/sched.c renamed from kernel/sched.c.orig +1 -0
@@ -2279,6 +2279,7 @@ switch_tasks:
 	prev->timestamp = now;
 
 	if (likely(prev != next)) {
+		WARN_ON(rt_task(prev) && (prev->state == TASK_RUNNING) && !rt_task(next));
 		next->timestamp = now;
 		rq->nr_switches++;
 		rq->curr = next;
#9 lore raw
In reply to Ingo Molnar
Quoted 4 lines · Ingo Molnar

you mean the spurious 'queue to end of prio-queue' bug noticed by Joe Korty? tvtime should not be affected by this one. This bug only hits if there are multiple SCHED_FIFO tasks on the same priority level - tvtime is a single-process application.

i checked out the source and it uses multiple threads if recording is done to disk - but the disk writer thread is at normal priority, only the capture thread is using SCHED_FIFO.

Ingo

#14 lore raw
In reply to Ingo Molnar
Quoted 7 lines · Ingo Molnar
Quoted 2 lines · Billy Biggs

The theory is that Linux classifies this as a CPU hog regardless of
its priority, and preempts tvtime with other processes. [...]

this would indicate a pretty broken scheduler. To prove (or exclude) this possibility, could you apply the attached debugging patch?

done. no messages in the kernel syslog.

greetings

Signature · 3 lines
Hi! I'm a .signature virus! Copy me into your ~/.signature, please!
--
Chaos, panic, and disorder - my work here is done.
#11 lore raw
In reply to szonyi calin
Quoted 13 lines · szonyi calin

--- Billy Biggs <[email protected]> a écrit : > I am the
author of tvtime, a TV application with advanced

Quoted 10 lines

image
processing algorithms. Some users are complaining about poor performance under Linux 2.6, and I would like more information about how
tvtime will be treated by the scheduler. Here is an example of the
intended usage:

- Program running as root and SCHED_FIFO
- NTSC, input ~30 fps, each field processed for an output of ~60 fps

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^

Quoted 4 lines
Quoted 4 lines

- CPU intensive processing, say 9 ms per field on my P3-733
- with a typical AGP card, the X driver takes 4 ms to draw
- Wait using /dev/rtc set to 1024 Hz

Quoted 9 lines
Quoted 9 lines

for(;;)
9 ms : process frame
4 ms : draw frame
3 ms : wait until next field time using /dev/rtc 9 ms : process frame
4 ms : draw frame
3 ms : block on /dev/video0 for next frame
-----
33 ms : time per NTSC frame

	[___snip___]
Quoted 5 lines

For me it works ok
Running tvtime with 2.6.6-mm5 load average 26 (twenty six) it works almost perfect (no frame skipped -- at least not reported by tvtime)
at full rate (50 frames/sec). Computer: AMD Duron 700 MHz, 256MB

~~~~~~~~~~~~~~~~^^^

Quoted 2 lines

Ram,
VIA KT133 chipset, AGP 4x, Ati Radeon 7200

There have been pretty good explanations of this, so I'll just say that it might be that someone with a machine of limited capacity would be able to generate 50fps and not 60fps. In much of Europe the TV flickers at the same frequency as the lights ;-)

Signature · 3 lines
   -bill davidsen ([email protected])
"The secret to procrastination is to put things off until the
 last possible moment - but no longer"  -me
#12 lore raw
Quoted 4 lines · Bill Davidsen

There have been pretty good explanations of this, so I'll just say that it might be that someone with a machine of limited capacity would be able to generate 50fps and not 60fps. In much of Europe the TV flickers at the same frequency as the lights ;-)

Rather the other way around.

768*288*50*2 = 22118400
640*240*60000*2/1001 = 18413586

So the amount of data to process is higher for PAL (in square pixel resolution).

Signature · 3 lines
Tobias						PGP: http://9ac7e0bc.2ya.com
He said he hadn't had a byte in three days.
I had a short, so I split it with him.
#15 lore raw
In reply to Billy Biggs

using nicksched solves this, now tvtime only uses 1% again! yay!

Quoted 46 lines · Billy Biggs

I am the author of tvtime, a TV application with advanced image
processing algorithms. Some users are complaining about poor performance under Linux 2.6, and I would like more information about how tvtime will be treated by the scheduler. Here is an example of the intended usage:

- Program running as root and SCHED_FIFO
- NTSC, input ~30 fps, each field processed for an output of ~60 fps
- CPU intensive processing, say 9 ms per field on my P3-733
- with a typical AGP card, the X driver takes 4 ms to draw
- Wait using /dev/rtc set to 1024 Hz

for(;;)
9 ms : process frame
4 ms : draw frame
3 ms : wait until next field time using /dev/rtc 9 ms : process frame
4 ms : draw frame
3 ms : block on /dev/video0 for next frame
-----
33 ms : time per NTSC frame

The theory is that Linux classifies this as a CPU hog regardless of
its priority, and preempts tvtime with other processes. Oswald Buddenhagen describes the effect as this:

"[...] it starts up fine, but after a few seconds (when the scheduler
gathered some stats) ... well, it looks funny: the scene goes roughly exponentially into slow motion, then there is a frame drop and the process starts over. this behaviour can be observed at any priority, which is clearly against the claim "no normally priorized interactive process will preempt a highly priorized cpu-hog" that i've read somewhere. the xserver priority does not change anything, either;"

Avoiding root/SCHED_FIFO and using usleep() instead of /dev/rtc seems
to exhibit the same behavior.

Thoughts?

-Billy

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 2 lines
Regards, Redeeman
[email protected]