OMAP3 ISP - interlaced data incorrect
For days, I've been chasing ghosts :-) I know they are still there, but I think they are more a function of the source than the ISP setup. So, I went looking for a better source, NTSC in my case. My choice is is a DVD player with known good video (I'm convinced that my cheap NTSC camera produces crap, especially when there is a lot of motion in the frames). Looking at this on an analogue TV (yes, they still exist!), the picture is not bad, so I think it's a good choice, at least when trying to understand what's happening with the OMAP3 ISP.
Look at these two pictures:
http://www.mlbassoc.com/misc/nemo-00001.png
http://www.mlbassoc.com/misc/nemo-swapped-00001.png
These represent one frame of data captured via my OMAP3 ISP + TVP5150 from a DVD (sorry, Disney). The first is a raw conversion of the frame using ffmpeg. As you can see, there seem to be lines swapped, so I wrote a little program to swap the lines even/odd. The second (nemo-swapped) shows what this looks like. Obviously, the data is not being stored in memory correctly. Does anyone know how to adjust the ISP to make this work the right way around? Currently in ispccdc.c, we have:
ccdc_config_outlineoffset(ccdc, pix.bytesperline, EVENEVEN, 1); ccdc_config_outlineoffset(ccdc, pix.bytesperline, EVENODD, 1); ccdc_config_outlineoffset(ccdc, pix.bytesperline, ODDEVEN, 1); ccdc_config_outlineoffset(ccdc, pix.bytesperline, ODDODD, 1);
I tried this:
ccdc_config_outlineoffset(ccdc, pix.bytesperline, EVENEVEN, 2); ccdc_config_outlineoffset(ccdc, pix.bytesperline, EVENODD, 0); ccdc_config_outlineoffset(ccdc, pix.bytesperline, ODDEVEN, 2); ccdc_config_outlineoffset(ccdc, pix.bytesperline, ODDODD, 0);
but this lead to a kernel panic :-(
Somehow, we need to be storing the data something like this:
EE EE EE EE ...
EO EO EO EO ...
OE OE OE OE ...
OO OO OO OO ...
but the current layout is ccdc_config_outlineoffset(ccdc, pix.bytesperline, EVENEVEN, 1);
ccdc_config_outlineoffset(ccdc, pix.bytesperline, EVENODD, 1); ccdc_config_outlineoffset(ccdc, pix.bytesperline, ODDEVEN, 1); ccdc_config_outlineoffset(ccdc, pix.bytesperline, ODDODD, 1);
EO EO EO EO ...
EE EE EE EE ...
OO OO OO OO ...
OE OE OE OE ...
First, I need to get the data into memory in the correct order :-)
Note: these results are consistent, i.e. if I stop things and do another grab, they are incorrect in the same [wrong] order.
Note 2: I think I have explanations for much of the ghosting that has been observed:
* Frame to frame "tearing" results from the fact that the frames are interlaced and the actual data changes from one half of the frame to the other. This would not be observed with traditional media, e.g. film, where nothing can move from
one frame to the next because of the speed of shutters. In the purely digital
capture case, every pixel has the opportunity to change constantly and to find some that change with the 33ms (NTSC) window (1 frame) is quite probable. You can see some of this in the [otherwise quite good] sequence of images
http://www.mlbassoc.com/misc/nemo-swapped-00001.png
...
http://www.mlbassoc.com/misc/nemo-swapped-00062.png
* Frames being skipped &/or very stale data being reused - I think this is a [user] software problem. The ISP driver assumes that it always has an empty buffer to move captured data into. Depending on the [user] program which is consuming the data, this may or not be true. In the case of ffmpeg, if I capture raw images, ffmpeg can almost always keep up and there is always a free buffer. However, if I have ffmpeg turn the raw frames into compressed video (mp4), nearly 1/2 of the time, the ISP will run dry on buffers. I think I know how to fix this (untested) but it shows that some of the issues may be with the userland code we rely on. I've not done any recent tests with the gstreamer modules and the TI DSP code, but I will shortly. We'll see how well that does.
Note 3: The image viewer found at http://djv.sourceforge.net/ works great for these
analyses. Just run 'djv_view' and point at the first image in a sequence :-) n.b. I'm not associated with that project, but I really like it!
Signature · 4 lines
------------------------------------------------------------ Gary Thomas | Consulting for the MLB Associates | Embedded world ------------------------------------------------------------
Quoted 52 lines · Gary Thomas
For days, I've been chasing ghosts :-) I know they are still there, but I think they are more a function of the source than the ISP setup. So, I went looking for a better source, NTSC in my case. My choice is is a DVD player with known good video (I'm convinced that my cheap NTSC camera produces crap, especially when there is a lot of motion in the frames). Looking at this on an analogue TV (yes, they still exist!), the picture is not bad, so I think it's a good choice, at least when trying to understand what's happening with the OMAP3 ISP.
Look at these two pictures:
http://www.mlbassoc.com/misc/nemo-00001.png
http://www.mlbassoc.com/misc/nemo-swapped-00001.pngThese represent one frame of data captured via my OMAP3 ISP + TVP5150 from a DVD (sorry, Disney). The first is a raw conversion of the frame using ffmpeg. As you can see, there seem to be lines swapped, so I wrote a little program to swap the lines even/odd. The second (nemo-swapped) shows what this looks like. Obviously, the data is not being stored in memory correctly. Does anyone know how to adjust the ISP to make this work the right way around? Currently in
ispccdc.c, we have:ccdc_config_outlineoffset(ccdc, pix.bytesperline, EVENEVEN, 1);ccdc_config_outlineoffset(ccdc, pix.bytesperline, EVENODD, 1);ccdc_config_outlineoffset(ccdc, pix.bytesperline, ODDEVEN, 1);ccdc_config_outlineoffset(ccdc, pix.bytesperline, ODDODD, 1);I tried this:
ccdc_config_outlineoffset(ccdc, pix.bytesperline, EVENEVEN, 2);ccdc_config_outlineoffset(ccdc, pix.bytesperline, EVENODD, 0);ccdc_config_outlineoffset(ccdc, pix.bytesperline, ODDEVEN, 2);ccdc_config_outlineoffset(ccdc, pix.bytesperline, ODDODD, 0); but this lead to a kernel panic :-(Somehow, we need to be storing the data something like this:
EE EE EE EE ...
EO EO EO EO ...
OE OE OE OE ...
OO OO OO OO ...
but the current layout isccdc_config_outlineoffset(ccdc, pix.bytesperline, EVENEVEN, 1);ccdc_config_outlineoffset(ccdc, pix.bytesperline, EVENODD, 1);ccdc_config_outlineoffset(ccdc, pix.bytesperline, ODDEVEN, 1);ccdc_config_outlineoffset(ccdc, pix.bytesperline, ODDODD, 1);EO EO EO EO ...
EE EE EE EE ...
OO OO OO OO ...
OE OE OE OE ...First, I need to get the data into memory in the correct order :-)
Note: these results are consistent, i.e. if I stop things and do another grab, they are incorrect in the same [wrong] order.
Quoted 8 lines
Note 2: I think I have explanations for much of the ghosting that has been observed:
* Frame to frame "tearing" results from the fact that the frames are interlaced and the actual data changes from one half of the frame to the other. This would not be observed with traditional media, e.g. film, where nothing can move from one frame to the next because of the speed of shutters. In the purely digital capture case, every pixel has the opportunity to change constantly and to find some that change with the 33ms (NTSC) window (1 frame) is quite probable.
To be clear, moving from film to video, there would be no change of data within a single frame between the two interleaved halves. I'm sure this was even true of old cameras, which were not digital capture devices being used to send interleaved analogue data.
Quoted 4 lines
You can see some of this in the [otherwise quite good] sequence of images
http://www.mlbassoc.com/misc/nemo-swapped-00001.png
...
http://www.mlbassoc.com/misc/nemo-swapped-00062.png
Quoted 8 lines
* Frames being skipped &/or very stale data being reused - I think this is a [user] software problem. The ISP driver assumes that it always has an empty buffer to move captured data into. Depending on the [user] program which is consuming the data, this may or not be true. In the case of ffmpeg, if I capture raw images, ffmpeg can almost always keep up and there is always a free buffer. However, if I have ffmpeg turn the raw frames into compressed video (mp4), nearly 1/2 of the time, the ISP will run dry on buffers. I think I know how to fix this (untested) but it shows that some of the issues may be with the userland code we rely on.
In the case of ffmpeg capturing raw data, there were no times that the ISP driver wanted a buffer and failed to get one, at least when storing the frames in a RAM disk. If stored to a physical device like MMC card, this might not be true.
However, when ffmpeg is used to create an MP4 image, even to RAM disk, nearly 1/2 of the time the ISP goes wanting, which certainly can't be good!
Quoted 7 lines
I've not done any recent tests with the gstreamer modules and the TI DSP code, but I will shortly. We'll see how well that does.
Note 3: The image viewer found at http://djv.sourceforge.net/ works great for these analyses. Just run '
djv_view' and point at the first image in a sequence :-) n.b. I'm not associated with that project, but I really like it!
Signature · 4 lines
------------------------------------------------------------ Gary Thomas | Consulting for the MLB Associates | Embedded world ------------------------------------------------------------
Quoted 32 lines · Gary Thomas
To be clear, moving from film to video, there would be no change of data within a single frame between the two interleaved halves. I'm sure this was even true of old cameras, which were not digital capture devices being used to send interleaved analogue data.
Quoted 12 lines
You can see some of this in the [otherwise quite good] sequence of images
http://www.mlbassoc.com/misc/nemo-swapped-00001.png
...
http://www.mlbassoc.com/misc/nemo-swapped-00062.png
* Frames being skipped &/or very stale data being reused - I think this is a [user]
software problem. The ISP driver assumes that it always has an empty buffer to
move captured data into. Depending on the [user] program which is consuming the
data, this may or not be true. In the case of ffmpeg, if I capture raw images,Quoted 8 lines
ffmpeg can almost always keep up and there is always a free buffer. However, if
I have ffmpeg turn the raw frames into compressed video (mp4), nearly 1/2 of
the time, the ISP will run dry on buffers. I think I know how to fix this (untested)
but it shows that some of the issues may be with the userland code we rely on.In the case of ffmpeg capturing raw data, there were no times that the ISP driver
wanted a buffer and failed to get one, at least when storing the frames in a RAM
disk. If stored to a physical device like MMC card, this might not be true.
Quoted 3 lines
However, when ffmpeg is used to create an MP4 image, even to RAM disk, nearly 1/2
of the time the ISP goes wanting, which certainly can't be good!
That is to be expected, it's not good but the worst thing that could happen is that you will lose some frames, not fields (unless you are losing interrupts too). So you will get laggy video, but with no ghosting.
But i can't remember what the isp buffer code do when "out of buffer" so i'm assuming it will do a sane thing (drop the current frame).
Enrico
Quoted 57 lines · Gary Thomas
For days, I've been chasing ghosts :-) I know they are still there, but I think they are more a function of the source than the ISP setup. So, I went looking for a better source, NTSC in my case. My choice is is a DVD player with known good video (I'm convinced that my cheap NTSC camera produces crap, especially when there is a lot of motion in the frames). Looking at this on an analogue TV (yes, they still exist!), the picture is not bad, so I think it's a good choice, at least when trying to understand what's happening with the OMAP3 ISP.
Look at these two pictures:
http://www.mlbassoc.com/misc/nemo-00001.png
http://www.mlbassoc.com/misc/nemo-swapped-00001.pngThese represent one frame of data captured via my OMAP3 ISP + TVP5150 from a DVD (sorry, Disney). The first is a raw conversion of the frame using ffmpeg. As you can see, there seem to be lines swapped, so I wrote a little program to swap the lines even/odd. The second (nemo-swapped) shows what this looks like. Obviously, the data is not being stored in memory correctly. Does anyone know how to adjust the ISP to make this work the right way around? Currently in
ispccdc.c, we have:
ccdc_config_outlineoffset(ccdc, pix.bytesperline, EVENEVEN, 1);
ccdc_config_outlineoffset(ccdc, pix.bytesperline, EVENODD, 1);
ccdc_config_outlineoffset(ccdc, pix.bytesperline, ODDEVEN, 1);
ccdc_config_outlineoffset(ccdc, pix.bytesperline, ODDODD, 1);I tried this:
ccdc_config_outlineoffset(ccdc, pix.bytesperline, EVENEVEN, 2);
ccdc_config_outlineoffset(ccdc, pix.bytesperline, EVENODD, 0);
ccdc_config_outlineoffset(ccdc, pix.bytesperline, ODDEVEN, 2);
ccdc_config_outlineoffset(ccdc, pix.bytesperline, ODDODD, 0); but this lead to a kernel panic :-(Somehow, we need to be storing the data something like this:
EE EE EE EE ...
EO EO EO EO ...
OE OE OE OE ...
OO OO OO OO ...
but the current layout isccdc_config_outlineoffset(ccdc, pix.bytesperline, EVENEVEN, 1);
ccdc_config_outlineoffset(ccdc, pix.bytesperline, EVENODD, 1);
ccdc_config_outlineoffset(ccdc, pix.bytesperline, ODDEVEN, 1);
ccdc_config_outlineoffset(ccdc, pix.bytesperline, ODDODD, 1);EO EO EO EO ...
EE EE EE EE ...
OO OO OO OO ...
OE OE OE OE ...
Quoted 4 lines
First, I need to get the data into memory in the correct order :-)
Note: these results are consistent, i.e. if I stop things and do another grab, they are incorrect in the same [wrong] order.
Just set the FINV bit (search for it in ispccdc.c), i tested it before and i had the opposite result (from a good looking nemo-swapped-like picture to a bad one).
Quoted 3 lines
I've not done any recent tests with the gstreamer modules and the TI DSP code,
but I will shortly. We'll see how well that does.
I've tested it with the dsp and nothing changes, same problems. But i will be happy if proven wrong!
Enrico
Quoted 60 lines · Enrico
Quoted 59 lines · Gary Thomas
For days, I've been chasing ghosts :-) I know they are still there, but I think they are more a function of the source than the ISP setup. So, I went looking for a better source, NTSC in my case. My choice is is a DVD player with known good video (I'm convinced that my cheap NTSC camera produces crap, especially when there is a lot of motion in the frames). Looking at this on an analogue TV (yes, they still exist!), the picture is not bad, so I think it's a good choice, at least when trying to understand what's happening with the OMAP3 ISP.
Look at these two pictures:
http://www.mlbassoc.com/misc/nemo-00001.png
http://www.mlbassoc.com/misc/nemo-swapped-00001.pngThese represent one frame of data captured via my OMAP3 ISP + TVP5150 from a DVD (sorry, Disney). The first is a raw conversion of the frame using ffmpeg. As you can see, there seem to be lines swapped, so I wrote a little program to swap the lines even/odd. The second (nemo-swapped) shows what this looks like. Obviously, the data is not being stored in memory correctly. Does anyone know how to adjust the ISP to make this work the right way around? Currently in
ispccdc.c, we have:
ccdc_config_outlineoffset(ccdc, pix.bytesperline, EVENEVEN,
1);
ccdc_config_outlineoffset(ccdc, pix.bytesperline, EVENODD,
1);
ccdc_config_outlineoffset(ccdc, pix.bytesperline, ODDEVEN,1); ccdc_config_outlineoffset(ccdc, pix.bytesperline, ODDODD, 1);I tried this:
ccdc_config_outlineoffset(ccdc, pix.bytesperline, EVENEVEN,
2);
ccdc_config_outlineoffset(ccdc, pix.bytesperline, EVENODD,
0);
ccdc_config_outlineoffset(ccdc, pix.bytesperline, ODDEVEN,2); ccdc_config_outlineoffset(ccdc, pix.bytesperline, ODDODD, 0);but this lead to a kernel panic :-(
Somehow, we need to be storing the data something like this:
EE EE EE EE ...
EO EO EO EO ...
OE OE OE OE ...
OO OO OO OO ...
but the current layout is ccdc_config_outlineoffset(ccdc,
pix.bytesperline, EVENEVEN, 1);
ccdc_config_outlineoffset(ccdc, pix.bytesperline, EVENODD,
1);
ccdc_config_outlineoffset(ccdc, pix.bytesperline, ODDEVEN,1); ccdc_config_outlineoffset(ccdc, pix.bytesperline, ODDODD, 1);EO EO EO EO ...
EE EE EE EE ...
OO OO OO OO ...
OE OE OE OE ...First, I need to get the data into memory in the correct order :-)
Quoted 7 lines
Quoted 2 lines
Note: these results are consistent, i.e. if I stop things and do another grab, they are incorrect in the same [wrong] order.
Just set the FINV bit (search for it in
ispccdc.c), i tested it before and i had the opposite result (from a good looking nemo-swapped-like picture to a bad one).
That works great, thanks. Maybe we need another user flag, like fldmode, for this?
Quoted 8 lines
Quoted 5 lines
I've not done any recent tests with the gstreamer modules and the TI DSP
code,
but I will shortly. We'll see how well that does.I've tested it with the dsp and nothing changes, same problems. But i will be happy if proven wrong!
I'll play with this a bit more tomorrow.
Signature · 4 lines
------------------------------------------------------------ Gary Thomas | Consulting for the MLB Associates | Embedded world ------------------------------------------------------------