i.MX8MP Hantro G2 HEVC decoding problems
Hi,
I'm trying to decode H.265 full HD RTP stream on an i.MX8MP CPU. Should I expect it to work, or is there something missing?
Small streams (640x360 etc.) are decoded. With 1080p, the board panics or does something similar.
Generally Hantro G2 doesn't finish decoding some frame. I have a test video - it can probably choke on random frames, including P-frames (this is I/P-frame only stream, produced by the H2 counterpart).
The details are foggy at this point, but I'm receiving interrupts:
# dmesg | grep irq
...
[ 75.002276] hantro_g2_irq: 0x1100 masked 0x1000 <<< status registers,
shows decoding complete[ 75.143611] hantro_g2_irq: 0x2101 masked 0x0 <<< this is the problem
What could that mean? DEC_E is probably "ERROR", lack of bit 12 is "not ready", but why do I get this and what does bit 13 possibly mean?
#define G2_REG_INTERRUPT G2_SWREG(1) #define G2_REG_INTERRUPT_DEC_RDY_INT BIT(12) #define G2_REG_INTERRUPT_DEC_ABORT_E BIT(5) #define G2_REG_INTERRUPT_DEC_IRQ_DIS BIT(4) #define G2_REG_INTERRUPT_DEC_E BIT(0)
... then the condition doesn't seem to improve:
[ 95.198362] hantro_watchdog:126: frame processing timed out! [ 95.323410] imx-pgc imx-pgc-domain.8: failed to power down ADB400
and it can randomly panic, show an SError, or do nothing of the sort.
How do I even start debugging such stuff?
H.264 doesn't have this problem.
Signature · 5 lines
Krzysztof "Chris" Hałasa Sieć Badawcza Łukasiewicz Przemysłowy Instytut Automatyki i Pomiarów PIAP Al. Jerozolimskie 202, 02-486 Warszawa
Hi Krzysztof,
Le lundi 03 novembre 2025 à 14:52 +0100, Krzysztof Hałasa a écrit :
Quoted 7 lines
Hi,
I'm trying to decode H.265 full HD RTP stream on an i.MX8MP CPU. Should I expect it to work, or is there something missing?
Small streams (640x360 etc.) are decoded. With 1080p, the board panics or does something similar.
If its size related, always make sure you have enough CMA for this type of system. Though lower description seems to indicated its not the issue.
Quoted 12 lines
Generally Hantro G2 doesn't finish decoding some frame. I have a test video - it can probably choke on random frames, including P-frames (this is I/P-frame only stream, produced by the H2 counterpart).
The details are foggy at this point, but I'm receiving interrupts:
# dmesg | grep irq ... [ 75.002276] hantro_g2_irq: 0x1100 masked 0x1000 <<< status registers, shows decoding complete
Quoted 4 lines
[ 75.143611]
hantro_g2_irq:0x2101masked0x0<<< this is the problemWhat could that mean?
DEC_Eis probably "ERROR", lack of bit 12 is "not ready", but why do I get this and what does bit 13 possibly mean?
BUS Error, typically an AXI errors. See pending patch below:
Quoted 7 lines
#define G2_REG_INTERRUPT
G2_SWREG(1)#define G2_REG_INTERRUPT_DEC_RDY_INT BIT(12) #define G2_REG_INTERRUPT_DEC_ABORT_E BIT(5) #define G2_REG_INTERRUPT_DEC_IRQ_DIS BIT(4) #define G2_REG_INTERRUPT_DEC_E BIT(0)
Quoted 5 lines
... then the condition doesn't seem to improve:
[ 95.198362] hantro_watchdog:126: frame processing timed out! [ 95.323410] imx-pgc imx-pgc-domain.8: failed to power down ADB400and it can randomly panic, show an SError, or do nothing of the sort.
In queue for the next kernel release, we have a rewrite of the IRQ handler:
The important take away for this chip is that it does not always stop on errors. It can also raise a dry IRQ (I don't have explanation, only happen on IMX8 boards). The new handler properly check if the hardware is still running, which avoid programming over a live IP. The G2 has fancy internal clock gating, so doing so can lead to CPU hang if the register you set is hooked to a gated portion of the IP. At least this is my mental model of it.
And there is an important bug fix for missing references, without that, you get thousands of error IRQs while the IP is fetch from address 0x0 and its surrounding. This large burst of IRQ, combine with multiple memory related erratas of IMX8 boards cause random other things to fall appart.
Quoted 4 lines
How do I even start debugging such stuff?
H.264 doesn't have this problem.
The Hantro G1 driver is a lot more robust indeed, it has been deployed on ChromeOS for many years (though that chips is a bit obsolete now). HEVC was contributed later on, I don't think it have had that many users. It has been mostly tested with reliable media, though it can also fails there if you seek to a GOPs with CRA keyframes and RASL (skippable) leading frames, and your userspace does not pre-emptively skip these (GStreamer, which I'll fix soon).
I'm sure we will find more things to harden around this driver, so your feedback is all very appreciated.
regards,
Nicolas
Hi Nicolas,
Quoted 10 lines
In queue for the next kernel release, we have a rewrite of the IRQ handler:
The important take away for this chip is that it does not always stop on errors. It can also raise a dry IRQ (I don't have explanation, only happen on IMX8 boards). The new handler properly check if the hardware is still running, which avoid programming over a live IP. The G2 has fancy internal clock gating, so doing so can lead to CPU hang if the register you set is hooked to a gated portion of the IP. At least this is my mental model of it.
Right. This patch seems to fix the crashes. The (possibly badly transformed by GStreamer?) HEVC stream isn't displayed correctly yet, though a direct HEVC 1080p RTP from camera is now ok (certain rate of packet loss is possible here). Then:
Quoted 6 lines
And there is an important bug fix for missing references, without that, you get thousands of error IRQs while the IP is fetch from address
0x0and its surrounding. This large burst of IRQ, combine with multiple memory related erratas of IMX8 boards cause random other things to fall appart.
After applying the above, the stream from GStreamer is fixed, too. It seems it all works fine now.
From time to time I'm getting:hantro_g2_irq: 0x80100 masked 0x0
hantro-vpu 38310000.video-codec: not all macroblocks were decoded.
But I guess it's due to my network setup.
Thank you very much for your help. You have saved me a lot of work - a work with an uncertain outcome.
Signature · 5 lines
Krzysztof "Chris" Hałasa Sieć Badawcza Łukasiewicz Przemysłowy Instytut Automatyki i Pomiarów PIAP Al. Jerozolimskie 202, 02-486 Warszawa
Hi Krzysztof,
Le mardi 04 novembre 2025 à 09:12 +0100, Krzysztof Hałasa a écrit :
Quoted 35 lines
Hi Nicolas,
Quoted 4 lines
In queue for the next kernel release, we have a rewrite of the IRQ handler:
Quoted 8 lines
The important take away for this chip is that it does not always stop on errors.
It can also raise a dry IRQ (I don't have explanation, only happen on IMX8 boards). The new handler properly check if the hardware is still running, which
avoid programming over a live IP. The G2 has fancy internal clock gating, so doing so can lead to CPU hang if the register you set is hooked to a gated portion of the IP. At least this is my mental model of it.Right. This patch seems to fix the crashes. The (possibly badly transformed by GStreamer?) HEVC stream isn't displayed correctly yet, though a direct HEVC 1080p RTP from camera is now ok (certain rate of packet loss is possible here). Then:
Quoted 7 lines
And there is an important bug fix for missing references, without that, you get
thousands of error IRQs while the IP is fetch from address0x0and its surrounding. This large burst of IRQ, combine with multiple memory related erratas of IMX8 boards cause random other things to fall appart.After applying the above, the stream from GStreamer is fixed, too. It seems it all works fine now.
From time to time I'm getting:
hantro_g2_irq:0x80100masked0x0
hantro-vpu 38310000.video-codec: not all macroblocks were decoded.
Quoted 5 lines
But I guess it's due to my network setup.
Thank you very much for your help. You have saved me a lot of work - a work with an uncertain outcome.
Glad my recent fixes helped. These fixes took me several weeks to find and fix, now be ware that the IRQ coding pattern needs to be extended to all other hantro cores type, its in my todo, but not has urgent as it does not break as badly.
As for the remaining error, it matches my observation. Though, I strongly believe its yet another hidden problem in the driver. More research needed, that hard part is being able to isolate the bitstream that triggers it. I have very little information about this specific error, most of the time, there is no evident visual artifacts (so its not really broken for the users).
On another IMX8, the MQ, we often see the display controller fails at the same time we get this error. Though, the MP have a bit less problems, I believe the memory QoS is working and the that interconnect does a smoother job at load balancing the bandwidth.
Nicolas
Hi Nicolas,
Quoted 9 lines · Nicolas Dufresne
Quoted 3 lines
From time to time I'm getting:
hantro_g2_irq:0x80100masked0x0
hantro-vpu 38310000.video-codec: not all macroblocks were decoded.As for the remaining error, it matches my observation. Though, I strongly believe its yet another hidden problem in the driver. More research needed, that hard part is being able to isolate the bitstream that triggers it. I have very little information about this specific error, most of the time, there is no evident visual artifacts (so its not really broken for the users).
I see. I don't know if the above IRQ status comes with artifacts or not. What I have is a printk() in the logs from time to time. I'll let you know if I have a stream reliably triggering it.
Thanks again,
Signature · 5 lines
Krzysztof "Chris" Hałasa Sieć Badawcza Łukasiewicz Przemysłowy Instytut Automatyki i Pomiarów PIAP Al. Jerozolimskie 202, 02-486 Warszawa