Skip to contentSkip to conversation

dvbv5-zap crash dvb-tool ARMHF builds

Bug report Gregor Jasny 4 participants 19 messages Mar 12, 2019 – Apr 8, 2019 3 messages not in the archive Media (V4L/DVB)

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#2 lore raw

Hello Mauro,

below you find a bug report about an use-after-free in dvbv5-zap.

Quoted 2 lines · CHEMLA Samuel

please find a bug report that seems to concern ARMHF builds of dvbv5-zap (dvb-tool package) : https://bugs.launchpad.net/raspbian/+bug/1819650
I filed it against raspbian because I thought it was a raspbian problem, but don't think they re-build their own package, but use debian ones instead...

Thanks,
Gregor

#3 lore raw

Hi,

Quoted 11 lines · Gregor Jasny

Hello Mauro,

below you find a bug report about an use-after-free in dvbv5-zap.

Quoted 5 lines · CHEMLA Samuel

please find a bug report that seems to concern ARMHF builds of dvbv5-zap (dvb-tool package) : https://bugs.launchpad.net/raspbian/+bug/1819650
I filed it against raspbian because I thought it was a raspbian problem, but don't think they re-build their own package, but use debian ones instead...

So I can reproduce the issue with v4l-utils 1.12.3 but not with current v4l-utils (or dvbv5-zap). It looks exactly like the issue fixed in commit 6e21f6f34c1d7c3a7a059062e1ddd9705c984e2c (but I did not cherry-pick and test that on top of 1.12.3 to test that theory).

Sean

#4 lore raw

Em Fri, 15 Mar 2019 22:34:25 +0000
Sean Young <[email protected]> escreveu:

Quoted 15 lines

Hi,

Quoted 11 lines · Gregor Jasny

Hello Mauro,

below you find a bug report about an use-after-free in dvbv5-zap.

Quoted 5 lines · CHEMLA Samuel

please find a bug report that seems to concern ARMHF builds of dvbv5-zap (dvb-tool package) : https://bugs.launchpad.net/raspbian/+bug/1819650
I filed it against raspbian because I thought it was a raspbian problem, but don't think they re-build their own package, but use debian ones instead...

Quoted 4 lines

So I can reproduce the issue with v4l-utils 1.12.3 but not with current v4l-utils (or dvbv5-zap). It looks exactly like the issue fixed in commit 6e21f6f34c1d7c3a7a059062e1ddd9705c984e2c (but I did not cherry-pick and test that on top of 1.12.3 to test that theory).

I added it to stable/1.12 and another patch fixing the initialization of the parameters struct, with Samuel reported to fix the issue upstream (He pinged me in priv too, and I'm helping him to track it).

Samuel,

Could you please check if the 1.12 stable branch is OK now?

Regards,
Mauro
Thanks,
Mauro

#5 lore raw

Hi Gregor,

Samuel reported in priv that the issues he had with user after free were solved by the patchsets merged at 1.12 and 1.16 stable branches.

Could you please generate a new staging release for them?

Thanks!
Mauro

Em Sun, 17 Mar 2019 06:52:42 -0300
Mauro Carvalho Chehab <[email protected]> escreveu:

Quoted 27 lines

Em Fri, 15 Mar 2019 22:34:25 +0000
Sean Young <[email protected]> escreveu:

Quoted 15 lines

Hi,

Quoted 11 lines · Gregor Jasny

Hello Mauro,

below you find a bug report about an use-after-free in dvbv5-zap.

Quoted 5 lines · CHEMLA Samuel

please find a bug report that seems to concern ARMHF builds of dvbv5-zap (dvb-tool package) : https://bugs.launchpad.net/raspbian/+bug/1819650
I filed it against raspbian because I thought it was a raspbian problem, but don't think they re-build their own package, but use debian ones instead...

Quoted 4 lines

So I can reproduce the issue with v4l-utils 1.12.3 but not with current v4l-utils (or dvbv5-zap). It looks exactly like the issue fixed in commit 6e21f6f34c1d7c3a7a059062e1ddd9705c984e2c (but I did not cherry-pick and test that on top of 1.12.3 to test that theory).

I added it to stable/1.12 and another patch fixing the initialization of the parameters struct, with Samuel reported to fix the issue upstream (He pinged me in priv too, and I'm helping him to track it).

Quoted 8 lines

Samuel,

Could you please check if the 1.12 stable branch is OK now?

Regards,
Mauro
Thanks,
Mauro

Thanks,
Mauro

#6 lore raw

Hello Mauro,

Quoted 6 lines · Mauro Carvalho Chehab

Hi Gregor,

Samuel reported in priv that the issues he had with user after free were solved by the patchsets merged at 1.12 and 1.16 stable branches.

Could you please generate a new staging release for them?

Sure, I can create a new 1.12 and 1.16 stable release. But when reviewing the patches for approval by debian release managers I noticed an additional double-free that Sean addressed with the following patch:

Quoted 1 line

https://git.linuxtv.org/v4l-utils.git/commit/?id=ebd890019ba7383b8b486d829f6683c8f49fdbda

Could you please give that patch a thorough review, some testing, and cherry-pick it to stable-1.12 and -1.16 as well?

Thanks,
Gregor

#7 lore raw
Quoted 10 lines · Gregor Jasny

Hello Mauro,

Quoted 6 lines · Mauro Carvalho Chehab

Hi Gregor,

Samuel reported in priv that the issues he had with user after free were solved by the patchsets merged at 1.12 and 1.16 stable branches.

Could you please generate a new staging release for them?

Quoted 8 lines

Sure, I can create a new 1.12 and 1.16 stable release. But when reviewing the patches for approval by debian release managers I noticed an additional double-free that Sean addressed with the following patch:

Quoted 1 line

https://git.linuxtv.org/v4l-utils.git/commit/?id=ebd890019ba7383b8b486d829f6683c8f49fdbda

Could you please give that patch a thorough review, some testing, and cherry-pick it to stable-1.12 and -1.16 as well?

I did test it myself (and also under valgrind). The bad paths are hard to hit though. I'd say just go ahead with merging and releasing, the patch isn't that controversial (I hope!).

Sean

#8 lore raw

Em Thu, 21 Mar 2019 09:41:28 +0000
Sean Young <[email protected]> escreveu:

Quoted 15 lines · Gregor Jasny
Quoted 14 lines

Hello Mauro,

Quoted 6 lines · Mauro Carvalho Chehab

Hi Gregor,

Samuel reported in priv that the issues he had with user after free were solved by the patchsets merged at 1.12 and 1.16 stable branches.

Could you please generate a new staging release for them?

Sure, I can create a new 1.12 and 1.16 stable release. But when reviewing the patches for approval by debian release managers I noticed an additional double-free that Sean addressed with the following patch:

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

https://git.linuxtv.org/v4l-utils.git/commit/?id=ebd890019ba7383b8b486d829f6683c8f49fdbda

Could you please give that patch a thorough review, some testing, and cherry-pick it to stable-1.12 and -1.16 as well?

I did test it myself (and also under valgrind). The bad paths are hard to hit though. I'd say just go ahead with merging and releasing, the patch isn't that controversial (I hope!).

I went ahead and cherry-picked the relevant patches to -1.12, -1.14 and -1.16, and tested both dvbv5-zap and dvbv5-scan with all versions. So, we can release a new minor version for all those stable branches.

After the patches, on my tests, I didn't get any memory leaks or double-free issues.

It should be noticed that I had to add a new patch at -1.14, due to the usage of minor() and major() macros, as one of the files there were not including sys/sysmacros.h. Without that, I was getting compilation errors.

Funny enough, this header was already included on two other places within -1.14.

As the major() and minor() macros were added at glibc 2.3.3[1], released in 2004 [2], it should be save to include sys/sysmacros.h unconditionally at stable-1.14.

[1] according with "man 3 makedev", as pointed by:
https://stackoverflow.com/questions/22240973/major-and-minor-macros-defined-in-sys-sysmacros-h-pulled-in-by-iterator

[2] https://ftp.gnu.org/gnu/libc/'s glibc tarball is from 2004-08-03.

Thanks,
Mauro

#9 lore raw

Hello,

Quoted 6 lines · Mauro Carvalho Chehab

I went ahead and cherry-picked the relevant patches to -1.12, -1.14 and -1.16, and tested both dvbv5-zap and dvbv5-scan with all versions. So, we can release a new minor version for all those stable branches.

After the patches, on my tests, I didn't get any memory leaks or double-free issues.

I issues a new 1.12, 1.14, and 1.16 release.

Thanks,
Gregor

A message that is not in the archive (sent off-list or not yet received): CANJnhGfRtEwAony5Z4rFMPcu58aF2k0G+9NSkMKsq_PhfmSNqw@mail.gmail.com

#11 lore raw

Em Mon, 25 Mar 2019 17:33:30 +0100
Samuel CHEMLA <[email protected]> escreveu:

Quoted 9 lines

Hi guys,

I'm afraid I'm coming with sad news.
I just tried both stable-1.12 and stable-1.16 on a raspberry pi 2, and random hangs are back (see https://bugs.launchpad.net/raspbian/+bug/1819650
).
I previously test both branches on a raspberry zero and issues were gone (same raspbian version).
There may be more memory issues somewhere...

Could you test it with valgrind?

Quoted 22 lines

Sam

Quoted 17 lines · Gregor Jasny

Hello,

Quoted 2 lines · Mauro Carvalho Chehab

I went ahead and cherry-picked the relevant patches to -1.12, -1.14 and -1.16, and tested both dvbv5-zap and dvbv5-scan with all versions. So,

we can

Quoted 4 lines

release a new minor version for all those stable branches.

After the patches, on my tests, I didn't get any memory leaks or double-free issues.

I issues a new 1.12, 1.14, and 1.16 release.

Thanks,
Gregor

Thanks,
Mauro

#12 lore raw

Hi,

I am struggling with valgrind because it always complain with either :
ASan runtime does not come first in initial library list; you
should either link runtime to your application or manually preload it with LD_PRELOAD

    -> When I LD_PRELOAD, I'm getting a segfault, but I couldn't find

any core dump

or, if I link statically libasan with -static-libasan:
Shadow memory range interleaves with an existing memory mapping.
ASan cannot proceed correctly. ABORTING.
ASan shadow was supposed to be located in the
[0x00007fff7000-0x10007fff7fff] range.

I retested again on my raspberry zero W, and I confirm i cannot reproduce the hang.
Your fix did work on that device.
I am testing with same OS (raspbian with latest updates, same kernel), same configure options, same USB dongle... :-(
The only differences are CPU architecture (armv6 vs armv7), memory constraints, and I was not using the same channels.conf, I'll fix that today and re-check

Sam

On 25/03/2019 18:08, Mauro Carvalho Chehab wrote:

Em Mon, 25 Mar 2019 17:33:30 +0100
Samuel CHEMLA <[email protected]> escreveu:

Hi guys,

I'm afraid I'm coming with sad news.
I just tried both stable-1.12 and stable-1.16 on a raspberry pi 2, and random hangs are back (see https://bugs.launchpad.net/raspbian/+bug/1819650
).
I previously test both branches on a raspberry zero and issues were gone (same raspbian version).
There may be more memory issues somewhere...

Could you test it with valgrind?

Sam

Le jeu. 21 mars 2019 à 20:59, Gregor Jasny <[email protected]> a écrit :

Hello,

On 21.03.19 12:30, Mauro Carvalho Chehab wrote:

I went ahead and cherry-picked the relevant patches to -1.12, -1.14 and -1.16, and tested both dvbv5-zap and dvbv5-scan with all versions. So,

we can

release a new minor version for all those stable branches.

After the patches, on my tests, I didn't get any memory leaks or double-free issues.

I issues a new 1.12, 1.14, and 1.16 release.

Thanks,
Gregor

Thanks,
Mauro

#13 lore raw

Hi Sam,

Quoted 17 lines · Samuel CHEMLA

Hi,

I am struggling with valgrind because it always complain with either :
ASan runtime does not come first in initial library list; you
should either link runtime to your application or manually preload it with LD_PRELOAD

    -> When I LD_PRELOAD, I'm getting a segfault, but I couldn't find

any core dump

or, if I link statically libasan with -static-libasan:
Shadow memory range interleaves with an existing memory mapping.
ASan cannot proceed correctly. ABORTING.
ASan shadow was supposed to be located in the
[0x00007fff7000-0x10007fff7fff] range.

Quoted 8 lines

I retested again on my raspberry zero W, and I confirm i cannot reproduce the hang.
Your fix did work on that device.
I am testing with same OS (raspbian with latest updates, same kernel), same configure options, same USB dongle... :-(
The only differences are CPU architecture (armv6 vs armv7), memory constraints, and I was not using the same channels.conf, I'll fix that today and re-check

Earlier you said "random hangs are back". When this happens, does the whole device become unresponsive or just dvbv5-zap? Since this issue is "back", I wouldn't be surprised this is unrelated to the fixes in 1.12.7 and 1.16.4.

It would be useful to see the output from dmesg (best thing would be after the issue occurs).

Also what dvb hardware are you using?

Thanks,

san

Quoted 48 lines

Sam

On 25/03/2019 18:08, Mauro Carvalho Chehab wrote:

Em Mon, 25 Mar 2019 17:33:30 +0100
Samuel CHEMLA <[email protected]> escreveu:

Hi guys,

I'm afraid I'm coming with sad news.
I just tried both stable-1.12 and stable-1.16 on a raspberry pi 2, and random hangs are back (see https://bugs.launchpad.net/raspbian/+bug/1819650
).
I previously test both branches on a raspberry zero and issues were gone (same raspbian version).
There may be more memory issues somewhere...

Could you test it with valgrind?

Sam

Le jeu. 21 mars 2019 à 20:59, Gregor Jasny <[email protected]> a écrit :

Hello,

On 21.03.19 12:30, Mauro Carvalho Chehab wrote:

I went ahead and cherry-picked the relevant patches to -1.12, -1.14 and -1.16, and tested both dvbv5-zap and dvbv5-scan with all versions. So,

we can

release a new minor version for all those stable branches.

After the patches, on my tests, I didn't get any memory leaks or double-free issues.

I issues a new 1.12, 1.14, and 1.16 release.

Thanks,
Gregor

Thanks,
Mauro

#14 lore raw

Hi,

Quoted 2 lines

Earlier you said "random hangs are back". When this happens, does the whole device become unresponsive or just dvbv5-zap?

The device completely freeze, you can't even switch numlock on/off. I said "the issue is back", it is because I **thought** it was gone. To be more precise:
- on raspberry zero W, the issue is gone since Mauro's patch
(https://git.linuxtv.org/v4l-utils.git/commit/?id=22b06353227e04695b1b0a9622b896b948adba89)
- on raspberry 2, the issue, it is still there and the patch has no effect (the issue was and is still there)

Quoted 2 lines

Since this issue is "back",
I wouldn't be surprised this is unrelated to the fixes in 1.12.7 and 1.16.4.

The issue doesn't appear anymore on raspberry zero W since Mauro's commit. So it did improve on that platform.

Quoted 2 lines

It would be useful to see the output from dmesg (best thing would be after the issue occurs).

You can't, device is frozen.
Logs are not flushed to disk, and journalctl -f freeze before showing anything

Quoted 1 line

Also what dvb hardware are you using?

I reproduced it with different two tuners: rtl2832U from RTL-SDR.COM and a TerraTec Cinergy T Stick+

You can found all the details here:
https://bugs.launchpad.net/raspbian/+bug/1819650

Sam

Quoted 91 lines · Sean Young

Hi Sam,

Quoted 17 lines · Samuel CHEMLA

Hi,

I am struggling with valgrind because it always complain with either :
ASan runtime does not come first in initial library list; you
should either link runtime to your application or manually preload it with LD_PRELOAD

    -> When I LD_PRELOAD, I'm getting a segfault, but I couldn't find

any core dump

or, if I link statically libasan with -static-libasan:
Shadow memory range interleaves with an existing memory mapping.
ASan cannot proceed correctly. ABORTING.
ASan shadow was supposed to be located in the
[0x00007fff7000-0x10007fff7fff] range.

Quoted 8 lines

I retested again on my raspberry zero W, and I confirm i cannot reproduce the hang.
Your fix did work on that device.
I am testing with same OS (raspbian with latest updates, same kernel), same configure options, same USB dongle... :-(
The only differences are CPU architecture (armv6 vs armv7), memory constraints, and I was not using the same channels.conf, I'll fix that today and re-check

Earlier you said "random hangs are back". When this happens, does the whole device become unresponsive or just dvbv5-zap? Since this issue is "back", I wouldn't be surprised this is unrelated to the fixes in 1.12.7 and 1.16.4.

It would be useful to see the output from dmesg (best thing would be after the issue occurs).

Also what dvb hardware are you using?

Thanks,

san

Quoted 48 lines

Sam

On 25/03/2019 18:08, Mauro Carvalho Chehab wrote:

Em Mon, 25 Mar 2019 17:33:30 +0100
Samuel CHEMLA <[email protected]> escreveu:

Hi guys,

I'm afraid I'm coming with sad news.
I just tried both stable-1.12 and stable-1.16 on a raspberry pi 2, and random hangs are back (see https://bugs.launchpad.net/raspbian/+bug/1819650
).
I previously test both branches on a raspberry zero and issues were gone (same raspbian version).
There may be more memory issues somewhere...

Could you test it with valgrind?

Sam

Le jeu. 21 mars 2019 ŕ 20:59, Gregor Jasny <[email protected]> a écrit :

Hello,

On 21.03.19 12:30, Mauro Carvalho Chehab wrote:

I went ahead and cherry-picked the relevant patches to -1.12, -1.14 and -1.16, and tested both dvbv5-zap and dvbv5-scan with all versions. So,

we can

release a new minor version for all those stable branches.

After the patches, on my tests, I didn't get any memory leaks or double-free issues.

I issues a new 1.12, 1.14, and 1.16 release.

Thanks,
Gregor

Thanks,
Mauro

#15 lore raw

Em Tue, 26 Mar 2019 16:10:33 +0100
Samuel CHEMLA <[email protected]> escreveu:

Quoted 6 lines

Hi,

Quoted 2 lines

Earlier you said "random hangs are back". When this happens, does the whole device become unresponsive or just dvbv5-zap?

The device completely freeze, you can't even switch numlock on/off.

dvbv5-tools can't hang the machine. this is very likely happening due to a Kernel crash.

Quoted 6 lines

I said "the issue is back", it is because I **thought** it was gone. To be more precise:
- on raspberry zero W, the issue is gone since Mauro's patch
(https://git.linuxtv.org/v4l-utils.git/commit/?id=22b06353227e04695b1b0a9622b896b948adba89)
- on raspberry 2, the issue, it is still there and the patch has no effect (the issue was and is still there)

RPi2 has a serious issue with their USB ports: on devices that require more than a few mW to work, it causes several device disconnection and re-connection, as it cannot sustain the required 5V.

Depending on how fast this happens, it could be triggering some Kernel bug.

That affects most V4L and DVB devices. You should either use a powered USB 2.0 hub (with may be problematic, as the USB host driver on RPi has issues - and may cause data loss on high sustained ISOC traffic, specially when used with hubs) or a device that has its own power supply, like DVBSky T680C or S960.

Quoted 9 lines
Quoted 2 lines

Since this issue is "back",
I wouldn't be surprised this is unrelated to the fixes in 1.12.7 and 1.16.4.

The issue doesn't appear anymore on raspberry zero W since Mauro's commit. So it did improve on that platform.

Quoted 2 lines

It would be useful to see the output from dmesg (best thing would be after the issue occurs).

You can't, device is frozen.
Logs are not flushed to disk, and journalctl -f freeze before showing anything

You can use a serial port in order to get the logs. On a serial console, use something like:

	# dmesg -n 8

In order to make sure it will display all Kernel messages at console.

Quoted 4 lines
Quoted 2 lines

Also what dvb hardware are you using?

I reproduced it with different two tuners: rtl2832U from RTL-SDR.COM and a TerraTec Cinergy T Stick+

None of them supports an external power supply.

Quoted 4 lines

You can found all the details here:
https://bugs.launchpad.net/raspbian/+bug/1819650

Quoted 95 lines

Sam

Quoted 91 lines · Sean Young

Hi Sam,

Quoted 17 lines · Samuel CHEMLA

Hi,

I am struggling with valgrind because it always complain with either :
ASan runtime does not come first in initial library list; you
should either link runtime to your application or manually preload it with LD_PRELOAD

    -> When I LD_PRELOAD, I'm getting a segfault, but I couldn't find  

any core dump

or, if I link statically libasan with -static-libasan:
Shadow memory range interleaves with an existing memory mapping.
ASan cannot proceed correctly. ABORTING.
ASan shadow was supposed to be located in the
[0x00007fff7000-0x10007fff7fff] range.

Quoted 8 lines

I retested again on my raspberry zero W, and I confirm i cannot reproduce the hang.
Your fix did work on that device.
I am testing with same OS (raspbian with latest updates, same kernel), same configure options, same USB dongle... :-(
The only differences are CPU architecture (armv6 vs armv7), memory constraints, and I was not using the same channels.conf, I'll fix that today and re-check

Earlier you said "random hangs are back". When this happens, does the whole device become unresponsive or just dvbv5-zap? Since this issue is "back", I wouldn't be surprised this is unrelated to the fixes in 1.12.7 and 1.16.4.

It would be useful to see the output from dmesg (best thing would be after the issue occurs).

Also what dvb hardware are you using?

Thanks,

san

Quoted 48 lines

Sam

On 25/03/2019 18:08, Mauro Carvalho Chehab wrote:

Em Mon, 25 Mar 2019 17:33:30 +0100
Samuel CHEMLA <[email protected]> escreveu:

Hi guys,

I'm afraid I'm coming with sad news.
I just tried both stable-1.12 and stable-1.16 on a raspberry pi 2, and random hangs are back (see https://bugs.launchpad.net/raspbian/+bug/1819650
).
I previously test both branches on a raspberry zero and issues were gone (same raspbian version).
There may be more memory issues somewhere...

Could you test it with valgrind?

Sam

Le jeu. 21 mars 2019 ŕ 20:59, Gregor Jasny <[email protected]> a écrit :

Hello,

On 21.03.19 12:30, Mauro Carvalho Chehab wrote:

I went ahead and cherry-picked the relevant patches to -1.12, -1.14 and -1.16, and tested both dvbv5-zap and dvbv5-scan with all versions. So,

we can

release a new minor version for all those stable branches.

After the patches, on my tests, I didn't get any memory leaks or double-free issues.

I issues a new 1.12, 1.14, and 1.16 release.

Thanks,
Gregor

Thanks,
Mauro

Thanks,
Mauro

#16 lore raw

Hi,

1) I did reproduce the bug with a serial console, but the serial console was also frozen, and there was no message before freeze.
2) I also tried a powered USB HUB but it didn't change anything.
3) I tried DVBJet, from https://github.com/lightful/DVBdirect , it is a tuner that directly does ioctl on linux kernel DVB device. I couldn't reproduce the issue with DVBJet.
It's been running for almost 48Hrs now, without a crash.
I collect all PIDs on the multiplex, just like dvbv5-zap, so the USB port is solicited at it's maximum.

That makes me think it is not a hardware issue.
I'm pretty sure dvbv5-zap can still improve, and it already did when you fixed: struct arguments args = {};

Regards

Quoted 162 lines · Mauro Carvalho Chehab

Em Tue, 26 Mar 2019 16:10:33 +0100
Samuel CHEMLA <[email protected]> escreveu:

Quoted 6 lines

Hi,

Quoted 2 lines

Earlier you said "random hangs are back". When this happens, does the whole device become unresponsive or just dvbv5-zap?

The device completely freeze, you can't even switch numlock on/off.

dvbv5-tools can't hang the machine. this is very likely happening due to a Kernel crash.

Quoted 6 lines

I said "the issue is back", it is because I **thought** it was gone. To be more precise:
- on raspberry zero W, the issue is gone since Mauro's patch
(https://git.linuxtv.org/v4l-utils.git/commit/?id=22b06353227e04695b1b0a9622b896b948adba89)
- on raspberry 2, the issue, it is still there and the patch has no effect (the issue was and is still there)

RPi2 has a serious issue with their USB ports: on devices that require more than a few mW to work, it causes several device disconnection and re-connection, as it cannot sustain the required 5V.

Depending on how fast this happens, it could be triggering some Kernel bug.

That affects most V4L and DVB devices. You should either use a powered USB 2.0 hub (with may be problematic, as the USB host driver on RPi has issues - and may cause data loss on high sustained ISOC traffic, specially when used with hubs) or a device that has its own power supply, like DVBSky T680C or S960.

Quoted 9 lines
Quoted 2 lines

Since this issue is "back",
I wouldn't be surprised this is unrelated to the fixes in 1.12.7 and 1.16.4.

The issue doesn't appear anymore on raspberry zero W since Mauro's commit. So it did improve on that platform.

Quoted 2 lines

It would be useful to see the output from dmesg (best thing would be after the issue occurs).

You can't, device is frozen.
Logs are not flushed to disk, and journalctl -f freeze before showing anything

You can use a serial port in order to get the logs. On a serial console, use something like:

        # dmesg -n 8

In order to make sure it will display all Kernel messages at console.

Quoted 4 lines
Quoted 2 lines

Also what dvb hardware are you using?

I reproduced it with different two tuners: rtl2832U from RTL-SDR.COM and a TerraTec Cinergy T Stick+

None of them supports an external power supply.

Quoted 4 lines

You can found all the details here:
https://bugs.launchpad.net/raspbian/+bug/1819650

Quoted 95 lines

Sam

Quoted 91 lines · Sean Young

Hi Sam,

Quoted 17 lines · Samuel CHEMLA

Hi,

I am struggling with valgrind because it always complain with either :
ASan runtime does not come first in initial library list; you
should either link runtime to your application or manually preload it with LD_PRELOAD

    -> When I LD_PRELOAD, I'm getting a segfault, but I couldn't find

any core dump

or, if I link statically libasan with -static-libasan:
Shadow memory range interleaves with an existing memory mapping.
ASan cannot proceed correctly. ABORTING.
ASan shadow was supposed to be located in the
[0x00007fff7000-0x10007fff7fff] range.

Quoted 8 lines

I retested again on my raspberry zero W, and I confirm i cannot reproduce the hang.
Your fix did work on that device.
I am testing with same OS (raspbian with latest updates, same kernel), same configure options, same USB dongle... :-(
The only differences are CPU architecture (armv6 vs armv7), memory constraints, and I was not using the same channels.conf, I'll fix that today and re-check

Earlier you said "random hangs are back". When this happens, does the whole device become unresponsive or just dvbv5-zap? Since this issue is "back", I wouldn't be surprised this is unrelated to the fixes in 1.12.7 and 1.16.4.

It would be useful to see the output from dmesg (best thing would be after the issue occurs).

Also what dvb hardware are you using?

Thanks,

san

Quoted 48 lines

Sam

On 25/03/2019 18:08, Mauro Carvalho Chehab wrote:

Em Mon, 25 Mar 2019 17:33:30 +0100
Samuel CHEMLA <[email protected]> escreveu:

Hi guys,

I'm afraid I'm coming with sad news.
I just tried both stable-1.12 and stable-1.16 on a raspberry pi 2, and random hangs are back (see https://bugs.launchpad.net/raspbian/+bug/1819650
).
I previously test both branches on a raspberry zero and issues were gone (same raspbian version).
There may be more memory issues somewhere...

Could you test it with valgrind?

Sam

Le jeu. 21 mars 2019 ŕ 20:59, Gregor Jasny <[email protected]> a écrit :

Hello,

On 21.03.19 12:30, Mauro Carvalho Chehab wrote:

I went ahead and cherry-picked the relevant patches to -1.12, -1.14 and -1.16, and tested both dvbv5-zap and dvbv5-scan with all versions. So,

we can

release a new minor version for all those stable branches.

After the patches, on my tests, I didn't get any memory leaks or double-free issues.

I issues a new 1.12, 1.14, and 1.16 release.

Thanks,
Gregor

Thanks,
Mauro

Thanks,
Mauro

#17 lore raw

Em Thu, 28 Mar 2019 19:48:35 +0100
Samuel CHEMLA <[email protected]> escreveu:

Quoted 1 line

Hi,

Please, don't top post.

Quoted 4 lines

1) I did reproduce the bug with a serial console, but the serial console was also frozen, and there was no message before freeze.

The only way for a machine to freeze even on serial console is due to a very serious Kernel or hardware bug, like a bad lock/semaphore.

Quoted 1 line

2) I also tried a powered USB HUB but it didn't change anything.

Ok.

Quoted 3 lines

3) I tried DVBJet, from https://github.com/lightful/DVBdirect , it is a tuner that directly does ioctl on linux kernel DVB device. I couldn't reproduce the issue with DVBJet.

I don't know DVBJet. If it uses the same Kernel drivers, maybe it could then be a race issue: send commands on a slower way to the Kernel would be solving the issue.

Quoted 7 lines

It's been running for almost 48Hrs now, without a crash. I collect all PIDs on the multiplex, just like dvbv5-zap, so the USB port is solicited at it's maximum.

That makes me think it is not a hardware issue.
I'm pretty sure dvbv5-zap can still improve, and it already did when you fixed: struct arguments args = {};

Everything can be improved, but the thing is that we need first to discover the root cause :-)

Can you post on pastebin (or equivalent) the dmesg with the DVBJet running? Maybe it could be printing some Kernel messages that might help to discover what's going wrong.

Btw, could you also apply this patch:

	https://patchwork.linuxtv.org/patch/55274/

It is probably unrelated, but this is the kind of bug that could cause such issues.

Quoted 168 lines

Regards

Quoted 162 lines · Mauro Carvalho Chehab

Em Tue, 26 Mar 2019 16:10:33 +0100
Samuel CHEMLA <[email protected]> escreveu:

Quoted 6 lines

Hi,

Quoted 2 lines

Earlier you said "random hangs are back". When this happens, does the whole device become unresponsive or just dvbv5-zap?

The device completely freeze, you can't even switch numlock on/off.

dvbv5-tools can't hang the machine. this is very likely happening due to a Kernel crash.

Quoted 6 lines

I said "the issue is back", it is because I **thought** it was gone. To be more precise:
- on raspberry zero W, the issue is gone since Mauro's patch
(https://git.linuxtv.org/v4l-utils.git/commit/?id=22b06353227e04695b1b0a9622b896b948adba89)
- on raspberry 2, the issue, it is still there and the patch has no effect (the issue was and is still there)

RPi2 has a serious issue with their USB ports: on devices that require more than a few mW to work, it causes several device disconnection and re-connection, as it cannot sustain the required 5V.

Depending on how fast this happens, it could be triggering some Kernel bug.

That affects most V4L and DVB devices. You should either use a powered USB 2.0 hub (with may be problematic, as the USB host driver on RPi has issues - and may cause data loss on high sustained ISOC traffic, specially when used with hubs) or a device that has its own power supply, like DVBSky T680C or S960.

Quoted 9 lines
Quoted 2 lines

Since this issue is "back",
I wouldn't be surprised this is unrelated to the fixes in 1.12.7 and 1.16.4.

The issue doesn't appear anymore on raspberry zero W since Mauro's commit. So it did improve on that platform.

Quoted 2 lines

It would be useful to see the output from dmesg (best thing would be after the issue occurs).

You can't, device is frozen.
Logs are not flushed to disk, and journalctl -f freeze before showing anything

You can use a serial port in order to get the logs. On a serial console, use something like:

        # dmesg -n 8

In order to make sure it will display all Kernel messages at console.

Quoted 4 lines
Quoted 2 lines

Also what dvb hardware are you using?

I reproduced it with different two tuners: rtl2832U from RTL-SDR.COM and a TerraTec Cinergy T Stick+

None of them supports an external power supply.

Quoted 4 lines

You can found all the details here:
https://bugs.launchpad.net/raspbian/+bug/1819650

Quoted 95 lines

Sam

Quoted 91 lines · Sean Young

Hi Sam,

Quoted 17 lines · Samuel CHEMLA

Hi,

I am struggling with valgrind because it always complain with either :
ASan runtime does not come first in initial library list; you
should either link runtime to your application or manually preload it with LD_PRELOAD

    -> When I LD_PRELOAD, I'm getting a segfault, but I couldn't find  

any core dump

or, if I link statically libasan with -static-libasan:
Shadow memory range interleaves with an existing memory mapping.
ASan cannot proceed correctly. ABORTING.
ASan shadow was supposed to be located in the
[0x00007fff7000-0x10007fff7fff] range.

Quoted 8 lines

I retested again on my raspberry zero W, and I confirm i cannot reproduce the hang.
Your fix did work on that device.
I am testing with same OS (raspbian with latest updates, same kernel), same configure options, same USB dongle... :-(
The only differences are CPU architecture (armv6 vs armv7), memory constraints, and I was not using the same channels.conf, I'll fix that today and re-check

Earlier you said "random hangs are back". When this happens, does the whole device become unresponsive or just dvbv5-zap? Since this issue is "back", I wouldn't be surprised this is unrelated to the fixes in 1.12.7 and 1.16.4.

It would be useful to see the output from dmesg (best thing would be after the issue occurs).

Also what dvb hardware are you using?

Thanks,

san

Quoted 48 lines

Sam

On 25/03/2019 18:08, Mauro Carvalho Chehab wrote:

Em Mon, 25 Mar 2019 17:33:30 +0100
Samuel CHEMLA <[email protected]> escreveu:

Hi guys,

I'm afraid I'm coming with sad news.
I just tried both stable-1.12 and stable-1.16 on a raspberry pi 2, and random hangs are back (see https://bugs.launchpad.net/raspbian/+bug/1819650
).
I previously test both branches on a raspberry zero and issues were gone (same raspbian version).
There may be more memory issues somewhere...

Could you test it with valgrind?

Sam

Le jeu. 21 mars 2019 ŕ 20:59, Gregor Jasny <[email protected]> a écrit :

Hello,

On 21.03.19 12:30, Mauro Carvalho Chehab wrote:

I went ahead and cherry-picked the relevant patches to -1.12, -1.14 and -1.16, and tested both dvbv5-zap and dvbv5-scan with all versions. So,

we can

release a new minor version for all those stable branches.

After the patches, on my tests, I didn't get any memory leaks or double-free issues.

I issues a new 1.12, 1.14, and 1.16 release.

Thanks,
Gregor

Thanks,
Mauro

Thanks,
Mauro

Thanks,
Mauro

#18 lore raw

Hello,

Quoted 7 lines · Mauro Carvalho Chehab

Em Thu, 28 Mar 2019 19:48:35 +0100
Samuel CHEMLA <[email protected]> escreveu:

Quoted 2 lines

1) I did reproduce the bug with a serial console, but the serial console was also frozen, and there was no message before freeze.

The only way for a machine to freeze even on serial console is due to a very serious Kernel or hardware bug, like a bad lock/semaphore.

Does anything speak against upgrading Debian to latest 1.16.5 release?

Thanks,
Gregor

#19 lore raw

Em Thu, 28 Mar 2019 20:48:34 +0100
Gregor Jasny <[email protected]> escreveu:

Quoted 5 lines

Hello,

Quoted 2 lines · Mauro Carvalho Chehab

Em Thu, 28 Mar 2019 19:48:35 +0100
Samuel CHEMLA <[email protected]> escreveu:

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

1) I did reproduce the bug with a serial console, but the serial console was also frozen, and there was no message before freeze.

The only way for a machine to freeze even on serial console is due to a very serious Kernel or hardware bug, like a bad lock/semaphore.

Does anything speak against upgrading Debian to latest 1.16.5 release?

I don't think so. The changes we did were just at memory allocation/free with standard glibc malloc()/calloc()/free() functions.

The kind of errors Samuel are experiencing are machine freezes. dvbv5-utils can't cause such kind of errors, as it doesn't run in privileged mode, nor do anything potentially harmful.

So, even the most serious issue there won't cause a machine crash.

For a machine to crash, it has to be triggering a Kernel bug or some memory corruption outside its memory space (e.g. triggering some glibc bug - with also seems very unlikely).

Thanks,
Mauro

#20 lore raw
In reply to Mauro Carvalho Chehab
Quoted 35 lines · Mauro Carvalho Chehab

Em Thu, 28 Mar 2019 19:48:35 +0100
Samuel CHEMLA <[email protected]> escreveu:

Quoted 1 line

Hi,

Please, don't top post.

Quoted 4 lines

1) I did reproduce the bug with a serial console, but the serial console was also frozen, and there was no message before freeze.

The only way for a machine to freeze even on serial console is due to a very serious Kernel or hardware bug, like a bad lock/semaphore.

Quoted 1 line

2) I also tried a powered USB HUB but it didn't change anything.

Ok.

Quoted 3 lines

3) I tried DVBJet, from https://github.com/lightful/DVBdirect , it is a tuner that directly does ioctl on linux kernel DVB device. I couldn't reproduce the issue with DVBJet.

I don't know DVBJet. If it uses the same Kernel drivers, maybe it could then be a race issue: send commands on a slower way to the Kernel would be solving the issue.

Quoted 7 lines

It's been running for almost 48Hrs now, without a crash. I collect all PIDs on the multiplex, just like dvbv5-zap, so the USB port is solicited at it's maximum.

That makes me think it is not a hardware issue.
I'm pretty sure dvbv5-zap can still improve, and it already did when you fixed: struct arguments args = {};

Quoted 6 lines

Everything can be improved, but the thing is that we need first to discover the root cause :-)

Can you post on pastebin (or equivalent) the dmesg with the DVBJet running? Maybe it could be printing some Kernel messages that might help to discover what's going wrong.

Here is the dmesg: https://pastebin.com/3XRim4XL
Just look at the code, it's quite straightforward, just a few ioctl.

Quoted 3 lines

Btw, could you also apply this patch:

        https://patchwork.linuxtv.org/patch/55274/

OK, I'll give it a try and provide feedback

Quoted 177 lines

It is probably unrelated, but this is the kind of bug that could cause such issues.

Quoted 168 lines

Regards

Quoted 162 lines · Mauro Carvalho Chehab

Em Tue, 26 Mar 2019 16:10:33 +0100
Samuel CHEMLA <[email protected]> escreveu:

Quoted 6 lines

Hi,

Quoted 2 lines

Earlier you said "random hangs are back". When this happens, does the whole device become unresponsive or just dvbv5-zap?

The device completely freeze, you can't even switch numlock on/off.

dvbv5-tools can't hang the machine. this is very likely happening due to a Kernel crash.

Quoted 6 lines

I said "the issue is back", it is because I **thought** it was gone. To be more precise:
- on raspberry zero W, the issue is gone since Mauro's patch
(https://git.linuxtv.org/v4l-utils.git/commit/?id=22b06353227e04695b1b0a9622b896b948adba89)
- on raspberry 2, the issue, it is still there and the patch has no effect (the issue was and is still there)

RPi2 has a serious issue with their USB ports: on devices that require more than a few mW to work, it causes several device disconnection and re-connection, as it cannot sustain the required 5V.

Depending on how fast this happens, it could be triggering some Kernel bug.

That affects most V4L and DVB devices. You should either use a powered USB 2.0 hub (with may be problematic, as the USB host driver on RPi has issues - and may cause data loss on high sustained ISOC traffic, specially when used with hubs) or a device that has its own power supply, like DVBSky T680C or S960.

Quoted 9 lines
Quoted 2 lines

Since this issue is "back",
I wouldn't be surprised this is unrelated to the fixes in 1.12.7 and 1.16.4.

The issue doesn't appear anymore on raspberry zero W since Mauro's commit. So it did improve on that platform.

Quoted 2 lines

It would be useful to see the output from dmesg (best thing would be after the issue occurs).

You can't, device is frozen.
Logs are not flushed to disk, and journalctl -f freeze before showing anything

You can use a serial port in order to get the logs. On a serial console, use something like:

        # dmesg -n 8

In order to make sure it will display all Kernel messages at console.

Quoted 4 lines
Quoted 2 lines

Also what dvb hardware are you using?

I reproduced it with different two tuners: rtl2832U from RTL-SDR.COM and a TerraTec Cinergy T Stick+

None of them supports an external power supply.

Quoted 4 lines

You can found all the details here:
https://bugs.launchpad.net/raspbian/+bug/1819650

Quoted 95 lines

Sam

Quoted 91 lines · Sean Young

Hi Sam,

Quoted 17 lines · Samuel CHEMLA

Hi,

I am struggling with valgrind because it always complain with either :
ASan runtime does not come first in initial library list; you
should either link runtime to your application or manually preload it with LD_PRELOAD

    -> When I LD_PRELOAD, I'm getting a segfault, but I couldn't find

any core dump

or, if I link statically libasan with -static-libasan:
Shadow memory range interleaves with an existing memory mapping.
ASan cannot proceed correctly. ABORTING.
ASan shadow was supposed to be located in the
[0x00007fff7000-0x10007fff7fff] range.

Quoted 8 lines

I retested again on my raspberry zero W, and I confirm i cannot reproduce the hang.
Your fix did work on that device.
I am testing with same OS (raspbian with latest updates, same kernel), same configure options, same USB dongle... :-(
The only differences are CPU architecture (armv6 vs armv7), memory constraints, and I was not using the same channels.conf, I'll fix that today and re-check

Earlier you said "random hangs are back". When this happens, does the whole device become unresponsive or just dvbv5-zap? Since this issue is "back", I wouldn't be surprised this is unrelated to the fixes in 1.12.7 and 1.16.4.

It would be useful to see the output from dmesg (best thing would be after the issue occurs).

Also what dvb hardware are you using?

Thanks,

san

Quoted 48 lines

Sam

On 25/03/2019 18:08, Mauro Carvalho Chehab wrote:

Em Mon, 25 Mar 2019 17:33:30 +0100
Samuel CHEMLA <[email protected]> escreveu:

Hi guys,

I'm afraid I'm coming with sad news.
I just tried both stable-1.12 and stable-1.16 on a raspberry pi 2, and random hangs are back (see https://bugs.launchpad.net/raspbian/+bug/1819650
).
I previously test both branches on a raspberry zero and issues were gone (same raspbian version).
There may be more memory issues somewhere...

Could you test it with valgrind?

Sam

Le jeu. 21 mars 2019 ŕ 20:59, Gregor Jasny <[email protected]> a écrit :

Hello,

On 21.03.19 12:30, Mauro Carvalho Chehab wrote:

I went ahead and cherry-picked the relevant patches to -1.12, -1.14 and -1.16, and tested both dvbv5-zap and dvbv5-scan with all versions. So,

we can

release a new minor version for all those stable branches.

After the patches, on my tests, I didn't get any memory leaks or double-free issues.

I issues a new 1.12, 1.14, and 1.16 release.

Thanks,
Gregor

Thanks,
Mauro

Thanks,
Mauro

Thanks,
Mauro

#21 lore raw
Quoted 36 lines · Samuel CHEMLA
Quoted 28 lines · Mauro Carvalho Chehab

Em Thu, 28 Mar 2019 19:48:35 +0100
Samuel CHEMLA <[email protected]> escreveu:

Quoted 1 line

Hi,

Please, don't top post.

Quoted 3 lines

1) I did reproduce the bug with a serial console, but the serial console was also frozen, and there was no message before freeze.

The only way for a machine to freeze even on serial console is due to a very serious Kernel or hardware bug, like a bad lock/semaphore.

Quoted 1 line

2) I also tried a powered USB HUB but it didn't change anything.

Ok.

Quoted 3 lines

3) I tried DVBJet, from https://github.com/lightful/DVBdirect , it is a tuner that directly does ioctl on linux kernel DVB device. I couldn't reproduce the issue with DVBJet.

I don't know DVBJet. If it uses the same Kernel drivers, maybe it could then be a race issue: send commands on a slower way to the Kernel would be solving the issue.

Quoted 7 lines

It's been running for almost 48Hrs now, without a crash. I collect all PIDs on the multiplex, just like dvbv5-zap, so the USB port is solicited at it's maximum.

That makes me think it is not a hardware issue.
I'm pretty sure dvbv5-zap can still improve, and it already did when you fixed: struct arguments args = {};

Quoted 6 lines

Everything can be improved, but the thing is that we need first to discover the root cause :-)

Can you post on pastebin (or equivalent) the dmesg with the DVBJet running? Maybe it could be printing some Kernel messages that might help to discover what's going wrong.

Quoted 6 lines

Here is the dmesg: https://pastebin.com/3XRim4XL
Just look at the code, it's quite straightforward, just a few ioctl.

Quoted 2 lines

Btw, could you also apply this patch:

         https://patchwork.linuxtv.org/patch/55274/

OK, I'll give it a try and provide feedback

I tested you patch against 4.14.98 which is the current raspbian kernel, but it didn't change anything...

Quoted 165 lines
Quoted 165 lines

It is probably unrelated, but this is the kind of bug that could cause such issues.

Quoted 157 lines

Regards

Quoted 152 lines · Mauro Carvalho Chehab

Em Tue, 26 Mar 2019 16:10:33 +0100
Samuel CHEMLA <[email protected]> escreveu:

Quoted 6 lines

Hi,

Quoted 2 lines

Earlier you said "random hangs are back". When this happens, does the whole device become unresponsive or just dvbv5-zap?

The device completely freeze, you can't even switch numlock on/off.

dvbv5-tools can't hang the machine. this is very likely happening due to a Kernel crash.

Quoted 6 lines

I said "the issue is back", it is because I **thought** it was gone. To be more precise:
- on raspberry zero W, the issue is gone since Mauro's patch
(https://git.linuxtv.org/v4l-utils.git/commit/?id=22b06353227e04695b1b0a9622b896b948adba89)
- on raspberry 2, the issue, it is still there and the patch has no effect (the issue was and is still there)

RPi2 has a serious issue with their USB ports: on devices that require more than a few mW to work, it causes several device disconnection and re-connection, as it cannot sustain the required 5V.

Depending on how fast this happens, it could be triggering some Kernel bug.

That affects most V4L and DVB devices. You should either use a powered USB 2.0 hub (with may be problematic, as the USB host driver on RPi has issues - and may cause data loss on high sustained ISOC traffic, specially when used with hubs) or a device that has its own power supply, like DVBSky T680C or S960.

Quoted 9 lines
Quoted 2 lines

Since this issue is "back",
I wouldn't be surprised this is unrelated to the fixes in 1.12.7 and 1.16.4.

The issue doesn't appear anymore on raspberry zero W since Mauro's commit. So it did improve on that platform.

Quoted 2 lines

It would be useful to see the output from dmesg (best thing would be after the issue occurs).

You can't, device is frozen.
Logs are not flushed to disk, and journalctl -f freeze before showing anything

You can use a serial port in order to get the logs. On a serial console, use something like:

         # dmesg -n 8

In order to make sure it will display all Kernel messages at console.

Quoted 3 lines
Quoted 1 line

Also what dvb hardware are you using?

I reproduced it with different two tuners: rtl2832U from RTL-SDR.COM and a TerraTec Cinergy T Stick+

None of them supports an external power supply.

Quoted 4 lines

You can found all the details here:
https://bugs.launchpad.net/raspbian/+bug/1819650

Quoted 92 lines

Sam

Quoted 88 lines · Sean Young

Hi Sam,

Quoted 17 lines · Samuel CHEMLA

Hi,

I am struggling with valgrind because it always complain with either :
ASan runtime does not come first in initial library list; you
should either link runtime to your application or manually preload it with LD_PRELOAD

     -> When I LD_PRELOAD, I'm getting a segfault, but I couldn't find

any core dump

or, if I link statically libasan with -static-libasan:
Shadow memory range interleaves with an existing memory mapping.
ASan cannot proceed correctly. ABORTING.
ASan shadow was supposed to be located in the
[0x00007fff7000-0x10007fff7fff] range.

Quoted 8 lines

I retested again on my raspberry zero W, and I confirm i cannot reproduce the hang.
Your fix did work on that device.
I am testing with same OS (raspbian with latest updates, same kernel), same configure options, same USB dongle... :-(
The only differences are CPU architecture (armv6 vs armv7), memory constraints, and I was not using the same channels.conf, I'll fix that today and re-check

Earlier you said "random hangs are back". When this happens, does the whole device become unresponsive or just dvbv5-zap? Since this issue is "back", I wouldn't be surprised this is unrelated to the fixes in 1.12.7 and 1.16.4.

It would be useful to see the output from dmesg (best thing would be after the issue occurs).

Also what dvb hardware are you using?

Thanks,

san

Quoted 47 lines

Sam

On 25/03/2019 18:08, Mauro Carvalho Chehab wrote:

Em Mon, 25 Mar 2019 17:33:30 +0100
Samuel CHEMLA <[email protected]> escreveu:

Hi guys,

I'm afraid I'm coming with sad news.
I just tried both stable-1.12 and stable-1.16 on a raspberry pi 2, and random hangs are back (see https://bugs.launchpad.net/raspbian/+bug/1819650
).
I previously test both branches on a raspberry zero and issues were gone (same raspbian version).
There may be more memory issues somewhere...

Could you test it with valgrind?

Sam

Le jeu. 21 mars 2019 ŕ 20:59, Gregor Jasny <[email protected]> a écrit :

Hello,

On 21.03.19 12:30, Mauro Carvalho Chehab wrote:

I went ahead and cherry-picked the relevant patches to -1.12, -1.14 and -1.16, and tested both dvbv5-zap and dvbv5-scan with all versions. So,

we can

release a new minor version for all those stable branches.

After the patches, on my tests, I didn't get any memory leaks or double-free issues.

I issues a new 1.12, 1.14, and 1.16 release.

Thanks,
Gregor

Thanks,
Mauro

Thanks,
Mauro

Thanks,
Mauro