mainline build failure due to cf21f328fcaf ("media: nxp: Add i.MX8 ISI driver")

Hi All,

The latest mainline kernel branch fails to build xtensa, mips, arm allmodconfig with the error:

drivers/media/platform/nxp/imx8-isi/imx8-isi-hw.c: In function 'mxc_isi_channel_set_inbuf':
drivers/media/platform/nxp/imx8-isi/imx8-isi-hw.c:33:5: error: "CONFIG_ARCH_DMA_ADDR_T_64BIT" is not defined, evaluates to 0 [-Werror=undef]
33 | #if CONFIG_ARCH_DMA_ADDR_T_64BIT

      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
drivers/media/platform/nxp/imx8-isi/imx8-isi-hw.c: In function 'mxc_isi_channel_set_outbuf':
drivers/media/platform/nxp/imx8-isi/imx8-isi-hw.c:51:5: error: "CONFIG_ARCH_DMA_ADDR_T_64BIT" is not defined, evaluates to 0 [-Werror=undef]
   51 | #if CONFIG_ARCH_DMA_ADDR_T_64BIT
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
drivers/media/platform/nxp/imx8-isi/imx8-isi-hw.c:66:5: error: "CONFIG_ARCH_DMA_ADDR_T_64BIT" is not defined, evaluates to 0 [-Werror=undef]
   66 | #if CONFIG_ARCH_DMA_ADDR_T_64BIT
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~

git bisect pointed to cf21f328fcaf ("media: nxp: Add i.MX8 ISI driver")

Hi All,

The latest mainline kernel branch fails to build xtensa, mips, arm allmodconfig with the error:

drivers/media/platform/nxp/imx8-isi/imx8-isi-hw.c: In function 'mxc_isi_channel_set_inbuf':
drivers/media/platform/nxp/imx8-isi/imx8-isi-hw.c:33:5: error: "CONFIG_ARCH_DMA_ADDR_T_64BIT" is not defined, evaluates to 0 [-Werror=undef]
33 | #if CONFIG_ARCH_DMA_ADDR_T_64BIT

      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
drivers/media/platform/nxp/imx8-isi/imx8-isi-hw.c: In function 'mxc_isi_channel_set_outbuf':
drivers/media/platform/nxp/imx8-isi/imx8-isi-hw.c:51:5: error: "CONFIG_ARCH_DMA_ADDR_T_64BIT" is not defined, evaluates to 0 [-Werror=undef]
   51 | #if CONFIG_ARCH_DMA_ADDR_T_64BIT
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
drivers/media/platform/nxp/imx8-isi/imx8-isi-hw.c:66:5: error: "CONFIG_ARCH_DMA_ADDR_T_64BIT" is not defined, evaluates to 0 [-Werror=undef]
   66 | #if CONFIG_ARCH_DMA_ADDR_T_64BIT
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~

git bisect pointed to cf21f328fcaf ("media: nxp: Add i.MX8 ISI driver")

I will be happy to test any patch or provide any extra log if needed.

The latest mainline kernel branch fails to build xtensa, mips, arm allmodconfig with the error:

drivers/media/platform/nxp/imx8-isi/imx8-isi-hw.c: In function 'mxc_isi_channel_set_inbuf':
drivers/media/platform/nxp/imx8-isi/imx8-isi-hw.c:33:5: error: "CONFIG_ARCH_DMA_ADDR_T_64BIT" is not defined, evaluates to 0 [-Werror=undef]
33 | #if CONFIG_ARCH_DMA_ADDR_T_64BIT

      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
drivers/media/platform/nxp/imx8-isi/imx8-isi-hw.c: In function 'mxc_isi_channel_set_outbuf':
drivers/media/platform/nxp/imx8-isi/imx8-isi-hw.c:51:5: error: "CONFIG_ARCH_DMA_ADDR_T_64BIT" is not defined, evaluates to 0 [-Werror=undef]
   51 | #if CONFIG_ARCH_DMA_ADDR_T_64BIT
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
drivers/media/platform/nxp/imx8-isi/imx8-isi-hw.c:66:5: error: "CONFIG_ARCH_DMA_ADDR_T_64BIT" is not defined, evaluates to 0 [-Werror=undef]
   66 | #if CONFIG_ARCH_DMA_ADDR_T_64BIT
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~

git bisect pointed to cf21f328fcaf ("media: nxp: Add i.MX8 ISI driver")

Thanks for the report. The fixes (see the mail from Laurent) apparently are still not mainlined (or am I missing something?), so let me add this report to the tracking to ensure this is not forgotten:

Quoted 4 lines · Linux regression tracking #adding (Thorsten Leemhuis)

Thanks for the report. The fixes (see the mail from Laurent) apparently are still not mainlined (or am I missing something?), so let me add this report to the tracking to ensure this is not forgotten:

Gaah. I was intending to apply the patch directly before rc1, but then I forgot about this issue.

Mauro: I'm currently really *really* fed up with the media tree. This exact same thing happened last merge window, where the media tree caused pointless build errors, and it took way too long to get the fixes the proper ways.

If something doesn't even build, it should damn well be fixed ASAP.

Last release it was imx290.c and PM support being disabled, and I had to apply the fix manually because it continued to not come in the proper way.

See commit 7b50567bdcad ("media: i2c: imx290: fix conditional function defintions").

But also see commit b928db940448 ("media: i2c: imx290: fix conditional function definitions"), which you *did* commit, but note this on that commit:

    AuthorDate: Tue Feb 7 17:13
    CommitDate: Sat Mar 18 08:44

so it took you a MONTH AND A HALF to react to a build failure.

And see this:

    git name-rev b928db940448
    b928db940448 tags/v6.4-rc1~161^2~458

ie that build fix that you finally committed came in *AFTER* the 6.3 release, even though the bug it fixes was introduced in the 6.3 merge window:

    git name-rev 02852c01f654
    02852c01f654 tags/v6.3-rc1~72^2~2^2~193

and now we're in the *EXACT*SAME* situation, with me applying a build fix directly, because you couldn't get it fixed in a timely manner.

Em Mon, 8 May 2023 09:27:28 -0700
Linus Torvalds <[email protected]> escreveu:

Quoted 7 lines

Quoted 4 lines · Linux regression tracking #adding (Thorsten Leemhuis)

Thanks for the report. The fixes (see the mail from Laurent) apparently are still not mainlined (or am I missing something?), so let me add this report to the tracking to ensure this is not forgotten:

Quoted 7 lines

Gaah. I was intending to apply the patch directly before rc1, but then I forgot about this issue.

Mauro: I'm currently really *really* fed up with the media tree. This exact same thing happened last merge window, where the media tree caused pointless build errors, and it took way too long to get the fixes the proper ways.

[...]

In the specific case of this fixup patch, I didn't identify it as a build issue, so it followed the usual workflow. We have a huge number of patches for media, and it usually takes some time to handle all of them. This one just followed the normal flow, as it didn't break Jenkins builds nor the subject mentioned anything about build breakage.

Quoted 16 lines

Em Mon, 8 May 2023 09:27:28 -0700
Linus Torvalds <[email protected]> escreveu:

Quoted 7 lines

Quoted 4 lines · Linux regression tracking #adding (Thorsten Leemhuis)

Thanks for the report. The fixes (see the mail from Laurent) apparently are still not mainlined (or am I missing something?), so let me add this report to the tracking to ensure this is not forgotten:

Quoted 7 lines

Gaah. I was intending to apply the patch directly before rc1, but then I forgot about this issue.

Mauro: I'm currently really *really* fed up with the media tree. This exact same thing happened last merge window, where the media tree caused pointless build errors, and it took way too long to get the fixes the proper ways.

Quoted 7 lines

[...]

In the specific case of this fixup patch, I didn't identify it as a build issue, so it followed the usual workflow. We have a huge number of patches for media, and it usually takes some time to handle all of them. This one just followed the normal flow, as it didn't break Jenkins builds nor the subject mentioned anything about build breakage.

Makes me wonder again if we should start adding

CC [email protected]

to any patches that fix regressions, that way maintainers and reviewers would have something to filter for -- and I would become aware of all regression fixes in the work, too.

Ciao, Thorsten

P.S.: BTW, let me tell regzbot that Linus merged the fix for the build failure.

#regzbot fix: ba0ad6ed89f

FWIW, the one for the gcc warnings[1] Laurent mentioned elsewhere in this thread is not merged yet afaics.

[1] https://lore.kernel.org/all/[email protected]/

Em Wed, 10 May 2023 11:02:57 +0200 "Linux regression tracking (Thorsten Leemhuis)" escreveu:

Quoted 27 lines · Mauro Carvalho Chehab

Quoted 16 lines

Em Mon, 8 May 2023 09:27:28 -0700
Linus Torvalds <[email protected]> escreveu:

Quoted 7 lines

Quoted 4 lines · Linux regression tracking #adding (Thorsten Leemhuis)

Thanks for the report. The fixes (see the mail from Laurent) apparently are still not mainlined (or am I missing something?), so let me add this report to the tracking to ensure this is not forgotten:

Quoted 7 lines

Gaah. I was intending to apply the patch directly before rc1, but then I forgot about this issue.

Mauro: I'm currently really *really* fed up with the media tree. This exact same thing happened last merge window, where the media tree caused pointless build errors, and it took way too long to get the fixes the proper ways.

Quoted 7 lines

[...]

In the specific case of this fixup patch, I didn't identify it as a build issue, so it followed the usual workflow. We have a huge number of patches for media, and it usually takes some time to handle all of them. This one just followed the normal flow, as it didn't break Jenkins builds nor the subject mentioned anything about build breakage.

Makes me wonder again if we should start adding

Quoted 5 lines

CC [email protected]

to any patches that fix regressions, that way maintainers and reviewers would have something to filter for -- and I would become aware of all regression fixes in the work, too.

Having some way that could be parsed by e-mail filters would be nice.

Quoted 11 lines

Ciao, Thorsten

P.S.: BTW, let me tell regzbot that Linus merged the fix for the build failure.

#regzbot fix: ba0ad6ed89f

FWIW, the one for the gcc warnings[1] Laurent mentioned elsewhere in this thread is not merged yet afaics.

[1] https://lore.kernel.org/all/[email protected]/

Just sent a pull request.

Btw, I did some changes at linux-media Jenkins instance to help early track some extra build issues. They're all against
https://git.linuxtv.org/media_stage.git/, which is the tree where we place media patches that are ready. We move them later, after a couple of days to https://git.linuxtv.org/media_tree.git/. So, if something bad happens, we have a chance to fix before setting them into a stone. With such changes, we now have:

1. https://builder.linuxtv.org/job/patchwork/

This is a pre-merge test. It tests patch per patch the PRs with patch sets ready to be merged, with W=1, allyesconfig/almodconfig[1] on x86_64. Builds drivers/media and drivers/staging/media.
This is there already for a long time;

2. https://builder.linuxtv.org/job/media_stage_clang/

Checks build with clang-12 on x86_64 with W=1. Builds drivers/media and drivers/staging/media with allyesconfig[1].

It was building with WERROR disabled, as some core macros were producing errors at the time I created it (and for a while). It was modified to enable WERROR as well.

3. https://builder.linuxtv.org/job/media_stage_gcc-pipeline/

It replaces another job that was just doing builds for x86_64 with W=1. Builds drivers/media and drivers/staging/media with different configurations[1]:

      x86_64: allyesconfig, allmodconfig, almodconfig with PM disabled;
      arm32: allyesconfig
      arm64: allyesconfig

4. https://builder.linuxtv.org/job/linux-media/

Does full builds with different configurations[1]:
x86_64: allyesconfig, allmodconfig, almodconfig with PM disabled;
arm32: allyesconfig
arm64: allyesconfig
docs: htmldocs and pdfdocs

I hope this will help avoiding future build regressions from our side. Feel free to suggest a couple of other configs that we might add to jobs (3) and (4).

I'm still adjusting the pipeline for (4), but currently, it is failing on an issue that seems unrelated with the media subsystem with gcc 10.2.1:

	  AR      drivers/built-in.a
	  AR      built-in.a
	  AR      vmlinux.a
	  LD      vmlinux.o
	vmlinux.o: warning: objtool: vmx_vcpu_enter_exit+0x2d8: call to vmread_error_trampoline() leaves .noinstr.text section
	vmlinux.o: warning: objtool: lkdtm_UNSET_SMEP+0xe1: relocation to !ENDBR: native_write_cr4+0x40

Is this a known regression? The media-stage tree is on the top of Kernel 6.4-rc1.

Regards,
Mauro

-

[1] On all builds, the jobs disable some symbols that should not affect media subsystem, to speedup the builds:

scripts/config -d MODULE_SIG -d KEYS -d IMA -d CONFIG_DEBUG_INFO -d SYSTEM_TRUSTED_KEYRING -d MODVERSIONS

Hi Linus,

Em Mon, 8 May 2023 09:27:28 -0700
Linus Torvalds <[email protected]> escreveu:

Quoted 42 lines

Quoted 4 lines · Linux regression tracking #adding (Thorsten Leemhuis)

Thanks for the report. The fixes (see the mail from Laurent) apparently are still not mainlined (or am I missing something?), so let me add this report to the tracking to ensure this is not forgotten:

Gaah. I was intending to apply the patch directly before rc1, but then I forgot about this issue.

Mauro: I'm currently really *really* fed up with the media tree. This exact same thing happened last merge window, where the media tree caused pointless build errors, and it took way too long to get the fixes the proper ways.

If something doesn't even build, it should damn well be fixed ASAP.

Last release it was imx290.c and PM support being disabled, and I had to apply the fix manually because it continued to not come in the proper way.

See commit 7b50567bdcad ("media: i2c: imx290: fix conditional function defintions").

But also see commit b928db940448 ("media: i2c: imx290: fix conditional function definitions"), which you *did* commit, but note this on that commit:

    AuthorDate: Tue Feb 7 17:13
    CommitDate: Sat Mar 18 08:44

so it took you a MONTH AND A HALF to react to a build failure.

And see this:

    git name-rev b928db940448
    b928db940448 tags/v6.4-rc1~161^2~458

ie that build fix that you finally committed came in *AFTER* the 6.3 release, even though the bug it fixes was introduced in the 6.3 merge window:

Quoted 5 lines

    git name-rev 02852c01f654
    02852c01f654 tags/v6.3-rc1~72^2~2^2~193

and now we're in the *EXACT*SAME* situation, with me applying a build fix directly, because you couldn't get it fixed in a timely manner.

Sorry for the mess. I'll work to improve the process to avoid this to happen again.

FYI, in order to reduce build issues, we have a Jenkins instance doing builds with gcc and CLANG at the media stage tree, before we even merge them at the main media development tree. They run with allyesconfig for x86_64 arch, with W=1:

https://builder.linuxtv.org/job/media_stage_clang/
https://builder.linuxtv.org/job/media_stage_gcc/

And another CI job testing bisect breakages as I receive pull requests, applying patch per patch and using both allyesconfig and allmodconfig, also on x86_64 arch with W=1:

https://builder.linuxtv.org/job/patchwork/

The rule is to not merge stuff on media tree if any of those jobs fail. I also fast-forward merging patches whose subject states that the build has failed.

In order to help with that, on normal situation, I usually take one week to merge stuff from media_stage into media_tree, doing rebases at media_stage if needed to avoid git bisect build breakages at media_tree (which is from where I send my update PRs to you).

Unfortunately, currently we don't have resources to do multiple randconfig on Jenkins, as the build machines on the server are very slow. Yet, I'll add CONFIG_PM disabled to the test set, as it seems to be a recurrent source of troubles those days. I'll also try to identify a couple of other randconfigs that would help to catch earlier problems like that. If some other problematic Kconfig variables comes to your mind, please feel free to suggest them for us to add to the CI automation.

-

In the specific case of this fixup patch, I didn't identify it as a build issue, so it followed the usual workflow. We have a huge number of patches for media, and it usually takes some time to handle all of them. This one just followed the normal flow, as it didn't break Jenkins builds nor the subject mentioned anything about build breakage.

And another CI job testing bisect breakages as I receive pull requests, applying patch per patch and using both allyesconfig and allmodconfig, also on x86_64 arch with W=1:

https://builder.linuxtv.org/job/patchwork/

The rule is to not merge stuff on media tree if any of those jobs fail. I also fast-forward merging patches whose subject states that the build has failed.

In order to help with that, on normal situation, I usually take one week to merge stuff from media_stage into media_tree, doing rebases at media_stage if needed to avoid git bisect build breakages at media_tree (which is from where I send my update PRs to you).

Unfortunately, currently we don't have resources to do multiple randconfig

Quoted 15 lines · Mauro Carvalho Chehab

And another CI job testing bisect breakages as I receive pull requests, applying patch per patch and using both allyesconfig and allmodconfig, also on x86_64 arch with W=1:

https://builder.linuxtv.org/job/patchwork/

The rule is to not merge stuff on media tree if any of those jobs fail. I also fast-forward merging patches whose subject states that the build has failed.

In order to help with that, on normal situation, I usually take one week to merge stuff from media_stage into media_tree, doing rebases at media_stage if needed to avoid git bisect build breakages at media_tree (which is from where I send my update PRs to you).

Quoted 1 line

Unfortunately, currently we don't have resources to do multiple randconfig

Is you media staging tree included in LKP (kernel test robot)? You would get huge build coverage after every push to your staging repo.

Hi Krzysztof,

Quoted 20 lines · Krzysztof Kozlowski

Quoted 10 lines · Mauro Carvalho Chehab

And another CI job testing bisect breakages as I receive pull requests, applying patch per patch and using both allyesconfig and allmodconfig, also on x86_64 arch with W=1:

https://builder.linuxtv.org/job/patchwork/

The rule is to not merge stuff on media tree if any of those jobs fail. I also fast-forward merging patches whose subject states that the build has failed.

Quoted 6 lines

In order to help with that, on normal situation, I usually take one week to merge stuff from media_stage into media_tree, doing rebases at media_stage if needed to avoid git bisect build breakages at media_tree (which is from where I send my update PRs to you).

Unfortunately, currently we don't have resources to do multiple randconfig

Is you media staging tree included in LKP (kernel test robot)? You would get huge build coverage after every push to your staging repo.

As (multiple[*[) fixes for the build issues were submitted before the opening of the merge window, there must have been some build coverage, with even some people acting upon it...

[*] General note, not limited to media: please apply build fixes and
regression fixes ASAP, to avoid multiple people running into the same issues, and spending time on bisecting, investigating, fixing, ...
Thanks a lot!

Em Mon, 15 May 2023 09:46:41 +0200
Geert Uytterhoeven <[email protected]> escreveu:

Quoted 19 lines

Hi Krzysztof,

Quoted 16 lines · Krzysztof Kozlowski

Quoted 15 lines · Mauro Carvalho Chehab

And another CI job testing bisect breakages as I receive pull requests, applying patch per patch and using both allyesconfig and allmodconfig, also on x86_64 arch with W=1:

https://builder.linuxtv.org/job/patchwork/

The rule is to not merge stuff on media tree if any of those jobs fail. I also fast-forward merging patches whose subject states that the build has failed.

In order to help with that, on normal situation, I usually take one week to merge stuff from media_stage into media_tree, doing rebases at media_stage if needed to avoid git bisect build breakages at media_tree (which is from where I send my update PRs to you).

Quoted 4 lines

Quoted 4 lines

Quoted 1 line

Unfortunately, currently we don't have resources to do multiple randconfig

Is you media staging tree included in LKP (kernel test robot)? You would get huge build coverage after every push to your staging repo.

No idea, as I don't know where LKP settings are stored, nor what frequency it is doing builds from git://linuxtv.org/media_stage.git, if any. Do you know where we can check such configuration?

In the end, patches there will end going to linux-next, so at least some sort of coverage is there, but I'm not sure if LKP will always reply to linux-media if the patch causing build regressions is there.

While being helpful, one problem with LKP is that it is hard to filter out reports per git tree. The only way to check if the report is applicable to media trees seems to be looking inside the e-mail's body.

Quoted 16 lines · Krzysztof Kozlowski

Quoted 15 lines · Mauro Carvalho Chehab

And another CI job testing bisect breakages as I receive pull requests, applying patch per patch and using both allyesconfig and allmodconfig, also on x86_64 arch with W=1:

https://builder.linuxtv.org/job/patchwork/

The rule is to not merge stuff on media tree if any of those jobs fail. I also fast-forward merging patches whose subject states that the build has failed.

In order to help with that, on normal situation, I usually take one week to merge stuff from media_stage into media_tree, doing rebases at media_stage if needed to avoid git bisect build breakages at media_tree (which is from where I send my update PRs to you).

Quoted 4 lines

Quoted 1 line

Unfortunately, currently we don't have resources to do multiple randconfig

Is you media staging tree included in LKP (kernel test robot)? You would get huge build coverage after every push to your staging repo.

As (multiple[*[) fixes for the build issues were submitted before the opening of the merge window, there must have been some build coverage, with even some people acting upon it...

[*] General note, not limited to media: please apply build fixes and
regression fixes ASAP, to avoid multiple people running into the same issues, and spending time on bisecting, investigating, fixing, ...
Thanks a lot!

End result: you and the media tree is on my shit-list, and I will not take any pull requests from you that aren't just fixes.

Not just this release, but the next one.

I'm expected to be flamed for getting in the way, but whatever. To me this decision lacks any kind of consideration toward who will be affected. This will hit those that makes the new features and are working hard to convince their customers to go mainline first.

Punishment and shame is not something I encourage or think is nice in general.