Skip to contentSkip to conversation

fixing the last failures in randconfig builds

PATCH In review Arnd Bergmann 4 participants 19 messages Feb 16, 2018 – Feb 23, 2018

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

cover fixing the last failures in randconfig builds

I've recently done some build testing on 4.14-stable and to my surprise found only a handful of bugfixes were required to build all x86 and arm64 randconfig output without any warnings using gcc-7.2.1. I have submitted the remaining trivial patches now, and this is the non-trivial remaining part, where we turn off some options that are known to cause problems in randconfig builds.

While I've carried the same patches in my randconfig test tree for a long time, I wasn't entirely sure about my approach and never submitted them for inclusion. Now that this is literally all that is remains, I'd like to propose all three for inclusion.

If someone has another idea for how to address these, that's fine with me as well, as long as we can make sure at least COMPILE_TEST is enabled for randconfig builds by default (everything else can be derived from that if necessary).

Note that the 4.15 release has a couple of build-time regressions that I have sent bugfixes for but that so far have not made it in, and 4.16-rc still has a number of other issues without patches.

Arnd

Arnd Bergmann (3):
Kconfig: disable PROFILE_ALL_BRANCHES for compile testing
Kconfig: improve handling for all{rand,yes,no,}.config fragments
Kconfig: add a default allrandom.config

 kernel/configs/allrandom.config |  6 ++++++
 kernel/trace/Kconfig            |  1 +
 scripts/kconfig/conf.c          | 27 +++++++++++++++++++--------
 3 files changed, 26 insertions(+), 8 deletions(-)
 create mode 100644 kernel/configs/allrandom.config
Signature · 1 line
2.9.0
#1 lore raw

PATCH 1/3 Kconfig: disable PROFILE_ALL_BRANCHES for compile testing

This can easily double the time for compiling a driver but does not provide any benefit for the compile tester, so it's better left disabled.

In addition, any 'inline' function that is not also 'static' and that contains an 'if' causes a warning like

include/linux/string.h:212:2: note: in expansion of macro 'if'
if (strscpy(p, q, p_size < q_size ? p_size : q_size) < 0)
^~
include/linux/compiler.h:162:4: warning: '______f' is static but declared in inline function 'strcpy' which is not static

without this patch, and I could not come up with a nice fix for that. In combination with my patch to always enable 'CONFIG_COMPILE_TEST' during 'randconfig' builds, we can at least hide these warnings for most users.

---
 kernel/trace/Kconfig | 1 +
 1 file changed, 1 insertion(+)
kernel/trace/Kconfig +1 -0
index 434c840e2d82..faaf687b13b1 100644
@@ -345,6 +345,7 @@ config PROFILE_ANNOTATED_BRANCHES
 config PROFILE_ALL_BRANCHES
 	bool "Profile all if conditionals"
 	select TRACE_BRANCH_PROFILING
+	depends on !COMPILE_TEST
 	help
 	  This tracer profiles all branch conditions. Every if ()
 	  taken in the kernel is recorded whether it hit or miss.
Signature · 1 line
2.9.0
maintainer
#2 lore raw

Re: [PATCH 1/3] Kconfig: disable PROFILE_ALL_BRANCHES for compile testing

Quoted 11 lines · On Fri, 16 Feb 2018 22:41:11 +0100 Arnd Bergmann <[email protected]> wrote:

This can easily double the time for compiling a driver but does not provide any benefit for the compile tester, so it's better left disabled.

In addition, any 'inline' function that is not also 'static' and that contains an 'if' causes a warning like

include/linux/string.h:212:2: note: in expansion of macro 'if'
if (strscpy(p, q, p_size < q_size ? p_size : q_size) < 0)
^~
include/linux/compiler.h:162:4: warning: '______f' is static but declared in inline function 'strcpy' which is not static

Quoted 4 lines

without this patch, and I could not come up with a nice fix for that. In combination with my patch to always enable 'CONFIG_COMPILE_TEST' during 'randconfig' builds, we can at least hide these warnings for most users.

This looks like it fixes the same issue that was already fixed and is in Linus's tree.

http://lkml.kernel.org/r/[email protected]

See commit 68e76e034b6b1 ("tracing: Prevent PROFILE_ALL_BRANCHES when FORTIFY_SOURCE=y")

-- Steve

#3 lore raw

Re: [PATCH 1/3] Kconfig: disable PROFILE_ALL_BRANCHES for compile testing

Quoted 19 lines · Steven Rostedt
Quoted 11 lines · On Fri, 16 Feb 2018 22:41:11 +0100 Arnd Bergmann <[email protected]> wrote:

This can easily double the time for compiling a driver but does not provide any benefit for the compile tester, so it's better left disabled.

In addition, any 'inline' function that is not also 'static' and that contains an 'if' causes a warning like

include/linux/string.h:212:2: note: in expansion of macro 'if'
if (strscpy(p, q, p_size < q_size ? p_size : q_size) < 0)
^~
include/linux/compiler.h:162:4: warning: '______f' is static but declared in inline function 'strcpy' which is not static

Quoted 4 lines

without this patch, and I could not come up with a nice fix for that. In combination with my patch to always enable 'CONFIG_COMPILE_TEST' during 'randconfig' builds, we can at least hide these warnings for most users.

Quoted 7 lines

This looks like it fixes the same issue that was already fixed and is in Linus's tree.

http://lkml.kernel.org/r/[email protected]

See commit 68e76e034b6b1 ("tracing: Prevent PROFILE_ALL_BRANCHES when FORTIFY_SOURCE=y")

Ah, right. I missed that when I wrote the new changelog text for this old patch of mine. It also means I should rebase the patch so it applies on mainline, as I still want PROFILE_ALL_BRANCHES to be disabled in COMPILE_TEST kernels for the build speed aspect.

Greg, could you add the 68e76e034b6b1 commit to 4.14-stable and 4.15-stable in the meantime?

Arnd

#4 lore raw

Re: [PATCH 1/3] Kconfig: disable PROFILE_ALL_BRANCHES for compile testing

Quoted 25 lines · Arnd Bergmann
Quoted 24 lines · Steven Rostedt
Quoted 11 lines · On Fri, 16 Feb 2018 22:41:11 +0100 Arnd Bergmann <[email protected]> wrote:

This can easily double the time for compiling a driver but does not provide any benefit for the compile tester, so it's better left disabled.

In addition, any 'inline' function that is not also 'static' and that contains an 'if' causes a warning like

include/linux/string.h:212:2: note: in expansion of macro 'if'
if (strscpy(p, q, p_size < q_size ? p_size : q_size) < 0)
^~
include/linux/compiler.h:162:4: warning: '______f' is static but declared in inline function 'strcpy' which is not static

Quoted 4 lines

without this patch, and I could not come up with a nice fix for that. In combination with my patch to always enable 'CONFIG_COMPILE_TEST' during 'randconfig' builds, we can at least hide these warnings for most users.

This looks like it fixes the same issue that was already fixed and is in Linus's tree.

http://lkml.kernel.org/r/[email protected]

Quoted 7 lines
Quoted 2 lines

See commit 68e76e034b6b1 ("tracing: Prevent PROFILE_ALL_BRANCHES when FORTIFY_SOURCE=y")

Ah, right. I missed that when I wrote the new changelog text for this old patch of mine. It also means I should rebase the patch so it applies on mainline, as I still want PROFILE_ALL_BRANCHES to be disabled in COMPILE_TEST kernels for the build speed aspect.

I retested on top of that patch and found a couple of other warnings show up in an allmodconfig build with PROFILE_ALL_BRANCHES:

lib/zstd/decompress.c: In function 'ZSTD_decompressStream':
lib/zstd/decompress.c:416:2: error: argument 1 null where non-null expected [-Werror=nonnull] drivers/crypto/qat/qat_common/qat_algs.c: In function 'qat_alg_do_precomputes':
drivers/crypto/qat/qat_common/qat_algs.c:156:7: error: argument 1 range [18446744071562067968, 18446744073709551615] exceeds maximum object size 9223372036854775807 [-Werror=alloc-size-larger-than=] drivers/isdn/hardware/eicon/message.c: In function 'mixer_notify_update':
drivers/isdn/hardware/eicon/message.c:11162:54: error: array subscript is above array bounds [-Werror=array-bounds]

     ((CAPI_MSG *) msg)->info.facility_req.structs[1] =
LI_REQ_SILENT_UPDATE & 0xff;
     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
drivers/isdn/hardware/eicon/message.c:11163:54: error: array subscript
is above array bounds [-Werror=array-bounds]
     ((CAPI_MSG *) msg)->info.facility_req.structs[2] =
LI_REQ_SILENT_UPDATE >> 8;
     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~
drivers/isdn/hardware/eicon/message.c:11164:54: error: array subscript
is above array bounds [-Werror=array-bounds]
     ((CAPI_MSG *) msg)->info.facility_req.structs[3] = 0;

All those are nonsense AFAICT, and we see them only because the "if()" override ends up confusing gcc's value-range tracking in the same way it used to cause lots of -Wmaybe-uninitialized warnings (which we just disable these days with PROFILE_ALL_BRANCHES).

Arnd

maintainer
#5 lore raw

Re: [PATCH 1/3] Kconfig: disable PROFILE_ALL_BRANCHES for compile testing

Quoted 12 lines · On Fri, 16 Feb 2018 23:40:22 +0100 Arnd Bergmann <[email protected]> wrote:
     ((CAPI_MSG *) msg)->info.facility_req.structs[1] =
LI_REQ_SILENT_UPDATE & 0xff;
     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
drivers/isdn/hardware/eicon/message.c:11163:54: error: array subscript
is above array bounds [-Werror=array-bounds]
     ((CAPI_MSG *) msg)->info.facility_req.structs[2] =
LI_REQ_SILENT_UPDATE >> 8;
     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~
drivers/isdn/hardware/eicon/message.c:11164:54: error: array subscript
is above array bounds [-Werror=array-bounds]
     ((CAPI_MSG *) msg)->info.facility_req.structs[3] = 0;
Quoted 4 lines

All those are nonsense AFAICT, and we see them only because the "if()" override ends up confusing gcc's value-range tracking in the same way it used to cause lots of -Wmaybe-uninitialized warnings (which we just disable these days with PROFILE_ALL_BRANCHES).

I'm fine with your patch then.

-- Steve

#6 lore raw

Re: [PATCH 1/3] Kconfig: disable PROFILE_ALL_BRANCHES for compile testing

In reply to Arnd Bergmann
Quoted 28 lines · Arnd Bergmann
Quoted 19 lines · Steven Rostedt
Quoted 11 lines · On Fri, 16 Feb 2018 22:41:11 +0100 Arnd Bergmann <[email protected]> wrote:

This can easily double the time for compiling a driver but does not provide any benefit for the compile tester, so it's better left disabled.

In addition, any 'inline' function that is not also 'static' and that contains an 'if' causes a warning like

include/linux/string.h:212:2: note: in expansion of macro 'if'
if (strscpy(p, q, p_size < q_size ? p_size : q_size) < 0)
^~
include/linux/compiler.h:162:4: warning: '______f' is static but declared in inline function 'strcpy' which is not static

Quoted 4 lines

without this patch, and I could not come up with a nice fix for that. In combination with my patch to always enable 'CONFIG_COMPILE_TEST' during 'randconfig' builds, we can at least hide these warnings for most users.

Quoted 7 lines

This looks like it fixes the same issue that was already fixed and is in Linus's tree.

http://lkml.kernel.org/r/[email protected]

See commit 68e76e034b6b1 ("tracing: Prevent PROFILE_ALL_BRANCHES when FORTIFY_SOURCE=y")

Quoted 7 lines

Ah, right. I missed that when I wrote the new changelog text for this old patch of mine. It also means I should rebase the patch so it applies on mainline, as I still want PROFILE_ALL_BRANCHES to be disabled in COMPILE_TEST kernels for the build speed aspect.

Greg, could you add the 68e76e034b6b1 commit to 4.14-stable and 4.15-stable in the meantime?

It's already in 4.15, so I'll just queue it up to 4.14.y.

thanks,

greg k-h

#7 lore raw

Re: [PATCH 1/3] Kconfig: disable PROFILE_ALL_BRANCHES for compile testing

In reply to Arnd Bergmann

2018-02-17 6:41 GMT+09:00 Arnd Bergmann <[email protected]>:

Quoted 11 lines

This can easily double the time for compiling a driver but does not provide any benefit for the compile tester, so it's better left disabled.

In addition, any 'inline' function that is not also 'static' and that contains an 'if' causes a warning like

include/linux/string.h:212:2: note: in expansion of macro 'if'
if (strscpy(p, q, p_size < q_size ? p_size : q_size) < 0)
^~
include/linux/compiler.h:162:4: warning: '______f' is static but declared in inline function 'strcpy' which is not static

Quoted 7 lines

without this patch, and I could not come up with a nice fix for that. In combination with my patch to always enable 'CONFIG_COMPILE_TEST' during 'randconfig' builds, we can at least hide these warnings for most users.

I took a look at this just in case
because the subject is prefixed with "Kconfig:",
but it is actually "trace:".

I expect "Kconfig:" for core changes of Kconfig.

Using precise patch prefix would get more attention from right people.

Quoted 23 lines
---
 kernel/trace/Kconfig | 1 +
 1 file changed, 1 insertion(+)
kernel/trace/Kconfig +1 -1
index 434c840e2d82..faaf687b13b1 100644
@@ -345,6 +345,7 @@ config PROFILE_ANNOTATED_BRANCHES
 config PROFILE_ALL_BRANCHES
        bool "Profile all if conditionals"
        select TRACE_BRANCH_PROFILING
+       depends on !COMPILE_TEST
        help
          This tracer profiles all branch conditions. Every if ()
          taken in the kernel is recorded whether it hit or miss.
--

2.9.0

--
To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 2 lines
Best Regards
Masahiro Yamada
#8 lore raw

PATCH 2/3 Kconfig: improve handling for all{rand,yes,no,}.config fragments

In reply to Arnd Bergmann

The kernel currently supports two methods of dealing with config fragments in the tree:

a) Running "make foo.config" looks for arch/$(ARCH)/configs/foo.config and kernel/configs/foo.config, and applies the defaults from those files on top of the current configuration.

b) Running "KCONFIG_ALLCONFIG=1 make randconfig" (or the equivalent allmodconfig/allnoconfig/allyesconfig/alldefconfig) will look for a "allrandconfig.config" file in the current directory or the top of the $(srctree). These are used as defaults before we generate the remaining options.

This is rather inconsistent, and prevents us from easily shipping good defaults for "randconfig". I'm extending the logic here so that the second case also looks for the hardcoded file names in the standard directories (first arch/$(ARCH)/configs/, then kernel/configs) in the source tree.

---
 scripts/kconfig/conf.c | 27 +++++++++++++++++++--------
 1 file changed, 19 insertions(+), 8 deletions(-)
scripts/kconfig/conf.c +19 -8
index 866369f10ff8..848bf4d15e9a 100644
@@ -493,8 +493,9 @@ int main(int ac, char **av)
 {
 	const char *progname = av[0];
 	int opt;
-	const char *name, *defconfig_file = NULL /* gcc uninit */;
+	const char *arch, *name, *defconfig_file = NULL /* gcc uninit */;
 	struct stat tmpstat;
+	char fullname[PATH_MAX+1];
 
 	setlocale(LC_ALL, "");
 	bindtextdomain(PACKAGE, LOCALEDIR);
@@ -621,14 +622,24 @@ int main(int ac, char **av)
 		case randconfig:	name = "allrandom.config"; break;
 		default: break;
 		}
-		if (conf_read_simple(name, S_DEF_USER) &&
-		    conf_read_simple("all.config", S_DEF_USER)) {
-			fprintf(stderr,
-				_("*** KCONFIG_ALLCONFIG set, but no \"%s\" or \"all.config\" file found\n"),
-				name);
-			exit(1);
+		/* try ./name, arch/$(ARCH)/configs/name and kernel/config/name */
+		if (!conf_read_simple(name, S_DEF_USER))
+			break;
+		arch = getenv("ARCH");
+		if (arch) {
+			snprintf(fullname, sizeof(fullname), "arch/%s/configs/%s",
+				 arch, name);
+			if (!conf_read_simple(fullname, S_DEF_USER))
+				break;
 		}
-		break;
+		snprintf(fullname, sizeof(fullname), "kernel/configs/%s", name);
+		if (!conf_read_simple(fullname, S_DEF_USER))
+			break;
+
+		fprintf(stderr,
+			_("*** KCONFIG_ALLCONFIG set, but no \"%s\" or \"all.config\" file found\n"),
+				name);
+		exit(1);
 	default:
 		break;
 	}
Signature · 1 line
2.9.0
#9 lore raw

Re: [PATCH 2/3] Kconfig: improve handling for all{rand,yes,no,}.config fragments

2018-02-17 6:41 GMT+09:00 Arnd Bergmann <[email protected]>:

Quoted 49 lines

The kernel currently supports two methods of dealing with config fragments in the tree:

a) Running "make foo.config" looks for arch/$(ARCH)/configs/foo.config and kernel/configs/foo.config, and applies the defaults from those files on top of the current configuration.

b) Running "KCONFIG_ALLCONFIG=1 make randconfig" (or the equivalent allmodconfig/allnoconfig/allyesconfig/alldefconfig) will look for a "allrandconfig.config" file in the current directory or the top of the $(srctree). These are used as defaults before we generate the remaining options.

This is rather inconsistent, and prevents us from easily shipping good defaults for "randconfig". I'm extending the logic here so that the second case also looks for the hardcoded file names in the standard directories (first arch/$(ARCH)/configs/, then kernel/configs) in the source tree.

---
 scripts/kconfig/conf.c | 27 +++++++++++++++++++--------
 1 file changed, 19 insertions(+), 8 deletions(-)
scripts/kconfig/conf.c +2 -7
index 866369f10ff8..848bf4d15e9a 100644
@@ -493,8 +493,9 @@ int main(int ac, char **av)
 {
        const char *progname = av[0];
        int opt;
-       const char *name, *defconfig_file = NULL /* gcc uninit */;
+       const char *arch, *name, *defconfig_file = NULL /* gcc uninit */;
        struct stat tmpstat;
+       char fullname[PATH_MAX+1];
 
        setlocale(LC_ALL, "");
        bindtextdomain(PACKAGE, LOCALEDIR);
@@ -621,14 +622,24 @@ int main(int ac, char **av)
                case randconfig:        name = "allrandom.config"; break;
                default: break;
                }
-               if (conf_read_simple(name, S_DEF_USER) &&
-                   conf_read_simple("all.config", S_DEF_USER)) {
-                       fprintf(stderr,
-                               _("*** KCONFIG_ALLCONFIG set, but no \"%s\" or \"all.config\" file found\n"),
-                               name);
-                       exit(1);

scripts/kconfig/conf.c int main(int ac, char **av) · line 634

Quoted 10 lines
scripts/kconfig/conf.c +9 -0
+               /* try ./name, arch/$(ARCH)/configs/name and kernel/config/name */
+               if (!conf_read_simple(name, S_DEF_USER))
+                       break;
+               arch = getenv("ARCH");
+               if (arch) {
+                       snprintf(fullname, sizeof(fullname), "arch/%s/configs/%s",
+                                arch, name);
+                       if (!conf_read_simple(fullname, S_DEF_USER))
+                               break;
                }

I am not a big fan of hard-coding the kernel directory structure.

We already do this [1], but I am thinking of kicking this out.
[1] https://github.com/torvalds/linux/blob/v4.16-rc1/scripts/kconfig/confdata.c#L33

BTW, I am trying to compiler capability check to Kconfig. If this work is done, I hope some "depends on !COMPILE_TEST" will go away (but not all?)
https://patchwork.kernel.org/patch/10225375/

Quoted 19 lines
scripts/kconfig/conf.c +8 -2
-               break;
+               snprintf(fullname, sizeof(fullname), "kernel/configs/%s", name);
+               if (!conf_read_simple(fullname, S_DEF_USER))
+                       break;
+
+               fprintf(stderr,
+                       _("*** KCONFIG_ALLCONFIG set, but no \"%s\" or \"all.config\" file found\n"),
+                               name);
+               exit(1);
        default:
                break;
        }
--

2.9.0

--
To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 2 lines
Best Regards
Masahiro Yamada
#10 lore raw

Re: [PATCH 2/3] Kconfig: improve handling for all{rand,yes,no,}.config fragments

Quoted 26 lines · Masahiro Yamada

2018-02-17 6:41 GMT+09:00 Arnd Bergmann <[email protected]>:

Quoted 10 lines
scripts/kconfig/conf.c +0 -6
@@ -621,14 +622,24 @@ int main(int ac, char **av)
                case randconfig:        name = "allrandom.config"; break;
                default: break;
                }
-               if (conf_read_simple(name, S_DEF_USER) &&
-                   conf_read_simple("all.config", S_DEF_USER)) {
-                       fprintf(stderr,
-                               _("*** KCONFIG_ALLCONFIG set, but no \"%s\" or \"all.config\" file found\n"),
-                               name);
-                       exit(1);

scripts/kconfig/conf.c int main(int ac, char **av) · line 634

Quoted 10 lines
scripts/kconfig/conf.c +9 -0
+               /* try ./name, arch/$(ARCH)/configs/name and kernel/config/name */
+               if (!conf_read_simple(name, S_DEF_USER))
+                       break;
+               arch = getenv("ARCH");
+               if (arch) {
+                       snprintf(fullname, sizeof(fullname), "arch/%s/configs/%s",
+                                arch, name);
+                       if (!conf_read_simple(fullname, S_DEF_USER))
+                               break;
                }

I am not a big fan of hard-coding the kernel directory structure.

Quoted 2 lines

We already do this [1], but I am thinking of kicking this out.
[1] https://github.com/torvalds/linux/blob/v4.16-rc1/scripts/kconfig/confdata.c#L33

Ok, I see. How about adding a way to detect that we are build-testing with randconfig instead?

Quoted 4 lines

BTW, I am trying to compiler capability check to Kconfig. If this work is done, I hope some "depends on !COMPILE_TEST" will go away (but not all?)
https://patchwork.kernel.org/patch/10225375/

I think most of the 'depends on !COMPILE_TEST' are for cases that impact compile-testing in some way (more false-positive warnings, fewer real warnings, much slower compile speed), rather than a lack of check for a specific compiler version.

We might be able to replace the CONFIG_STANDALONE and CONFIG_PREVENT_FIRMWARE_BUILD with Kconfig $(shell) checks if you think that leads to an improvement.

Arnd

#11 lore raw

Re: [PATCH 2/3] Kconfig: improve handling for all{rand,yes,no,}.config fragments

2018-02-20 18:59 GMT+09:00 Arnd Bergmann <[email protected]>:

Quoted 28 lines
Quoted 26 lines · Masahiro Yamada

2018-02-17 6:41 GMT+09:00 Arnd Bergmann <[email protected]>:

Quoted 10 lines
scripts/kconfig/conf.c +0 -6
@@ -621,14 +622,24 @@ int main(int ac, char **av)
                case randconfig:        name = "allrandom.config"; break;
                default: break;
                }
-               if (conf_read_simple(name, S_DEF_USER) &&
-                   conf_read_simple("all.config", S_DEF_USER)) {
-                       fprintf(stderr,
-                               _("*** KCONFIG_ALLCONFIG set, but no \"%s\" or \"all.config\" file found\n"),
-                               name);
-                       exit(1);

scripts/kconfig/conf.c int main(int ac, char **av) · line 634

Quoted 10 lines
scripts/kconfig/conf.c +9 -0
+               /* try ./name, arch/$(ARCH)/configs/name and kernel/config/name */
+               if (!conf_read_simple(name, S_DEF_USER))
+                       break;
+               arch = getenv("ARCH");
+               if (arch) {
+                       snprintf(fullname, sizeof(fullname), "arch/%s/configs/%s",
+                                arch, name);
+                       if (!conf_read_simple(fullname, S_DEF_USER))
+                               break;
                }

I am not a big fan of hard-coding the kernel directory structure.

Quoted 5 lines
Quoted 2 lines

We already do this [1], but I am thinking of kicking this out.
[1] https://github.com/torvalds/linux/blob/v4.16-rc1/scripts/kconfig/confdata.c#L33

Ok, I see. How about adding a way to detect that we are build-testing with randconfig instead?

How about implementing something in scripts/kconfig/Makefile?

merge_config collects config fragments into 'configfiles'

I was thinking of a similar thing.

If KCONFIG_ALLCONFIG is 1,
scripts/kconfig/Makefile searches
'./', '$srctree/', 'arch/$(SRCARCH)/configs/', 'kernel/configs/' and sets the found file in it,

If KCONFIG_ALLCONFIG already contains a file name, it is used as-is.

scripts/kconfig/conf.c will be simplified.

Quoted 5 lines
Quoted 4 lines

BTW, I am trying to compiler capability check to Kconfig. If this work is done, I hope some "depends on !COMPILE_TEST" will go away (but not all?)
https://patchwork.kernel.org/patch/10225375/

Quoted 9 lines

I think most of the 'depends on !COMPILE_TEST' are for cases that impact compile-testing in some way (more false-positive warnings, fewer real warnings, much slower compile speed), rather than a lack of check for a specific compiler version.

We might be able to replace the CONFIG_STANDALONE and CONFIG_PREVENT_FIRMWARE_BUILD with Kconfig $(shell) checks if you think that leads to an improvement.

OK, then your kernel/configs/allrandom.config approach will be useful.

Signature · 2 lines
Best Regards
Masahiro Yamada
#12 lore raw

Re: [PATCH 2/3] Kconfig: improve handling for all{rand,yes,no,}.config fragments

Quoted 43 lines · Masahiro Yamada

2018-02-20 18:59 GMT+09:00 Arnd Bergmann <[email protected]>:

Quoted 28 lines
Quoted 26 lines · Masahiro Yamada

2018-02-17 6:41 GMT+09:00 Arnd Bergmann <[email protected]>:

Quoted 10 lines
scripts/kconfig/conf.c +0 -6
@@ -621,14 +622,24 @@ int main(int ac, char **av)
                case randconfig:        name = "allrandom.config"; break;
                default: break;
                }
-               if (conf_read_simple(name, S_DEF_USER) &&
-                   conf_read_simple("all.config", S_DEF_USER)) {
-                       fprintf(stderr,
-                               _("*** KCONFIG_ALLCONFIG set, but no \"%s\" or \"all.config\" file found\n"),
-                               name);
-                       exit(1);

scripts/kconfig/conf.c int main(int ac, char **av) · line 634

Quoted 10 lines
scripts/kconfig/conf.c +9 -0
+               /* try ./name, arch/$(ARCH)/configs/name and kernel/config/name */
+               if (!conf_read_simple(name, S_DEF_USER))
+                       break;
+               arch = getenv("ARCH");
+               if (arch) {
+                       snprintf(fullname, sizeof(fullname), "arch/%s/configs/%s",
+                                arch, name);
+                       if (!conf_read_simple(fullname, S_DEF_USER))
+                               break;
                }

I am not a big fan of hard-coding the kernel directory structure.

Quoted 5 lines
Quoted 2 lines

We already do this [1], but I am thinking of kicking this out.
[1] https://github.com/torvalds/linux/blob/v4.16-rc1/scripts/kconfig/confdata.c#L33

Ok, I see. How about adding a way to detect that we are build-testing with randconfig instead?

How about implementing something in scripts/kconfig/Makefile?

merge_config collects config fragments into 'configfiles'

I was thinking of a similar thing.

Quoted 8 lines

If KCONFIG_ALLCONFIG is 1,
scripts/kconfig/Makefile searches
'./', '$srctree/', 'arch/$(SRCARCH)/configs/', 'kernel/configs/' and sets the found file in it,

If KCONFIG_ALLCONFIG already contains a file name, it is used as-is.

scripts/kconfig/conf.c will be simplified.

Yes, good idea. I'm struggling a bit with the implementation (fitting it into the
simple-tagets rule), can you come up with a prototype?

We also need to decide what happens when e.g. both a ./all.config and arch/${SRCARCH}/configs/allrand.config file exist, should the first path take precedence or the more specific file?

Arnd

#13 lore raw

Re: [PATCH 2/3] Kconfig: improve handling for all{rand,yes,no,}.config fragments

2018-02-22 1:57 GMT+09:00 Arnd Bergmann <[email protected]>:

Quoted 54 lines
Quoted 43 lines · Masahiro Yamada

2018-02-20 18:59 GMT+09:00 Arnd Bergmann <[email protected]>:

Quoted 28 lines
Quoted 26 lines · Masahiro Yamada

2018-02-17 6:41 GMT+09:00 Arnd Bergmann <[email protected]>:

Quoted 10 lines
scripts/kconfig/conf.c +0 -6
@@ -621,14 +622,24 @@ int main(int ac, char **av)
                case randconfig:        name = "allrandom.config"; break;
                default: break;
                }
-               if (conf_read_simple(name, S_DEF_USER) &&
-                   conf_read_simple("all.config", S_DEF_USER)) {
-                       fprintf(stderr,
-                               _("*** KCONFIG_ALLCONFIG set, but no \"%s\" or \"all.config\" file found\n"),
-                               name);
-                       exit(1);

scripts/kconfig/conf.c int main(int ac, char **av) · line 634

Quoted 10 lines
scripts/kconfig/conf.c +9 -0
+               /* try ./name, arch/$(ARCH)/configs/name and kernel/config/name */
+               if (!conf_read_simple(name, S_DEF_USER))
+                       break;
+               arch = getenv("ARCH");
+               if (arch) {
+                       snprintf(fullname, sizeof(fullname), "arch/%s/configs/%s",
+                                arch, name);
+                       if (!conf_read_simple(fullname, S_DEF_USER))
+                               break;
                }

I am not a big fan of hard-coding the kernel directory structure.

Quoted 5 lines
Quoted 2 lines

We already do this [1], but I am thinking of kicking this out.
[1] https://github.com/torvalds/linux/blob/v4.16-rc1/scripts/kconfig/confdata.c#L33

Ok, I see. How about adding a way to detect that we are build-testing with randconfig instead?

How about implementing something in scripts/kconfig/Makefile?

merge_config collects config fragments into 'configfiles'

I was thinking of a similar thing.

Quoted 8 lines

If KCONFIG_ALLCONFIG is 1,
scripts/kconfig/Makefile searches
'./', '$srctree/', 'arch/$(SRCARCH)/configs/', 'kernel/configs/' and sets the found file in it,

If KCONFIG_ALLCONFIG already contains a file name, it is used as-is.

scripts/kconfig/conf.c will be simplified.

Quoted 8 lines

Yes, good idea. I'm struggling a bit with the implementation (fitting it into the
simple-tagets rule), can you come up with a prototype?

We also need to decide what happens when e.g. both a ./all.config and arch/${SRCARCH}/configs/allrand.config file exist, should the first path take precedence or the more specific file?

Hmm, maybe the following order?

[1] ./allrandom.config
[2] $(srctree)/allrandom.config
[3] ./all.config
[4] $(srctree)/all.config
[5] kernel/configs/allrandom.config + arch/$(SRCARCH)/configs/allrandom.config

I think you want to include
both kernel/configs/allrandom.config and
arch/$(SRCARCH)/configs/allrandom.config
if exist.

Some options are common, some are arch-specific.

This is how tinyconfig works.

Signature · 2 lines
Best Regards
Masahiro Yamada
#14 lore raw

Re: [PATCH 2/3] Kconfig: improve handling for all{rand,yes,no,}.config fragments

In reply to Arnd Bergmann

2018-02-22 1:57 GMT+09:00 Arnd Bergmann <[email protected]>:

Quoted 50 lines
Quoted 48 lines · Masahiro Yamada

2018-02-20 18:59 GMT+09:00 Arnd Bergmann <[email protected]>:

Quoted 28 lines
Quoted 26 lines · Masahiro Yamada

2018-02-17 6:41 GMT+09:00 Arnd Bergmann <[email protected]>:

Quoted 10 lines
scripts/kconfig/conf.c +0 -6
@@ -621,14 +622,24 @@ int main(int ac, char **av)
                case randconfig:        name = "allrandom.config"; break;
                default: break;
                }
-               if (conf_read_simple(name, S_DEF_USER) &&
-                   conf_read_simple("all.config", S_DEF_USER)) {
-                       fprintf(stderr,
-                               _("*** KCONFIG_ALLCONFIG set, but no \"%s\" or \"all.config\" file found\n"),
-                               name);
-                       exit(1);

scripts/kconfig/conf.c int main(int ac, char **av) · line 634

Quoted 10 lines
scripts/kconfig/conf.c +9 -0
+               /* try ./name, arch/$(ARCH)/configs/name and kernel/config/name */
+               if (!conf_read_simple(name, S_DEF_USER))
+                       break;
+               arch = getenv("ARCH");
+               if (arch) {
+                       snprintf(fullname, sizeof(fullname), "arch/%s/configs/%s",
+                                arch, name);
+                       if (!conf_read_simple(fullname, S_DEF_USER))
+                               break;
                }

I am not a big fan of hard-coding the kernel directory structure.

Quoted 5 lines
Quoted 2 lines

We already do this [1], but I am thinking of kicking this out.
[1] https://github.com/torvalds/linux/blob/v4.16-rc1/scripts/kconfig/confdata.c#L33

Ok, I see. How about adding a way to detect that we are build-testing with randconfig instead?

How about implementing something in scripts/kconfig/Makefile?

merge_config collects config fragments into 'configfiles'

I was thinking of a similar thing.

If KCONFIG_ALLCONFIG is 1,
scripts/kconfig/Makefile searches
'./', '$srctree/', 'arch/$(SRCARCH)/configs/', 'kernel/configs/' and sets the found file in it,

Quoted 7 lines
Quoted 3 lines

If KCONFIG_ALLCONFIG already contains a file name, it is used as-is.

scripts/kconfig/conf.c will be simplified.

Yes, good idea. I'm struggling a bit with the implementation (fitting it into the
simple-tagets rule), can you come up with a prototype?

Just a prototype for the Makefile part.
(We need to agree with the search order specification, though)

------------>8-------------
configs_dirs := $(srctree)/kernel/configs/ $(srctree)/arch/$(SRCARCH)/configs/

allconfig_files = $(if $(filter 1, $(KCONFIG_ALLCONFIG)),$(firstword \
        $(foreach f, $(1), $(wildcard $(addsuffix $(f), ./ $(srctree)/))) \
        $(foreach f, $(1), $(subst $(space),$(comma),$(wildcard
$(addsuffix $(f), $(configs_dirs)))))), \
        $(KCONFIG_ALLCONFIG))

allyesconfig: override KCONFIG_ALLCONFIG := $(call allconfig_files, allyes.config all.config)
allmodconfig: override KCONFIG_ALLCONFIG := $(call allconfig_files, allmoyes.config all.config)
(likewise for allnoconfig, alldefconfig)
randconfig: override KCONFIG_ALLCONFIG := $(call allconfig_files, allrandom.config all.config)
------------->8-----------

If a user specifies a file path in KCONFIG_ALLCONFIG, it is used as-is.

IF KCONFIG_ALLCONFIG=1, the input file is searched in the following order. ( [1] - [4] is the same as the current specification)

[1] ./allrandom.config
[2] $(srctree)/allrandom.config
[3] ./all.config
[4] $(srctree)/all.config
[5] $(srctree)/kernel/configs/allrandom.config + $(srctree)/arch/$(SRCARCH)/configs/allrandom.config
[6] $(srctree)/kernel/configs/all.config + $(srctree)/arch/$(SRCARCH)/configs/all.config

For [5] and [6], if multiple files are found, KCONFIG_ALLCONFIG is set to comma-separated file paths.

scripts/kconfig/conf.c must be tweaked a little bit if we need to read multiple config fragments.

Currently, tinyconfig for x86 comes from common part and arch-specific part. ./kernel/configs/tiny.config
./arch/x86/configs/tiny.config

What do you think?

Quoted 4 lines

We also need to decide what happens when e.g. both a ./all.config and arch/${SRCARCH}/configs/allrand.config file exist, should the first path take precedence or the more specific file?

Signature · 2 lines
Best Regards
Masahiro Yamada
#15 lore raw

PATCH 3/3 Kconfig: add a default allrandom.config

In reply to Arnd Bergmann

Building randconfig kernels frequently leads to build errors from drivers that have additional build dependencies, or that we don't want to build for compile-testing for some other reason.

We already have a couple of compile-time options that can deal with this problem, but the complete set is not documented well. We also have a method to override some options during 'make randconfig' and a couple of other make targets. However, we don't yet combine those two.

This adds a new allrandom.config file to turn on CONFIG_COMPILE_TEST, CONFIG_STANDALONE and CONFIG_PREVENT_FIRMWARE_BUILD for all randconfig builds. This in turn disables some other options that we already try to avoid in 'allmodconfig' builds and that make no sense in a general randconfig build.

Building with 'make randconfig KCONFIG_ALLCONFIG=1' should now always succeed without warnings on x86 and arm64, aside from recent regressions. Other architectures probably need additional bugfixes.

---
 kernel/configs/allrandom.config | 6 ++++++
 1 file changed, 6 insertions(+)
 create mode 100644 kernel/configs/allrandom.config
kernel/configs/allrandom.config new file +6 -0
new file mode 100644
index 000000000000..67294ef2e3a2
@@ -0,0 +1,6 @@
+# maximize search space, disable options not worth testing
+CONFIG_COMPILE_TEST=y
+
+# reduce compile-time dependencies
+CONFIG_STANDALONE=y
+CONFIG_PREVENT_FIRMWARE_BUILD=y
Signature · 1 line
2.9.0
#16 lore raw

Re: [PATCH 3/3] Kconfig: add a default allrandom.config

2018-02-17 6:41 GMT+09:00 Arnd Bergmann <[email protected]>:

Quoted 3 lines

Building randconfig kernels frequently leads to build errors from drivers that have additional build dependencies, or that we don't want to build for compile-testing for some other reason.

Can you fix the former case?

The latter seems OK.

Quoted 7 lines

We already have a couple of compile-time options that can deal with this problem, but the complete set is not documented well. We also have a method to override some options during 'make randconfig' and a couple of other make targets. However, we don't yet combine those two.

From this statement,

kernel/configs/compile_test.config

would make more sense?

Quoted 7 lines

This adds a new allrandom.config file to turn on CONFIG_COMPILE_TEST, CONFIG_STANDALONE and CONFIG_PREVENT_FIRMWARE_BUILD for all randconfig builds. This in turn disables some other options that we already try to avoid in 'allmodconfig' builds and that make no sense in a general randconfig build.

Building with 'make randconfig KCONFIG_ALLCONFIG=1' should now always

Can you type 'make randconfig compile_test.config', instead?

(Or 'make allmodconfig compile_test.config' for full-build test)

The merge_config has a little bit different logic
from KCONFIG_ALLCONFIG=1, but I guess it leads to the almost same result.

Quoted 28 lines

succeed without warnings on x86 and arm64, aside from recent regressions. Other architectures probably need additional bugfixes.

---
 kernel/configs/allrandom.config | 6 ++++++
 1 file changed, 6 insertions(+)
 create mode 100644 kernel/configs/allrandom.config
kernel/configs/allrandom.config new file +6 -1
new file mode 100644
index 000000000000..67294ef2e3a2
@@ -0,0 +1,6 @@
+# maximize search space, disable options not worth testing
+CONFIG_COMPILE_TEST=y
+
+# reduce compile-time dependencies
+CONFIG_STANDALONE=y
+CONFIG_PREVENT_FIRMWARE_BUILD=y
--

2.9.0

--
To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 2 lines
Best Regards
Masahiro Yamada
#17 lore raw

Re: [PATCH 3/3] Kconfig: add a default allrandom.config

Quoted 6 lines · Masahiro Yamada

2018-02-17 6:41 GMT+09:00 Arnd Bergmann <[email protected]>:

Quoted 3 lines

Building randconfig kernels frequently leads to build errors from drivers that have additional build dependencies, or that we don't want to build for compile-testing for some other reason.

Can you fix the former case?

I'm sure it can be done with runtime detection of the tools, not sure if that's a good idea. What we would need here is

CONFIG_WANXL_BUILD_FIRMWARE: needs to check for as68k and ld68k CONFIG_AIC79XX_BUILD_FIRMWARE: needs flex, bison and berkeley db.
I have all those but it still failed to build for me.
STANDALONE now (after the removal of sound/oss) only controls
ACPI_CUSTOM_DSDT_FILE, we could remove STANDALONE now
I think, it doesn't actually get in the way of anything. For the stable kernels, I'd still need to deal with CONFIG_STANDALONE for the oss drivers.

My feeling right now is that we're better off making both STANDALONE and PREVENT_FIRMWARE_BUILD depend on !COMPILE_TEST,
and then find a way to force that.

Quoted 11 lines
Quoted 4 lines

We already have a couple of compile-time options that can deal with this problem, but the complete set is not documented well. We also have a method to override some options during 'make randconfig' and a couple of other make targets. However, we don't yet combine those two.

From this statement,

kernel/configs/compile_test.config

would make more sense?

My original thought was that the allrandom.config file name is already documented and I just wanted a more consistent behavior, not come up with something new.

Quoted 9 lines
Quoted 7 lines

This adds a new allrandom.config file to turn on CONFIG_COMPILE_TEST, CONFIG_STANDALONE and CONFIG_PREVENT_FIRMWARE_BUILD for all randconfig builds. This in turn disables some other options that we already try to avoid in 'allmodconfig' builds and that make no sense in a general randconfig build.

Building with 'make randconfig KCONFIG_ALLCONFIG=1' should now always

Quoted 7 lines

Can you type 'make randconfig compile_test.config', instead?

(Or 'make allmodconfig compile_test.config' for full-build test)

The merge_config has a little bit different logic
from KCONFIG_ALLCONFIG=1, but I guess it leads to the almost same result.

I suspect that one difference in a randconfig kernel is that all options that depend on CONFIG_COMPILE_TEST have a 50% lower chance of getting enabled that way, but eventually they will get enabled.

Arnd

#18 lore raw

Re: [PATCH 3/3] Kconfig: add a default allrandom.config

2018-02-20 20:11 GMT+09:00 Arnd Bergmann <[email protected]>:

Quoted 33 lines
Quoted 6 lines · Masahiro Yamada

2018-02-17 6:41 GMT+09:00 Arnd Bergmann <[email protected]>:

Quoted 3 lines

Building randconfig kernels frequently leads to build errors from drivers that have additional build dependencies, or that we don't want to build for compile-testing for some other reason.

Can you fix the former case?

I'm sure it can be done with runtime detection of the tools, not sure if that's a good idea. What we would need here is

CONFIG_WANXL_BUILD_FIRMWARE: needs to check for as68k and ld68k CONFIG_AIC79XX_BUILD_FIRMWARE: needs flex, bison and berkeley db.
I have all those but it still failed to build for me.
STANDALONE now (after the removal of sound/oss) only controls
ACPI_CUSTOM_DSDT_FILE, we could remove STANDALONE now
I think, it doesn't actually get in the way of anything. For the stable kernels, I'd still need to deal with CONFIG_STANDALONE for the oss drivers.

My feeling right now is that we're better off making both STANDALONE and PREVENT_FIRMWARE_BUILD depend on !COMPILE_TEST,
and then find a way to force that.

Quoted 8 lines
Quoted 4 lines

We already have a couple of compile-time options that can deal with this problem, but the complete set is not documented well. We also have a method to override some options during 'make randconfig' and a couple of other make targets. However, we don't yet combine those two.

From this statement,

Quoted 7 lines
Quoted 3 lines

kernel/configs/compile_test.config

would make more sense?

My original thought was that the allrandom.config file name is already documented and I just wanted a more consistent behavior, not come up with something new.

I thought this would be used by merge_config.sh, but I understood allrandom.config would be better.

Quoted 14 lines
Quoted 14 lines
Quoted 7 lines

This adds a new allrandom.config file to turn on CONFIG_COMPILE_TEST, CONFIG_STANDALONE and CONFIG_PREVENT_FIRMWARE_BUILD for all randconfig builds. This in turn disables some other options that we already try to avoid in 'allmodconfig' builds and that make no sense in a general randconfig build.

Building with 'make randconfig KCONFIG_ALLCONFIG=1' should now always

Can you type 'make randconfig compile_test.config', instead?

(Or 'make allmodconfig compile_test.config' for full-build test)

Quoted 7 lines
Quoted 2 lines

The merge_config has a little bit different logic
from KCONFIG_ALLCONFIG=1, but I guess it leads to the almost same result.

I suspect that one difference in a randconfig kernel is that all options that depend on CONFIG_COMPILE_TEST have a 50% lower chance of getting enabled that way, but eventually they will get enabled.

You are right.

The behavior is slightly different.

Signature · 2 lines
Best Regards
Masahiro Yamada