kconfig: fix missing choice values in auto.conf
Subsystems (3)
PATCH 1/1 kconfig: fix missing choice values in auto.conf
Since commit 00c864f8903d ("kconfig: allow all config targets to write auto.conf if missing"), Kconfig creates include/config/auto.conf in the defconfig stage when it is missing.
Joonas Kylmälä reported incorrect auto.conf generation under some circumstances.
Apply the following diff:
| --- a/arch/arm/configs/imx_v6_v7_defconfig | +++ b/arch/arm/configs/imx_v6_v7_defconfig | @@ -345,14 +345,7 @@ CONFIG_USB_CONFIGFS_F_MIDI=y | CONFIG_USB_CONFIGFS_F_HID=y | CONFIG_USB_CONFIGFS_F_UVC=y | CONFIG_USB_CONFIGFS_F_PRINTER=y | -CONFIG_USB_ZERO=m | -CONFIG_USB_AUDIO=m | -CONFIG_USB_ETH=m | -CONFIG_USB_G_NCM=m | -CONFIG_USB_GADGETFS=m | -CONFIG_USB_FUNCTIONFS=m | -CONFIG_USB_MASS_STORAGE=m | -CONFIG_USB_G_SERIAL=m | +CONFIG_USB_FUNCTIONFS=y | CONFIG_MMC=y | CONFIG_MMC_SDHCI=y | CONFIG_MMC_SDHCI_PLTFM=y
And then, run:
$ make ARCH=arm mrproper imx_v6_v7_defconfig
CONFIG_USB_FUNCTIONFS=y is correctly contained in the .config, but not in the auto.conf.
Please note drivers/usb/gadget/legacy/Kconfig is included from a choice block in drivers/usb/gadget/Kconfig. So USB_FUNCTIONFS is a choice value.
This is probably a similar situation described in commit beaaddb62540 ("kconfig: tests: test defconfig when two choices interact").
When sym_calc_choice() is called, the choice symbol forgets the SYMBOL_DEF_USER unless all of its choice values are explicitly set by the user.
The choice symbol is given just one chance to recall it because set_all_choice_values() is called if SYMBOL_NEED_SET_CHOICE_VALUES is set.
When sym_calc_choice() is called again, the choice symbol forgets it forever, since SYMBOL_NEED_SET_CHOICE_VALUES is a one-time aid. Hence, we cannot call sym_clear_all_valid() again and again.
It is crazy to set and clear internal flags. However, we cannot simply get rid of "sym->flags &= flags | ~SYMBOL_DEF_USER;" Doing so would re-introduce the problem solved by commit 5d09598d488f ("kconfig: fix new choices being skipped upon config update").
To work around the issue, conf_write_autoconf() stopped calling sym_clear_all_valid().
conf_write() must be changed accordingly. Currently, it clears SYMBOL_WRITE after the symbol is written into the .config file. This is needed to prevent it from writing the same symbol multiple times in case the symbol is declared in two or more locations. I added the new flag SYMBOL_WRITTEN, to track the symbols that have been written.
Anyway, this is a cheesy workaround in order to suppress the issue as far as defconfig is concerned.
Handling of choices is totally broken. sym_clear_all_valid() is called every time a user touches a symbol from the GUI interface. To reproduce it, just add a new symbol drivers/usb/gadget/legacy/Kconfig, then touch around unrelated symbols from menuconfig. USB_FUNCTIONFS will disappear from the .config file.
I added the Fixes tag since it is more fatal than before. But, this has been broken since long long time before, and still it is. We should take a closer look to fix this correctly somehow.
---
scripts/kconfig/confdata.c | 7 +++---- scripts/kconfig/expr.h | 1 + 2 files changed, 4 insertions(+), 4 deletions(-)
scripts/kconfig/confdata.c +3 -4
@@ -895,7 +895,8 @@ int conf_write(const char *name)"# %s\n" "#\n", str); need_newline = false; - }else if (!(sym->flags &SYMBOL_CHOICE )) { + }else if (!(sym->flags &SYMBOL_CHOICE ) && + !(sym->flags &SYMBOL_WRITTEN )) {sym_calc_value (sym);if (!(sym->flags &SYMBOL_WRITE )) goto next;
@@ -903,7 +904,7 @@ int conf_write(const char *name)fprintf (out,"\n" ); need_newline = false; } - sym->flags &= ~SYMBOL_WRITE ; + sym->flags |=SYMBOL_WRITTEN ;conf_write_symbol (out, sym, &kconfig_printer_cb,NULL ); }
@@ -1063,8 +1064,6 @@ int conf_write_autoconf(int overwrite)if (!overwrite &&is_present (autoconf_name))return 0 ; -sym_clear_all_valid (); -conf_write_dep ("include/config/auto.conf.cmd" );if (conf_touch_deps ())
scripts/kconfig/expr.h +1 -0
@@ -141,6 +141,7 @@ struct symbol {#define SYMBOL_OPTIONAL 0x0100/* choice is optional - values can be 'n' */ #define SYMBOL_WRITE 0x0200/* write symbol to file (KCONFIG_CONFIG) */ #define SYMBOL_CHANGED 0x0400/* ? */ +#define SYMBOL_WRITTEN 0x0800/* track info to avoid double-write to .config */ #define SYMBOL_NO_WRITE 0x1000/* Symbol for internal use only; it will not be written */ #define SYMBOL_CHECKED 0x2000/* used during dependency checking */ #define SYMBOL_WARNED 0x8000/* warning has been issued */
Signature · 1 line
2.17.1
Hi,
thanks a lot for this patch! I tested this and it fixes what is says in the commit message:
Masahiro Yamada:
Quoted 135 lines
Since commit
00c864f8903d("kconfig: allow all config targets to write auto.conf if missing"),Kconfigcreatesinclude/config/auto.confin the defconfig stage when it is missing.Joonas Kylmälä reported incorrect auto.conf generation under some circumstances.
Apply the following diff:
| --- a/arch/arm/configs/imx_v6_v7_defconfig | +++ b/arch/arm/configs/imx_v6_v7_defconfig | @@ -345,14 +345,7 @@ CONFIG_USB_CONFIGFS_F_MIDI=y | CONFIG_USB_CONFIGFS_F_HID=y | CONFIG_USB_CONFIGFS_F_UVC=y | CONFIG_USB_CONFIGFS_F_PRINTER=y | -CONFIG_USB_ZERO=m | -CONFIG_USB_AUDIO=m | -CONFIG_USB_ETH=m | -CONFIG_USB_G_NCM=m | -CONFIG_USB_GADGETFS=m | -CONFIG_USB_FUNCTIONFS=m | -CONFIG_USB_MASS_STORAGE=m | -CONFIG_USB_G_SERIAL=m | +CONFIG_USB_FUNCTIONFS=y | CONFIG_MMC=y | CONFIG_MMC_SDHCI=y | CONFIG_MMC_SDHCI_PLTFM=yAnd then, run:
$ make ARCH=arm mrproper imx_v6_v7_defconfig
CONFIG_USB_FUNCTIONFS=yis correctly contained in the .config, but not in the auto.conf.Please note
drivers/usb/gadget/legacy/Kconfigis included from a choice block indrivers/usb/gadget/Kconfig. SoUSB_FUNCTIONFSis a choice value.This is probably a similar situation described in commit
beaaddb62540("kconfig: tests: test defconfig when two choices interact").When
sym_calc_choice()is called, the choice symbol forgets theSYMBOL_DEF_USERunless all of its choice values are explicitly set by the user.The choice symbol is given just one chance to recall it because
set_all_choice_values()is called ifSYMBOL_NEED_SET_CHOICE_VALUESis set.When
sym_calc_choice()is called again, the choice symbol forgets it forever, sinceSYMBOL_NEED_SET_CHOICE_VALUESis a one-time aid. Hence, we cannot callsym_clear_all_valid()again and again.It is crazy to set and clear internal flags. However, we cannot simply get rid of "
sym->flags&= flags | ~SYMBOL_DEF_USER;" Doing so would re-introduce the problem solved by commit5d09598d488f("kconfig: fix new choices being skipped upon config update").To work around the issue,
conf_write_autoconf()stopped callingsym_clear_all_valid().
conf_write()must be changed accordingly. Currently, it clearsSYMBOL_WRITEafter the symbol is written into the .config file. This is needed to prevent it from writing the same symbol multiple times in case the symbol is declared in two or more locations. I added the new flagSYMBOL_WRITTEN, to track the symbols that have been written.Anyway, this is a cheesy workaround in order to suppress the issue as far as defconfig is concerned.
Handling of choices is totally broken.
sym_clear_all_valid()is called every time a user touches a symbol from the GUI interface. To reproduce it, just add a new symboldrivers/usb/gadget/legacy/Kconfig, then touch around unrelated symbols from menuconfig.USB_FUNCTIONFSwill disappear from the .config file.I added the Fixes tag since it is more fatal than before. But, this has been broken since long long time before, and still it is. We should take a closer look to fix this correctly somehow.
Fixes 00c864f8903d ("kconfig: allow all config targets to write auto.conf if missing")Cc linux-stable <[email protected]> # 4.19+Reported-by Joonas Kylmälä <[email protected]>Signed-off-by Masahiro Yamada <[email protected]>---
scripts/kconfig/confdata.c | 7 +++---- scripts/kconfig/expr.h | 1 + 2 files changed, 4 insertions(+), 4 deletions(-)scripts/kconfig/confdata.c +3
-4@@ -895,7 +895,8 @@ int conf_write(const char *name)"# %s\n" "#\n", str); need_newline = false; - }else if (!(sym->flags &SYMBOL_CHOICE )) { + }else if (!(sym->flags &SYMBOL_CHOICE ) && + !(sym->flags &SYMBOL_WRITTEN )) {sym_calc_value (sym);if (!(sym->flags &SYMBOL_WRITE )) goto next;@@ -903,7 +904,7 @@ int conf_write(const char *name)fprintf (out,"\n" ); need_newline = false; } - sym->flags &= ~SYMBOL_WRITE ; + sym->flags |=SYMBOL_WRITTEN ;conf_write_symbol (out, sym, &kconfig_printer_cb,NULL ); }@@ -1063,8 +1064,6 @@ int conf_write_autoconf(int overwrite)if (!overwrite &&is_present (autoconf_name))return 0 ; -sym_clear_all_valid (); -conf_write_dep ("include/config/auto.conf.cmd" );if (conf_touch_deps ())scripts/kconfig/expr.h +1
-0@@ -141,6 +141,7 @@ struct symbol {#define SYMBOL_OPTIONAL 0x0100/* choice is optional - values can be 'n' */ #define SYMBOL_WRITE 0x0200/* write symbol to file (KCONFIG_CONFIG) */ #define SYMBOL_CHANGED 0x0400/* ? */ +#define SYMBOL_WRITTEN 0x0800/* track info to avoid double-write to .config */ #define SYMBOL_NO_WRITE 0x1000/* Symbol for internal use only; it will not be written */ #define SYMBOL_CHECKED 0x2000/* used during dependency checking */ #define SYMBOL_WARNED 0x8000/* warning has been issued */
Hello,
Quoted 5 lines
conf_write()must be changed accordingly. Currently, it clearsSYMBOL_WRITEafter the symbol is written into the .config file. This is needed to prevent it from writing the same symbol multiple times in case the symbol is declared in two or more locations. I added the new flagSYMBOL_WRITTEN, to track the symbols that have been written.
[snip]
Quoted 4 lines
scripts/kconfig/confdata.c +0
-0
[snip]
scripts/kconfig/confdata.c int conf_write(const char *name) · line 907
Quoted 6 lines
scripts/kconfig/confdata.c +1
-1@@ -903,7 +904,7 @@ int conf_write(const char *name)fprintf (out,"\n" ); need_newline = false; } - sym->flags &= ~SYMBOL_WRITE ; + sym->flags |=SYMBOL_WRITTEN ;
The SYMBOL_WRITTEN flag is never cleared after being set in this function, which unfortunately causes data loss to occur when a user starts xconfig, gconfig, or nconfig and saves a config file more than once. Every save operation after the first one causes the saved .config file to contain only comments.
I am appending a patch that resolves this issue. The patch is a bit ugly because of the code duplication, but it fixes this bug. (I have lightly tested the patch.) Even if the patch is not merged, I would appreciate it if this bug could be fixed.
Thank you,
Vefa
=== 8< === Patch Follows === >8 ===
From: "M. Vefa Bicakci" <[email protected]>
Date: Fri, 2 Aug 2019 17:44:40 -0400
Subject: [PATCH] kconfig: Clear "written" flag to avoid data loss
Prior to this commit, starting nconfig, xconfig or gconfig, and saving the .config file more than once caused data loss, where a .config file that contained only comments would be written to disk starting from the second save operation.
This bug manifests itself because the SYMBOL_WRITTEN flag is never cleared after the first call to conf_write, and subsequent calls to conf_write then skip all of the configuration symbols due to the SYMBOL_WRITTEN flag being set.
This commit resolves this issue by clearing the SYMBOL_WRITTEN flag from all symbols before conf_write returns.
--- scripts/kconfig/confdata.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)
scripts/kconfig/confdata.c +31 -0
@@ -840,6 +840,35 @@ int conf_write_defconfig(const char *filename)return 0 ; } +static void conf_clear_written_flag (void ) +{ +struct menu *menu; +struct symbol *sym; + + menu = rootmenu.list; +while (menu) { + sym = menu->sym; +if (sym && (sym->flags &SYMBOL_WRITTEN )) + sym->flags &= ~SYMBOL_WRITTEN ; + +if (menu->list) { + menu = menu->list; +continue ; + } + +if (menu->next) { + menu = menu->next; + }else { +while ((menu = menu->parent)) { +if (menu->next) { + menu = menu->next; +break ; + } + } + } + } +} +int conf_write (const char *name) {FILE *out;
@@ -930,6 +959,8 @@ int conf_write(const char *name) }fclose (out); +conf_clear_written_flag (); +if (*tmpname) {if (is_same (name, tmpname)) {conf_message ("No change to %s" , name);
Signature · 1 line
2.21.0
Quoted 27 lines · M. Vefa Bicakci
Hello,
Quoted 5 lines
conf_write()must be changed accordingly. Currently, it clearsSYMBOL_WRITEafter the symbol is written into the .config file. This is needed to prevent it from writing the same symbol multiple times in case the symbol is declared in two or more locations. I added the new flagSYMBOL_WRITTEN, to track the symbols that have been written.[snip]
Quoted 4 lines
scripts/kconfig/confdata.c +0
-0[snip]
scripts/kconfig/confdata.c int conf_write(const char *name) · line 907
Quoted 6 lines
scripts/kconfig/confdata.c +1
-1@@ -903,7 +904,7 @@ int conf_write(const char *name)fprintf (out,"\n" ); need_newline = false; } - sym->flags &= ~SYMBOL_WRITE ; + sym->flags |=SYMBOL_WRITTEN ;The
SYMBOL_WRITTENflag is never cleared after being set in this function, which unfortunately causes data loss to occur when a user starts xconfig, gconfig, or nconfig and saves a config file more than once. Every save operation after the first one causes the saved .config file to contain only comments.
Quoted 6 lines
I am appending a patch that resolves this issue. The patch is a bit ugly because of the code duplication, but it fixes this bug. (I have lightly tested the patch.) Even if the patch is not merged, I would appreciate it if this bug could be fixed.
Thank you,
When you clear a flag, the tree traverse order is not a big deal. So, you can use for_all_symbols() iterator.
Could you send v2 with the following code ?
for_all_symbols(i, sym)
sym->flags &= ~SYMBOL_WRITTEN;This is quite simple, so you do not need to create a new helper function for this.
Thanks.
Quoted 81 lines
Vefa
=== 8< === Patch Follows === >8 ===
From: "M. Vefa Bicakci" <[email protected]>
Date: Fri, 2 Aug 2019 17:44:40 -0400
Subject: [PATCH] kconfig: Clear "written" flag to avoid data lossPrior to this commit, starting nconfig, xconfig or gconfig, and saving the .config file more than once caused data loss, where a .config file that contained only comments would be written to disk starting from the second save operation.
This bug manifests itself because the
SYMBOL_WRITTENflag is never cleared after the first call toconf_write, and subsequent calls toconf_writethen skip all of the configuration symbols due to theSYMBOL_WRITTENflag being set.This commit resolves this issue by clearing the
SYMBOL_WRITTENflag from all symbols beforeconf_writereturns.Fixes 8e2442a5f86e ("kconfig: fix missing choice values in auto.conf")Cc linux-stable <[email protected]> # 4.19+Signed-off-by M. Vefa Bicakci <[email protected]>--- scripts/kconfig/confdata.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)scripts/kconfig/confdata.c +31
-1@@ -840,6 +840,35 @@ int conf_write_defconfig(const char *filename)return 0 ; } +static void conf_clear_written_flag (void ) +{ +struct menu *menu; +struct symbol *sym; + + menu = rootmenu.list; +while (menu) { + sym = menu->sym; +if (sym && (sym->flags &SYMBOL_WRITTEN )) + sym->flags &= ~SYMBOL_WRITTEN ; + +if (menu->list) { + menu = menu->list; +continue ; + } + +if (menu->next) { + menu = menu->next; + }else { +while ((menu = menu->parent)) { +if (menu->next) { + menu = menu->next; +break ; + } + } + } + } +} +int conf_write (const char *name) {FILE *out;@@ -930,6 +959,8 @@ int conf_write(const char *name) }fclose (out); +conf_clear_written_flag (); +if (*tmpname) {if (is_same (name, tmpname)) {conf_message ("No change to %s" , name); --2.21.0
Signature · 2 lines
Best Regards Masahiro Yamada
PATCH v2 1/1 kconfig: Clear "written" flag to avoid data loss
Prior to this commit, starting nconfig, xconfig or gconfig, and saving the .config file more than once caused data loss, where a .config file that contained only comments would be written to disk starting from the second save operation.
This bug manifests itself because the SYMBOL_WRITTEN flag is never cleared after the first call to conf_write, and subsequent calls to conf_write then skip all of the configuration symbols due to the SYMBOL_WRITTEN flag being set.
This commit resolves this issue by clearing the SYMBOL_WRITTEN flag from all symbols before conf_write returns.
---
Changes since v1:
* As suggested by Masahiro Yamada, instead of defining a new helper function to traverse over all symbols in a pre-defined order, use the for_all_symbols iterator.
--- scripts/kconfig/confdata.c | 4 ++++ 1 file changed, 4 insertions(+)
scripts/kconfig/confdata.c +4 -0
@@ -848,6 +848,7 @@ int conf_write(const char *name)const char *str;char tmpname[PATH_MAX +1 ], oldname[PATH_MAX +1 ];char *env; +int i;bool need_newline = false;if (!name)
@@ -930,6 +931,9 @@ int conf_write(const char *name) }fclose (out); +for_all_symbols (i, sym) + sym->flags &= ~SYMBOL_WRITTEN; +if (*tmpname) {if (is_same (name, tmpname)) {conf_message ("No change to %s" , name);
Signature · 1 line
2.21.0
Sasha Levin
maintainerRe: [PATCH v2] kconfig: Clear "written" flag to avoid data loss
Hi,
[This is an automated email]
This commit has been processed because it contains a "Fixes:" tag, fixing commit: 8e2442a5f86e kconfig: fix missing choice values in auto.conf.
The bot has tested the following trees: v5.2.5, v4.19.63.
v5.2.5: Build OK!
v4.19.63: Failed to apply! Possible dependencies:
aff11cd983ec ("kconfig: Terminate menu blocks with a comment in the generated config")
NOTE: The patch will not be queued to stable trees until it is upstream.
How should we proceed with this patch?
--
Thanks,
Sasha
Re: [PATCH v2] kconfig: Clear "written" flag to avoid data loss
Quoted 3 lines · On Sat, 03 Aug 2019 13:22:11 +0000 Sasha Levin <[email protected]> wrote:
Hi,
[This is an automated email]
Hello,
Sorry about the late reply.
Quoted 12 lines
This commit has been processed because it contains a "Fixes:" tag, fixing commit:
8e2442a5f86ekconfig: fix missing choice values in auto.conf.The bot has tested the following trees: v5.2.5, v4.19.63.
v5.2.5: Build OK!
v4.19.63: Failed to apply! Possible dependencies:
aff11cd983ec("kconfig: Terminate menu blocks with a comment in
the generated config")
Quoted 4 lines
NOTE: The patch will not be queued to stable trees until it is upstream.
How should we proceed with this patch?
The merge conflict is a very minor one involving the difference between the following two code blocks:
v4.19.y
const char *str; char dirname[PATH_MAX+1], tmpname[PATH_MAX+22], newname[PATH_MAX+8]; char *env;
next/master (and v5.2.y)
const char *str; char tmpname[PATH_MAX + 1], oldname[PATH_MAX + 1]; char *env; bool need_newline = false;
This patch inserts the variable "int i;" between "char *env;" and "bool need_newline = false;", but this does not work due to v4.19.y not having the same context.
Quoted 3 lines
--
Thanks,
Sasha
Thank you,
Vefa
RE: [PATCH v2] kconfig: Clear "written" flag to avoid data loss
In reply to Sasha LevinQuoted 26 lines
-----Original Message-----
From: Sasha Levin [mailto:[email protected]]
Sent: Saturday, August 03, 2019 10:22 PM
To: Sasha Levin <[email protected]>; M. Vefa Bicakci <[email protected]>; Yamada, Masahiro/山田 真弘 <[email protected]>
Cc: [email protected]
Subject: Re: [PATCH v2] kconfig: Clear "written" flag to avoid data lossHi,
[This is an automated email]
This commit has been processed because it contains a "Fixes:" tag, fixing commit:
8e2442a5f86ekconfig: fix missing choice values in auto.conf.The bot has tested the following trees: v5.2.5, v4.19.63.
v5.2.5: Build OK!
v4.19.63: Failed to apply! Possible dependencies:
aff11cd983ec("kconfig: Terminate menu blocks with a comment in the
generated config")
Quoted 3 lines
NOTE: The patch will not be queued to stable trees until it is upstream.
How should we proceed with this patch?
It has landed in Linus' tree now.
Commit 0c5b6c28ed68becb692b43eae5e44d5aa7e160c upstream
I will resolve the conflict for 4.19.x later.
Thanks.
RE: [PATCH v2] kconfig: Clear "written" flag to avoid data loss
In reply to Sasha LevinHi Sasha,
Quoted 24 lines
-----Original Message-----
From: Sasha Levin [mailto:[email protected]]
Sent: Saturday, August 03, 2019 10:22 PM
To: Sasha Levin <[email protected]>; M. Vefa Bicakci <[email protected]>; Yamada, Masahiro/山田 真弘 <[email protected]>
Cc: [email protected]
Subject: Re: [PATCH v2] kconfig: Clear "written" flag to avoid data lossHi,
[This is an automated email]
This commit has been processed because it contains a "Fixes:" tag, fixing commit:
8e2442a5f86ekconfig: fix missing choice values in auto.conf.The bot has tested the following trees: v5.2.5, v4.19.63.
v5.2.5: Build OK!
v4.19.63: Failed to apply! Possible dependencies:
aff11cd983ec("kconfig: Terminate menu blocks with a comment in the
generated config")
Quoted 3 lines
NOTE: The patch will not be queued to stable trees until it is upstream.
How should we proceed with this patch?
For 4.19.x, I just sent a cleanly-applicable patch.
Thanks.
Re: [PATCH v2] kconfig: Clear "written" flag to avoid data loss
In reply to M. Vefa BicakciQuoted 11 lines · M. Vefa Bicakci
Prior to this commit, starting nconfig, xconfig or gconfig, and saving the .config file more than once caused data loss, where a .config file that contained only comments would be written to disk starting from the second save operation.
This bug manifests itself because the
SYMBOL_WRITTENflag is never cleared after the first call toconf_write, and subsequent calls toconf_writethen skip all of the configuration symbols due to theSYMBOL_WRITTENflag being set.
Quoted 6 lines
This commit resolves this issue by clearing the
SYMBOL_WRITTENflag from all symbols beforeconf_writereturns.Fixes 8e2442a5f86e ("kconfig: fix missing choice values in auto.conf")Cc linux-stable <[email protected]> # 4.19+Signed-off-by M. Vefa Bicakci <[email protected]>
Applied to linux-kbuild/fixes.
Thanks.
Signature · 2 lines
Best Regards Masahiro Yamada