Skip to contentSkip to conversation

tools: moveconfig: some more improvements

PATCH New Masahiro Yamada 4 participants 15 messages Aug 22, 2016 – Sep 7, 2016

lore
Share & tools
lore
#0 lore raw

cover tools: moveconfig: some more improvements

Masahiro Yamada (4):
tools: moveconfig: remove document about deprecated error message
tools: moveconfig: use sets instead of lists for failed/suspicious
boards
tools: moveconfig: warn loudly if moved option has no entry in Kconfig
tools: moveconfig: add --spl option to move options for SPL build

 tools/moveconfig.py | 102 ++++++++++++++++++++++++++++++++++++----------------
 1 file changed, 71 insertions(+), 31 deletions(-)
Signature · 1 line
1.9.1
#1 lore raw

PATCH 1/4 tools: moveconfig: remove document about deprecated error message

Since commit cc008299f852 ("tools: moveconfig: do not rely on type and default value given by users"), we do not have this error case.

---

 tools/moveconfig.py | 4 ----
 1 file changed, 4 deletions(-)
tools/moveconfig.py +0 -4
index 5283689..c188235 100755
@@ -58,10 +58,6 @@ It looks like one of the following:
    The define in the config header matched the one in Kconfig.
    We do not need to touch it.
 
- - Undefined.  Do nothing.
-   This config option was not found in the config header.
-   Nothing to do.
-
  - Compiler is missing.  Do nothing.
    The compiler specified for this architecture was not found
    in your PATH environment.
Signature · 1 line
1.9.1
#3 lore raw

PATCH 1/4 tools: moveconfig: remove document about deprecated error message

In reply to Masahiro Yamada
Quoted 5 lines · Masahiro Yamada

Since commit cc008299f852 ("tools: moveconfig: do not rely on type and default value given by users"), we do not have this error case.

Applied to u-boot/master, thanks!

Signature · 8 lines
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20160907/ac943f82/attachment.sig>
#4 lore raw

PATCH 2/4 tools: moveconfig: use sets instead of lists for failed/suspicious boards

In reply to Masahiro Yamada

The sets feature is handier for adding unique elements.

---

 tools/moveconfig.py | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)
tools/moveconfig.py +10 -10
index c188235..93f3781 100755
@@ -798,8 +798,8 @@ class Slot:
         self.reference_src_dir = reference_src_dir
         self.parser = KconfigParser(configs, options, self.build_dir)
         self.state = STATE_IDLE
-        self.failed_boards = []
-        self.suspicious_boards = []
+        self.failed_boards = set()
+        self.suspicious_boards = set()
 
     def __del__(self):
         """Delete the working directory
@@ -946,7 +946,7 @@ class Slot:
 
         log = self.parser.check_defconfig()
         if log:
-            self.suspicious_boards.append(self.defconfig)
+            self.suspicious_boards.add(self.defconfig)
             self.log += log
         orig_defconfig = os.path.join('configs', self.defconfig)
         new_defconfig = os.path.join(self.build_dir, 'defconfig')
@@ -980,19 +980,19 @@ class Slot:
                 sys.exit("Exit on error.")
             # If --exit-on-error flag is not set, skip this board and continue.
             # Record the failed board.
-            self.failed_boards.append(self.defconfig)
+            self.failed_boards.add(self.defconfig)
 
         self.progress.inc()
         self.progress.show()
         self.state = STATE_IDLE
 
     def get_failed_boards(self):
-        """Returns a list of failed boards (defconfigs) in this slot.
+        """Returns a set of failed boards (defconfigs) in this slot.
         """
         return self.failed_boards
 
     def get_suspicious_boards(self):
-        """Returns a list of boards (defconfigs) with possible misconversion.
+        """Returns a set of boards (defconfigs) with possible misconversion.
         """
         return self.suspicious_boards
 
@@ -1057,11 +1057,11 @@ class Slots:
 
     def show_failed_boards(self):
         """Display all of the failed boards (defconfigs)."""
-        boards = []
+        boards = set()
         output_file = 'moveconfig.failed'
 
         for slot in self.slots:
-            boards += slot.get_failed_boards()
+            boards |= slot.get_failed_boards()
 
         if boards:
             boards = '\n'.join(boards) + '\n'
@@ -1076,11 +1076,11 @@ class Slots:
 
     def show_suspicious_boards(self):
         """Display all boards (defconfigs) with possible misconversion."""
-        boards = []
+        boards = set()
         output_file = 'moveconfig.suspicious'
 
         for slot in self.slots:
-            boards += slot.get_suspicious_boards()
+            boards |= slot.get_suspicious_boards()
 
         if boards:
             boards = '\n'.join(boards) + '\n'
Signature · 1 line
1.9.1
#6 lore raw

PATCH 2/4 tools: moveconfig: use sets instead of lists for failed/suspicious boards

In reply to Masahiro Yamada
Quoted 4 lines · Masahiro Yamada

The sets feature is handier for adding unique elements.

Applied to u-boot/master, thanks!

Signature · 8 lines
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20160907/3237ec46/attachment.sig>
#7 lore raw

PATCH 3/4 tools: moveconfig: warn loudly if moved option has no entry in Kconfig

In reply to Masahiro Yamada

Currently, the tool gives up moving an option quietly if its entry was not found in Kconfig.

If the option is not defined in the config header in the first place, it is no problem (as the Kconfig entry may have been hidden by reasonable "depends on").

However, if the option is defined in the config header, the missing Kconfig entry is a sign of possible behavior change. It is highly recommended to manually check if the option has been moved as expected. In this case, let's add "suspicious" in the log and change the log color (if --color option is given) to make it stand out.

This was suggested by Tom in [1].

[1] http://lists.denx.de/pipermail/u-boot/2016-July/261988.html

---

 tools/moveconfig.py | 45 +++++++++++++++++++++++++++++++--------------
 1 file changed, 31 insertions(+), 14 deletions(-)
tools/moveconfig.py +31 -14
index 93f3781..98e8608 100755
@@ -47,12 +47,18 @@ It looks like one of the following:
    This config option was moved to the defconfig
 
  - CONFIG_... is not defined in Kconfig.  Do nothing.
-   The entry for this CONFIG was not found in Kconfig.
+   The entry for this CONFIG was not found in Kconfig.  The option is not
+   defined in the config header, either.  So, this case can be just skipped.
+
+ - CONFIG_... is not defined in Kconfig (suspicious).  Do nothing.
+   This option is defined in the config header, but its entry was not found
+   in Kconfig.
    There are two common cases:
      - You forgot to create an entry for the CONFIG before running
        this tool, or made a typo in a CONFIG passed to this tool.
      - The entry was hidden due to unmet 'depends on'.
-       This is correct behavior.
+   The tool does not know if the result is reasonable, so please check it
+   manually.
 
  - 'CONFIG_...' is the same as the define in Kconfig.  Do nothing.
    The define in the config header matched the one in Kconfig.
@@ -210,7 +216,8 @@ STATE_SAVEDEFCONFIG = 3
 
 ACTION_MOVE = 0
 ACTION_NO_ENTRY = 1
-ACTION_NO_CHANGE = 2
+ACTION_NO_ENTRY_WARN = 2
+ACTION_NO_CHANGE = 3
 
 COLOR_BLACK        = '0;30'
 COLOR_RED          = '0;31'
@@ -659,14 +666,6 @@ class KconfigParser:
         """
         not_set = '# %s is not set' % config
 
-        for line in dotconfig_lines:
-            line = line.rstrip()
-            if line.startswith(config + '=') or line == not_set:
-                old_val = line
-                break
-        else:
-            return (ACTION_NO_ENTRY, config)
-
         for line in autoconf_lines:
             line = line.rstrip()
             if line.startswith(config + '='):
@@ -675,6 +674,17 @@ class KconfigParser:
         else:
             new_val = not_set
 
+        for line in dotconfig_lines:
+            line = line.rstrip()
+            if line.startswith(config + '=') or line == not_set:
+                old_val = line
+                break
+        else:
+            if new_val == not_set:
+                return (ACTION_NO_ENTRY, config)
+            else:
+                return (ACTION_NO_ENTRY_WARN, config)
+
         # If this CONFIG is neither bool nor trisate
         if old_val[-2:] != '=y' and old_val[-2:] != '=m' and old_val != not_set:
             # tools/scripts/define2mk.sed changes '1' to 'y'.
@@ -704,6 +714,7 @@ class KconfigParser:
 
         results = []
         updated = False
+        suspicious = False
 
         with open(self.dotconfig) as f:
             dotconfig_lines = f.readlines()
@@ -725,6 +736,10 @@ class KconfigParser:
             elif action == ACTION_NO_ENTRY:
                 actlog = "%s is not defined in Kconfig.  Do nothing." % value
                 log_color = COLOR_LIGHT_BLUE
+            elif action == ACTION_NO_ENTRY_WARN:
+                actlog = "%s is not defined in Kconfig (suspicious).  Do nothing." % value
+                log_color = COLOR_YELLOW
+                suspicious = True
             elif action == ACTION_NO_CHANGE:
                 actlog = "'%s' is the same as the define in Kconfig.  Do nothing." \
                          % value
@@ -744,7 +759,7 @@ class KconfigParser:
         os.remove(self.config_autoconf)
         os.remove(self.autoconf)
 
-        return (updated, log)
+        return (updated, suspicious, log)
 
     def check_defconfig(self):
         """Check the defconfig after savedefconfig
@@ -923,7 +938,9 @@ class Slot:
     def do_savedefconfig(self):
         """Update the .config and run 'make savedefconfig'."""
 
-        (updated, log) = self.parser.update_dotconfig()
+        (updated, suspicious, log) = self.parser.update_dotconfig()
+        if suspicious:
+            self.suspicious_boards.add(self.defconfig)
         self.log += log
 
         if not self.options.force_sync and not updated:
@@ -994,7 +1011,7 @@ class Slot:
     def get_suspicious_boards(self):
         """Returns a set of boards (defconfigs) with possible misconversion.
         """
-        return self.suspicious_boards
+        return self.suspicious_boards - self.failed_boards
 
 class Slots:
 
Signature · 1 line
1.9.1
#8 lore raw

PATCH 3/4 tools: moveconfig: warn loudly if moved option has no entry in Kconfig

Quoted 14 lines · Masahiro Yamada

Currently, the tool gives up moving an option quietly if its entry was not found in Kconfig.

If the option is not defined in the config header in the first place, it is no problem (as the Kconfig entry may have been hidden by reasonable "depends on").

However, if the option is defined in the config header, the missing Kconfig entry is a sign of possible behavior change. It is highly recommended to manually check if the option has been moved as expected. In this case, let's add "suspicious" in the log and change the log color (if --color option is given) to make it stand out.

Quoted 6 lines

This was suggested by Tom in [1].

[1] http://lists.denx.de/pipermail/u-boot/2016-July/261988.html

Thanks!

Signature · 8 lines
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20160822/c9267d6e/attachment.sig>
#9 lore raw

PATCH 3/4 tools: moveconfig: warn loudly if moved option has no entry in Kconfig

In reply to Masahiro Yamada
Quoted 14 lines · Masahiro Yamada

Currently, the tool gives up moving an option quietly if its entry was not found in Kconfig.

If the option is not defined in the config header in the first place, it is no problem (as the Kconfig entry may have been hidden by reasonable "depends on").

However, if the option is defined in the config header, the missing Kconfig entry is a sign of possible behavior change. It is highly recommended to manually check if the option has been moved as expected. In this case, let's add "suspicious" in the log and change the log color (if --color option is given) to make it stand out.

Quoted 7 lines

This was suggested by Tom in [1].

[1] http://lists.denx.de/pipermail/u-boot/2016-July/261988.html

Applied to u-boot/master, thanks!

Signature · 8 lines
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20160907/b1200012/attachment.sig>
#10 lore raw

PATCH 4/4 tools: moveconfig: add --spl option to move options for SPL build

In reply to Masahiro Yamada

Prior to this commit, the tool could not move options guarded by CONFIG_SPL_BUILD ifdef conditionals because they do not show up in include/autoconf.mk. This new option, if given, makes the tool parse spl/include/autoconf.mk instead of include/autoconf.mk, which is probably preferred behavior when moving options for SPL.

---

 tools/moveconfig.py | 33 ++++++++++++++++++++++++++++++---
 1 file changed, 30 insertions(+), 3 deletions(-)
tools/moveconfig.py +30 -3
index 98e8608..5576b57 100755
@@ -138,6 +138,12 @@ Available options
    If not specified, "make savedefconfig" only occurs for cases
    where at least one CONFIG was moved.
 
+ -S, --spl
+   Look for moved config options in spl/include/autoconf.mk instead of
+   include/autoconf.mk.  This is useful for moving options for SPL build
+   because SPL related options (mostly prefixed with CONFIG_SPL_) are
+   sometimes blocked by CONFIG_SPL_BUILD ifdef conditionals.
+
  -H, --headers-only
    Only cleanup the headers; skip the defconfig processing
 
@@ -614,6 +620,8 @@ class KconfigParser:
         self.options = options
         self.dotconfig = os.path.join(build_dir, '.config')
         self.autoconf = os.path.join(build_dir, 'include', 'autoconf.mk')
+        self.spl_autoconf = os.path.join(build_dir, 'spl', 'include',
+                                         'autoconf.mk')
         self.config_autoconf = os.path.join(build_dir, 'include', 'config',
                                             'auto.conf')
         self.defconfig = os.path.join(build_dir, 'defconfig')
@@ -715,11 +723,25 @@ class KconfigParser:
         results = []
         updated = False
         suspicious = False
+        rm_files = [self.config_autoconf, self.autoconf]
+
+        if self.options.spl:
+            if os.path.exists(self.spl_autoconf):
+                autoconf_path = self.spl_autoconf
+                rm_files.append(self.spl_autoconf)
+            else:
+                for f in rm_files:
+                    os.remove(f)
+                return (updated, suspicious,
+                        color_text(self.options.color, COLOR_BROWN,
+                                   "SPL is not enabled.  Skipped.") + '\n')
+        else:
+            autoconf_path = self.autoconf
 
         with open(self.dotconfig) as f:
             dotconfig_lines = f.readlines()
 
-        with open(self.autoconf) as f:
+        with open(autoconf_path) as f:
             autoconf_lines = f.readlines()
 
         for config in self.configs:
@@ -744,6 +766,9 @@ class KconfigParser:
                 actlog = "'%s' is the same as the define in Kconfig.  Do nothing." \
                          % value
                 log_color = COLOR_LIGHT_PURPLE
+            elif action == ACTION_SPL_NOT_EXIST:
+                actlog = "SPL is not enabled for this defconfig.  Skip."
+                log_color = COLOR_PURPLE
             else:
                 sys.exit("Internal Error. This should not happen.")
 
@@ -756,8 +781,8 @@ class KconfigParser:
                     updated = True
 
         self.results = results
-        os.remove(self.config_autoconf)
-        os.remove(self.autoconf)
+        for f in rm_files:
+            os.remove(f)
 
         return (updated, suspicious, log)
 
@@ -1217,6 +1242,8 @@ def main():
                       help='exit immediately on any error')
     parser.add_option('-s', '--force-sync', action='store_true', default=False,
                       help='force sync by savedefconfig')
+    parser.add_option('-S', '--spl', action='store_true', default=False,
+                      help='parse config options defined for SPL build')
     parser.add_option('-H', '--headers-only', dest='cleanup_headers_only',
                       action='store_true', default=False,
                       help='only cleanup the headers')
Signature · 1 line
1.9.1
#11 lore raw

PATCH 4/4 tools: moveconfig: add --spl option to move options for SPL build

Quoted 7 lines · Masahiro Yamada

Prior to this commit, the tool could not move options guarded by CONFIG_SPL_BUILD ifdef conditionals because they do not show up in include/autoconf.mk. This new option, if given, makes the tool parse spl/include/autoconf.mk instead of include/autoconf.mk, which is probably preferred behavior when moving options for SPL.

Signature · 8 lines
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20160822/c4b49f26/attachment.sig>
#12 lore raw

PATCH 4/4 tools: moveconfig: add --spl option to move options for SPL build

Quoted 8 lines · Tom Rini
Quoted 6 lines · Masahiro Yamada

Prior to this commit, the tool could not move options guarded by CONFIG_SPL_BUILD ifdef conditionals because they do not show up in include/autoconf.mk. This new option, if given, makes the tool parse spl/include/autoconf.mk instead of include/autoconf.mk, which is probably preferred behavior when moving options for SPL.

Quoted 4 lines
Quoted 1 line

I think we are still missing the case of some option being defined one way in the SPL build case but another the regular case, which one should be added to the defconfig?

In a set of slides[0] I found on the subject it looks like there was going to be a system where we could conditionally define options in defconfig based on whether we were building SPL or not. So we could run moveconfig in multiple passes and find what kind of tag we need.

S:CONFIG_FOO=200
T:CONFIG_FOO=300
ST:CONFIG_BAR=y
+S:CONFIG_BAZ=y
+T:CONFIG_QUX=y
+ST:CONFIG_QUUX=y
etc..

Did this ever get implemented?

[0]
http://www.denx.de/wiki/pub/U-Boot/MiniSummitELCE2014/uboot2014_kconfig.pdf

#13 lore raw

PATCH 4/4 tools: moveconfig: add --spl option to move options for SPL build

Hi Andrew,

2016-08-22 23:57 GMT+09:00 Andrew F. Davis <[email protected]>:

Quoted 3 lines

I think we are still missing the case of some option being defined one way in the SPL build case but another the regular case, which one should be added to the defconfig?

  #ifdef CONFIG_SPL_BUILD
     #define CONFIG_FOO   100
  #else
     #define CONFIG_FOO   200
  #endif

is a case where we can not migrate as is.

Generally, it will be changed as follows:

config FOO
int "foo"

config SPL_FOO
int "foo for SPL"

It is too much to have regular/SPL variants
for all cases, so we need to draw a line somewhere, though.

Quoted 13 lines

In a set of slides[0] I found on the subject it looks like there was going to be a system where we could conditionally define options in defconfig based on whether we were building SPL or not. So we could run moveconfig in multiple passes and find what kind of tag we need.

S:CONFIG_FOO=200
T:CONFIG_FOO=300
ST:CONFIG_BAR=y

(quoted diff) +3 -0
+S:CONFIG_BAZ=y
+T:CONFIG_QUX=y
+ST:CONFIG_QUUX=y

etc..

Quoted 5 lines

Did this ever get implemented?

[0]
http://www.denx.de/wiki/pub/U-Boot/MiniSummitELCE2014/uboot2014_kconfig.pdf

At first, I implemented like that,
but I realized it was tedious to use.

So, commit e02ee2548 switched to single .config like Linux has. Instead, this requires separate CONFIGs for U-Boot proper and SPL as mentioned above.

Signature · 2 lines
Best Regards
Masahiro Yamada
#14 lore raw

PATCH 4/4 tools: moveconfig: add --spl option to move options for SPL build

In reply to Masahiro Yamada
Quoted 8 lines · Masahiro Yamada

Prior to this commit, the tool could not move options guarded by CONFIG_SPL_BUILD ifdef conditionals because they do not show up in include/autoconf.mk. This new option, if given, makes the tool parse spl/include/autoconf.mk instead of include/autoconf.mk, which is probably preferred behavior when moving options for SPL.

Applied to u-boot/master, thanks!

Signature · 8 lines
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20160907/9fae9f5b/attachment.sig>