moveconfig: Fix code relying on now-stripped newline characters
PATCH 1/1 moveconfig: Fix code relying on now-stripped newline characters
Commit 37f815cad07d ("moveconfig: Use a function to read files") adds a helper function that can read a file as lines, but strips the newline characters. This change broke parts of moveconfig code that relied on their existence, resulting in a few issues:
Configs that are defined as empty aren't removed from header files (e.g. "#define CONFIG_REMAKE_ELF"). Make regex patterns use '\b' to match word boundaries instead of '\W' (which matched the newlines) so these lines still match and get removed.
All changes in defconfig are considered removed by savedefconfig even if they weren't, and line continuations in the headers aren't recognized and removed properly, because their checks explicitly look for a newline character. Remove the character from both comparisons.
The printed diff of header files is wrongly formatted and raises an IndexError if a blank line was removed. Let print() print the new lines, and use size-independent ways to check strings to fix the diff output.
---
tools/moveconfig.py | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-)
tools/moveconfig.py +13 -13
@@ -205,12 +205,12 @@ def show_diff(alines, blines, file_path, color_enabled): tofile=os.path.join('b', file_path)) for line in diff: - if line[0] == '-' and line[1] != '-': - print(color_text(color_enabled, COLOR_RED, line), end=' ') - elif line[0] == '+' and line[1] != '+': - print(color_text(color_enabled, COLOR_GREEN, line), end=' ') + if line.startswith('-') and not line.startswith('--'): + print(color_text(color_enabled, COLOR_RED, line)) + elif line.startswith('+') and not line.startswith('++'): + print(color_text(color_enabled, COLOR_GREEN, line)) else: - print(line, end=' ') + print(line) def extend_matched_lines(lines, matched, pre_patterns, post_patterns, extend_pre, extend_post):
@@ -368,7 +368,7 @@ def cleanup_one_header(header_path, patterns, args): matched = [] for i, line in enumerate(lines): - if i - 1 in matched and lines[i - 1][-2:] == '\\\n': + if i - 1 in matched and lines[i - 1].endswith('\\'): matched.append(i) continue for pattern in patterns:
@@ -380,9 +380,9 @@ def cleanup_one_header(header_path, patterns, args): return # remove empty #ifdef ... #endif, successive blank lines - pattern_if = re.compile(r'#\s*if(def|ndef)?\W') # #if, #ifdef, #ifndef - pattern_elif = re.compile(r'#\s*el(if|se)\W') # #elif, #else - pattern_endif = re.compile(r'#\s*endif\W') # #endif + pattern_if = re.compile(r'#\s*if(def|ndef)?\b') # #if, #ifdef, #ifndef + pattern_elif = re.compile(r'#\s*el(if|se)\b') # #elif, #else + pattern_endif = re.compile(r'#\s*endif\b') # #endif pattern_blank = re.compile(r'^\s*$') # empty line while True:
@@ -424,8 +424,8 @@ def cleanup_headers(configs, args): patterns = [] for config in configs: - patterns.append(re.compile(r'#\s*define\s+%s\W' % config)) - patterns.append(re.compile(r'#\s*undef\s+%s\W' % config)) + patterns.append(re.compile(r'#\s*define\s+%s\b' % config)) + patterns.append(re.compile(r'#\s*undef\s+%s\b' % config)) for dir in 'include', 'arch', 'board': for (dirpath, dirnames, filenames) in os.walk(dir):
@@ -451,7 +451,7 @@ def cleanup_one_extra_option(defconfig_path, configs, args): """ start = 'CONFIG_SYS_EXTRA_OPTIONS="' - end = '"\n' + end = '"' lines = read_file(defconfig_path)
@@ -812,7 +812,7 @@ def check_defconfig(self): for (action, value) in self.results: if action != ACTION_MOVE: continue - if not value + '\n' in defconfig_lines: + if not value in defconfig_lines: log += color_text(self.args.color, COLOR_YELLOW, "'%s' was removed by savedefconfig.\n" % value)
Signature · 1 line
2.34.1
Hi Alper,
Quoted 20 lines · Alper Nebi Yasak
Commit
37f815cad07d("moveconfig: Use a function to read files") adds a helper function that can read a file as lines, but strips the newline characters. This change broke parts of moveconfig code that relied on their existence, resulting in a few issues:Configs that are defined as empty aren't removed from header files (e.g. "#define
CONFIG_REMAKE_ELF"). Make regex patterns use '\b' to match word boundaries instead of '\W' (which matched the newlines) so these lines still match and get removed.All changes in defconfig are considered removed by savedefconfig even if they weren't, and line continuations in the headers aren't recognized and removed properly, because their checks explicitly look for a newline character. Remove the character from both comparisons.
The printed diff of header files is wrongly formatted and raises an IndexError if a blank line was removed. Let
print()print the new lines, and use size-independent ways to check strings to fix the diff output.
Quoted 5 lines
Signed-off-by Alper Nebi Yasak <[email protected]>---
tools/moveconfig.py | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-)
I was a little worried about the subtleties here. Thanks for fixing it! I almost got annoyed and wrote some tests, but I guess we can limp on without them.
Quoted 57 lines
tools/moveconfig.py +11
-11@@ -205,12 +205,12 @@ def show_diff(alines, blines, file_path, color_enabled): tofile=os.path.join('b', file_path)) for line in diff: - if line[0] == '-' and line[1] != '-': - print(color_text(color_enabled, COLOR_RED, line), end=' ') - elif line[0] == '+' and line[1] != '+': - print(color_text(color_enabled, COLOR_GREEN, line), end=' ') + if line.startswith('-') and not line.startswith('--'): + print(color_text(color_enabled, COLOR_RED, line)) + elif line.startswith('+') and not line.startswith('++'): + print(color_text(color_enabled, COLOR_GREEN, line)) else: - print(line, end=' ') + print(line) def extend_matched_lines(lines, matched, pre_patterns, post_patterns, extend_pre, extend_post):@@ -368,7 +368,7 @@ def cleanup_one_header(header_path, patterns, args): matched = [] for i, line in enumerate(lines): - if i - 1 in matched and lines[i - 1][-2:] == '\\\n': + if i - 1 in matched and lines[i - 1].endswith('\\'): matched.append(i) continue for pattern in patterns:@@ -380,9 +380,9 @@ def cleanup_one_header(header_path, patterns, args): return # remove empty #ifdef ... #endif, successive blank lines - pattern_if = re.compile(r'#\s*if(def|ndef)?\W') # #if, #ifdef, #ifndef - pattern_elif = re.compile(r'#\s*el(if|se)\W') # #elif, #else - pattern_endif = re.compile(r'#\s*endif\W') # #endif + pattern_if = re.compile(r'#\s*if(def|ndef)?\b') # #if, #ifdef, #ifndef + pattern_elif = re.compile(r'#\s*el(if|se)\b') # #elif, #else + pattern_endif = re.compile(r'#\s*endif\b') # #endif pattern_blank = re.compile(r'^\s*$') # empty line while True:@@ -424,8 +424,8 @@ def cleanup_headers(configs, args): patterns = [] for config in configs: - patterns.append(re.compile(r'#\s*define\s+%s\W' % config)) - patterns.append(re.compile(r'#\s*undef\s+%s\W' % config)) + patterns.append(re.compile(r'#\s*define\s+%s\b' % config)) + patterns.append(re.compile(r'#\s*undef\s+%s\b' % config)) for dir in 'include', 'arch', 'board': for (dirpath, dirnames, filenames) in os.walk(dir):
tools/moveconfig.py def cleanup_one_extra_option(defconfig_path, configs, args): · line 454
Quoted 5 lines
tools/moveconfig.py +1
-1@@ -451,7 +451,7 @@ def cleanup_one_extra_option(defconfig_path, configs, args): """ start = 'CONFIG_SYS_EXTRA_OPTIONS="' - end = '"\n' + end = '"'
''
Quoted 13 lines
tools/moveconfig.py +1
-2lines = read_file(defconfig_path)@@ -812,7 +812,7 @@ def check_defconfig(self): for (action, value) in self.results: if action != ACTION_MOVE: continue - if not value + '\n' in defconfig_lines: + if not value in defconfig_lines: log += color_text(self.args.color, COLOR_YELLOW, "'%s' was removed by savedefconfig.\n" % value) --
Quoted 2 lines
2.34.1
Regards,
Simon
Quoted 9 lines · Simon Glass
Quoted 4 lines · Alper Nebi Yasak
Commit
37f815cad07d("moveconfig: Use a function to read files") adds a helper function that can read a file as lines, but strips the newline characters. This change broke parts of moveconfig code that relied on their existence, resulting in a few issuesI was a little worried about the subtleties here. Thanks for fixing it! I almost got annoyed and wrote some tests, but I guess we can limp on without them.
Can't guarantee I found everything, just lucky I was trying a migration on REMAKE_ELF :)
Quoted 8 lines
Quoted 8 lines
tools/moveconfig.py +0
-0@@ -451,7 +451,7 @@ def cleanup_one_extra_option(defconfig_path, configs, args): """
Quoted 5 lines
tools/moveconfig.py def cleanup_one_extra_option(defconfig_path, configs, args): · line 454
Quoted 3 lines
tools/moveconfig.py +1
-1start = 'CONFIG_SYS_EXTRA_OPTIONS="' - end = '"\n' + end = '"'''
Looks to me like the double-quote character is necessary here, if I add CONFIG_SYS_EXTRA_OPTIONS="REMAKE_ELF" to chromebook_bob_defconfig and try migrating CONFIG_REMAKE_ELF the line doesn't get removed otherwise.
Trying to add more lines of diff context below:
Quoted 3 lines
tools/moveconfig.py def cleanup_one_extra_option(defconfig_path, configs, args): · line 457
Quoted 3 lines
tools/moveconfig.py +0
-0lines = read_file(defconfig_path)
for i, line in enumerate(lines):
if line.startswith(start) and line.endswith(end):
break
else:
# CONFIG_SYS_EXTRA_OPTIONS was not found in this defconfig return
old_tokens = line[len(start):-len(end)].split(',') new_tokens = []
for token in old_tokens:
pos = token.find('=')
if not (token[:pos] if pos >= 0 else token) in configs:
new_tokens.append(token)
Hi Alper,
Quoted 8 lines · Alper Nebi Yasak
Quoted 6 lines · Simon Glass
Quoted 4 lines · Alper Nebi Yasak
Commit
37f815cad07d("moveconfig: Use a function to read files") adds a helper function that can read a file as lines, but strips the newline characters. This change broke parts of moveconfig code that relied on their existence, resulting in a few issues
Quoted 6 lines
Quoted 3 lines
I was a little worried about the subtleties here. Thanks for fixing it! I almost got annoyed and wrote some tests, but I guess we can limp on without them.
Can't guarantee I found everything, just lucky I was trying a migration on
REMAKE_ELF:)
Quoted 15 lines
Quoted 9 lines
Quoted 9 lines
tools/moveconfig.py +0
-0@@ -451,7 +451,7 @@ def cleanup_one_extra_option(defconfig_path, configs, args): """Quoted 5 lines
tools/moveconfig.py def cleanup_one_extra_option(defconfig_path, configs, args): · line 454
Quoted 3 lines
tools/moveconfig.py +1
-1start = 'CONFIG_SYS_EXTRA_OPTIONS="' - end = '"\n' + end = '"'''
Quoted 3 lines
Looks to me like the double-quote character is necessary here, if I add
CONFIG_SYS_EXTRA_OPTIONS="REMAKE_ELF" tochromebook_bob_defconfigand try migratingCONFIG_REMAKE_ELFthe line doesn't get removed otherwise.
Yes you are right, ignore this. It is barely legible in the font I am using here.
Quoted 13 lines
Trying to add more lines of diff context below:
Quoted 3 lines
tools/moveconfig.py def cleanup_one_extra_option(defconfig_path, configs, args): · line 457
Quoted 3 lines
tools/moveconfig.py +0
-0lines = read_file(defconfig_path)for i, line in enumerate(lines):
ifline.startswith(start)andline.endswith(end):
break
else:
#CONFIG_SYS_EXTRA_OPTIONSwas not found in this defconfig return
Quoted 7 lines
old_tokens= line[len(start):-len(end)].split(',')new_tokens= []for token in
old_tokens:
pos =token.find('=')
if not (token[:pos] if pos >= 0 else token) in configs:
new_tokens.append(token)
Regards,
Simon