Skip to contentSkip to conversation

devtool: remove duplicate overrides

PATCH In review Dan Dedrick 3 participants 3 messages Jan 21, 2019 – Feb 22, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 devtool: remove duplicate overrides

DEVTOOL_EXTRA_OVERRIDES only needs one entry for each instance of overrides. Previous to these changes it would find every override to SRC_URI and add it to the list. This would duplicate instances where SRC_URI is modified multiple times with the same override like:
SRC_URI_append_foo += "file://0001-foo.patch" SRC_URI_append_foo += "file://0002-bar.patch"

A bbappend might also overwrite a SRC_URI override, which would also cause multiple instances to occur.

When there are multiple instances of the same override in DEVTOOL_EXTRA_OVERRIDES it causes devtool modify to fail when creating override branches. The failure occurs when attempting to create the same override branch a second time and looks like this:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>

     0001:
 *** 0002:devtool_post_patch(d)
     0003:

File: '/build/poky/meta/classes/devtool-source.bbclass', lineno: 202, function: devtool_post_patch
0198:
0199: for override in extra_override_list:
0200: localdata = bb.data.createCopy(d)
0201: if override in default_overrides:
*** 0202: bb.process.run('git branch devtool-override-%s %s' % (override, devbranch), cwd=srcsubdir)
0203: else:
0204: # Reset back to the initial commit on a new branch
0205: bb.process.run('git checkout %s -b devtool-override-%s' % (initial_rev, override), cwd=srcsubdir)
0206: # Run do_patch function with the override applied
File: '/build/poky/bitbake/lib/bb/process.py', lineno: 178, function: run

     0174:        if not stderr is None:
     0175:            stderr = stderr.decode("utf-8")
     0176:
     0177:    if pipe.returncode != 0:

*** 0178: raise ExecutionError(cmd, pipe.returncode, stdout, stderr)
0179: return stdout, stderr
Exception: bb.process.ExecutionError: Execution of 'git branch devtool-override-foo devtool' failed with exit code 128:
fatal: A branch named 'devtool-override-foo' already exists.

---
 scripts/lib/devtool/standard.py | 5 +++++
 1 file changed, 5 insertions(+)
scripts/lib/devtool/standard.py +5 -0
index d14b7a6543..a45ad36812 100644
@@ -509,6 +509,11 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works
             if not 'flag' in event:
                 if event['op'].startswith(('_append[', '_prepend[')):
                     extra_overrides.append(event['op'].split('[')[1].split(']')[0])
+        # We want to remove duplicate overrides. If a recipe had multiple
+        # SRC_URI_override += values it would cause mulitple instances of
+        # overrides. This doesn't play nicely with things like creating a
+        # branch for every instance of DEVTOOL_EXTRA_OVERRIDES.
+        extra_overrides = list(set(extra_overrides))
         if extra_overrides:
             logger.info('SRC_URI contains some conditional appends/prepends - will create branches to represent these')
 
Signature · 1 line
2.20.1
#1 lore raw

This has already been merged, but see below.

Quoted 22 lines

-----Original Message-----
From: [email protected] <openembedded- [email protected]> On Behalf Of Dan Dedrick
Sent: den 21 januari 2019 17:05
To: [email protected]
Subject: [OE-core] [PATCH] devtool: remove duplicate overrides

DEVTOOL_EXTRA_OVERRIDES only needs one entry for each instance of overrides. Previous to these changes it would find every override to SRC_URI and add it to the list. This would duplicate instances where SRC_URI is modified multiple times with the same override like:
SRC_URI_append_foo += "file://0001-foo.patch" SRC_URI_append_foo += "file://0002-bar.patch"

A bbappend might also overwrite a SRC_URI override, which would also cause multiple instances to occur.

When there are multiple instances of the same override in DEVTOOL_EXTRA_OVERRIDES it causes devtool modify to fail when creating override branches. The failure occurs when attempting to create the same override branch a second time and looks like this:

Quoted 8 lines

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>

     0001:
 *** 0002:devtool_post_patch(d)
     0003:

File: '/build/poky/meta/classes/devtool-source.bbclass', lineno: 202, function: devtool_post_patch
0198:
0199: for override in extra_override_list:

The reference to extra_override_list above indicates that you developed this patch for an older version of OE-Core, since that variable was renamed on master of OE-Core in October when this problem was fixed (see commit 95a92195). However, that fix was made to devtool-source.bbclass, where it uses the value of DEVTOOL_EXTRA_OVERRIDES, instead of to scripts/lib/devtool/standard.py where you made the change. This is probably why you did not notice that the problem had already been fixed.

Quoted 40 lines

0200: localdata = bb.data.createCopy(d)
0201: if override in default_overrides:
*** 0202: bb.process.run('git branch devtool-override-%s %s' % (override, devbranch), cwd=srcsubdir)
0203: else:
0204: # Reset back to the initial commit on a new branch
0205: bb.process.run('git checkout %s -b devtool-override-%s' % (initial_rev, override), cwd=srcsubdir)
0206: # Run do_patch function with the override applied
File: '/build/poky/bitbake/lib/bb/process.py', lineno: 178, function: run

     0174:        if not stderr is None:
     0175:            stderr = stderr.decode("utf-8")
     0176:
     0177:    if pipe.returncode != 0:

*** 0178: raise ExecutionError(cmd, pipe.returncode, stdout, stderr)
0179: return stdout, stderr
Exception: bb.process.ExecutionError: Execution of 'git branch devtool-override-foo devtool' failed with exit code 128:
fatal: A branch named 'devtool-override-foo' already exists.

---
 scripts/lib/devtool/standard.py | 5 +++++
 1 file changed, 5 insertions(+)
scripts/lib/devtool/standard.py +5 -1
index d14b7a6543..a45ad36812 100644
@@ -509,6 +509,11 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works
             if not 'flag' in event:
                 if event['op'].startswith(('_append[', '_prepend[')):
 
 extra_overrides.append(event['op'].split('[')[1].split(']')[0])
+        # We want to remove duplicate overrides. If a recipe had multiple
+        # SRC_URI_override += values it would cause mulitple instances of
+        # overrides. This doesn't play nicely with things like creating a
+        # branch for every instance of DEVTOOL_EXTRA_OVERRIDES.
+        extra_overrides = list(set(extra_overrides))
         if extra_overrides:
             logger.info('SRC_URI contains some conditional appends/prepends - will create branches to represent these')
 
--
Quoted 1 line

2.20.1

//Peter

#2 lore raw

As Peter mentions the underlying issue is already mitigated when we read the value of DEVTOOL_EXTRA_OVERRIDES; for clarity I'd say let's revert this (90f667db2219f04e6d61588cd61056d3d8da6d7d). My apologies, I wasn't paying attention earlier.

Cheers,
Paul

Quoted 86 lines · NZDT Peter Kjellerstedt

This has already been merged, but see below.

Quoted 22 lines

-----Original Message-----
From: [email protected] <openembedded- [email protected]> On Behalf Of Dan Dedrick
Sent: den 21 januari 2019 17:05
To: [email protected]
Subject: [OE-core] [PATCH] devtool: remove duplicate overrides

DEVTOOL_EXTRA_OVERRIDES only needs one entry for each instance of overrides. Previous to these changes it would find every override to SRC_URI and add it to the list. This would duplicate instances where SRC_URI is modified multiple times with the same override like:
SRC_URI_append_foo += "file://0001-foo.patch" SRC_URI_append_foo += "file://0002-bar.patch"

A bbappend might also overwrite a SRC_URI override, which would also cause multiple instances to occur.

When there are multiple instances of the same override in DEVTOOL_EXTRA_OVERRIDES it causes devtool modify to fail when creating override branches. The failure occurs when attempting to create the same override branch a second time and looks like this:

Quoted 8 lines

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>

     0001:
 *** 0002:devtool_post_patch(d)
     0003:

File: '/build/poky/meta/classes/devtool-source.bbclass', lineno: 202, function: devtool_post_patch
0198:
0199: for override in extra_override_list:

The reference to extra_override_list above indicates that you developed this patch for an older version of OE-Core, since that variable was renamed on master of OE-Core in October when this problem was fixed (see commit 95a92195). However, that fix was made to devtool-source.bbclass, where it uses the value of DEVTOOL_EXTRA_OVERRIDES, instead of to scripts/lib/devtool/standard.py where you made the change. This is probably why you did not notice that the problem had already been fixed.

Quoted 40 lines

0200: localdata = bb.data.createCopy(d)
0201: if override in default_overrides:
*** 0202: bb.process.run('git branch devtool-override-%s %s' % (override, devbranch), cwd=srcsubdir)
0203: else:
0204: # Reset back to the initial commit on a new branch
0205: bb.process.run('git checkout %s -b devtool-override-%s' % (initial_rev, override), cwd=srcsubdir)
0206: # Run do_patch function with the override applied
File: '/build/poky/bitbake/lib/bb/process.py', lineno: 178, function: run

     0174:        if not stderr is None:
     0175:            stderr = stderr.decode("utf-8")
     0176:
     0177:    if pipe.returncode != 0:

*** 0178: raise ExecutionError(cmd, pipe.returncode, stdout, stderr)
0179: return stdout, stderr
Exception: bb.process.ExecutionError: Execution of 'git branch devtool-override-foo devtool' failed with exit code 128:
fatal: A branch named 'devtool-override-foo' already exists.

---
 scripts/lib/devtool/standard.py | 5 +++++
 1 file changed, 5 insertions(+)
scripts/lib/devtool/standard.py +5 -1
index d14b7a6543..a45ad36812 100644
@@ -509,6 +509,11 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works
             if not 'flag' in event:
                 if event['op'].startswith(('_append[', '_prepend[')):
 
 extra_overrides.append(event['op'].split('[')[1].split(']')[0])
+        # We want to remove duplicate overrides. If a recipe had multiple
+        # SRC_URI_override += values it would cause mulitple instances of
+        # overrides. This doesn't play nicely with things like creating a
+        # branch for every instance of DEVTOOL_EXTRA_OVERRIDES.
+        extra_overrides = list(set(extra_overrides))
         if extra_overrides:
             logger.info('SRC_URI contains some conditional appends/prepends - will create branches to represent these')
 
--
Quoted 1 line

2.20.1

//Peter

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre