Skip to contentSkip to conversation

devtool: upgrade: ignore changelogs from 3rd party

PATCH Superseded [email protected] 2 participants 4 messages Aug 3

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 devtool: upgrade: ignore changelogs from 3rd party

From: Daniel Turull <[email protected]>

Some upstream projects bundle vendored dependencies in their source tree (e.g. nghttp2 ships third-party/mruby, which has its own NEWS.md). The changelog extractor could mistake one of these vendored changelogs for the recipe's own, misattributing unrelated upstream changes to the package being upgraded.

Exclude paths under common vendoring directory names (third-party, vendor, external, deps, etc.) from changelog candidates.

AI-Generated: Kiro with Claude Sonnet 5

---
 scripts/lib/devtool/upgrade.py | 10 ++++++++++
 1 file changed, 10 insertions(+)
scripts/lib/devtool/upgrade.py +10 -0
index 13d51bf952..495a5b8217 100644
@@ -55,6 +55,13 @@ _CHANGELOG_BASENAMES = {
     'perldelta.pod',
 }
 
+# Path components indicating a bundled/vendored dependency rather than the
+# recipe's own source, so its changelog-like files should not be mistaken
+# for the recipe's own changes (e.g. third-party/mruby/NEWS.md in nghttp2).
+_VENDORED_PATH_RE = re.compile(
+    r'(^|/)(third[-_]party|vendor|vendored|external|extern|deps|3rdparty)(/|$)',
+    re.IGNORECASE)
+
 def _run(cmd, cwd=''):
     logger.debug("Running command %s> %s" % (cwd,cmd))
     return bb.process.run('%s' % cmd, cwd=cwd)
@@ -591,6 +598,9 @@ def _extract_changelog(srctree, pn, old_ver, new_ver, old_tag, new_tag, workspac
     try:
         stdout, _ = _run('git diff --name-only %s %s' % (old_tag, new_tag), srctree)
         changed_files = [f.strip() for f in stdout.splitlines() if f.strip()]
+        # Exclude bundled/vendored dependencies; their changelogs are not
+        # relevant to this recipe's own version bump.
+        changed_files = [f for f in changed_files if not _VENDORED_PATH_RE.search(f)]
 
         # First pass: collect per-version release notes that changed
         # Matches files with a version number whose path suggests release notes
#1 lore raw

PATCH 2/2 devtool: upgrade: diff git-log-style changelogs by commit hash

From: Daniel Turull <[email protected]>

Files like nghttp2's ChangeLog are the literal output of git log, regenerated wholesale on every release. New commits are prepended, so every existing commit shifts position even though its content is unchanged. A textual diff sees that shift as removed/added lines instead of showing just the commits that were actually added.

Diff these files structurally instead: compare commit hashes between the old and new file to tell moved commits from genuinely new ones, and report the subject lines of the new commits. Falls back to the existing line-based diff for changelogs that aren't git-log output.

AI-Generated: Kiro with Claude Sonnet 5

---
 scripts/lib/devtool/upgrade.py | 71 +++++++++++++++++++++++++++++-----
 1 file changed, 62 insertions(+), 9 deletions(-)
scripts/lib/devtool/upgrade.py +62 -9
index 495a5b8217..0155578d68 100644
@@ -589,6 +589,64 @@ def _resolve_rst_includes(content, srctree):
     return ''.join(result)
 
 
+_GIT_LOG_COMMIT_RE = re.compile(r'^commit ([0-9a-f]{7,40})\b.*$', re.MULTILINE)
+_GIT_LOG_HEADER_RE = re.compile(r'^(Merge|Author|AuthorDate|Commit|CommitDate):')
+
+def _diff_git_log_changelog(old_content, new_content):
+    """Structurally diff a ChangeLog file that is itself the output of
+    `git log` (e.g. nghttp2). Such files are regenerated wholesale on every
+    release: new commits are prepended, so every existing commit shifts
+    position even though its content is unchanged. A textual diff would
+    see that shift as removed/added lines instead of showing just the
+    commits that were actually added. Comparing commit hashes instead of
+    line content tells the two cases apart: hashes already present in
+    old_content are commits that merely moved, so they're skipped; any
+    other hash is a genuinely new commit. Returns the subject line of
+    each new commit as a single string, or None if either file doesn't
+    look like git log output."""
+    old_hashes = set(_GIT_LOG_COMMIT_RE.findall(old_content))
+    new_commits = _GIT_LOG_COMMIT_RE.split(new_content)[1:]  # [hash, block, hash, block, ...]
+    if not old_hashes or not new_commits:
+        return None
+
+    subjects = []
+    for commit_hash, block in zip(new_commits[0::2], new_commits[1::2]):
+        if commit_hash in old_hashes:
+            continue
+        message_lines = [line.strip() for line in block.splitlines()
+                          if line.strip() and not _GIT_LOG_HEADER_RE.match(line)]
+        # Skip the 'Merge pull request ...' line GitHub adds as the first
+        # message line of a merge commit; the actual change subject is the
+        # next line, and keeping the merge line would duplicate it.
+        subject = next((l for l in message_lines if not l.startswith('Merge pull request ')), None)
+        # Collapse adjacent duplicates: squash-merged PRs can produce
+        # consecutive commits with the same subject line.
+        if subject and (not subjects or subjects[-1] != subject):
+            subjects.append(subject)
+    return '\n'.join(subjects) if subjects else None
+
+
+def _changelog_file_candidate(srctree, old_tag, new_tag, fname):
+    """Return the changelog text to use for a single changelog-like file that
+    changed between old_tag and new_tag, or None if it has no usable
+    content. Tries a structural git-log diff first (for files like
+    nghttp2's ChangeLog that are `git log` output regenerated on every
+    release), falling back to the added lines of a plain-text diff."""
+    try:
+        old_file, _ = _run('git show %s' % shlex.quote('%s:%s' % (old_tag, fname)), srctree)
+        new_file, _ = _run('git show %s' % shlex.quote('%s:%s' % (new_tag, fname)), srctree)
+        candidate = _diff_git_log_changelog(old_file, new_file)
+        if candidate is not None:
+            return candidate
+    except bb.process.ExecutionError:
+        pass
+
+    diff_out, _ = _run('git diff %s %s -- %s' % (old_tag, new_tag, shlex.quote(fname)), srctree)
+    added_lines = [line[1:] for line in diff_out.splitlines()
+                   if line.startswith('+') and not line.startswith('+++')]
+    return '\n'.join(added_lines) if added_lines else None
+
+
 def _extract_changelog(srctree, pn, old_ver, new_ver, old_tag, new_tag, workspace_path, is_git_source):
     """Extract changelog between old and new version using devtool git tags."""
     changelog_content = None
@@ -630,15 +688,10 @@ def _extract_changelog(srctree, pn, old_ver, new_ver, old_tag, new_tag, workspac
             for fname in changed_files:
                 basename = os.path.basename(fname).lower()
                 if basename in _CHANGELOG_BASENAMES:
-                    diff_out, _ = _run('git diff %s %s -- %s' % (old_tag, new_tag, shlex.quote(fname)), srctree)
-                    if diff_out.strip():
-                        lines = [line[1:] for line in diff_out.splitlines()
-                                 if line.startswith('+') and not line.startswith('+++')]
-                        if lines:
-                            candidate = '\n'.join(lines)
-                            if not changelog_content or len(candidate) > len(changelog_content):
-                                changelog_content = candidate
-                                changelog_fname = fname
+                    candidate = _changelog_file_candidate(srctree, old_tag, new_tag, fname)
+                    if candidate and (not changelog_content or len(candidate) > len(changelog_content)):
+                        changelog_content = candidate
+                        changelog_fname = fname
     except bb.process.ExecutionError as e:
         logger.warning('Changelog file extraction failed: %s' % str(e))
 
#2 lore raw
In reply to [email protected]
Quoted 8 lines · Daniel Turull via lists.openembedded.org

From: Daniel Turull <[email protected]>

Some upstream projects bundle vendored dependencies in their source tree (e.g. nghttp2 ships third-party/mruby, which has its own NEWS.md). The changelog extractor could mistake one of these vendored changelogs for the recipe's own, misattributing unrelated upstream changes to the package being upgraded.

Quoted 8 lines

Exclude paths under common vendoring directory names (third-party, vendor, external, deps, etc.) from changelog candidates.

AI-Generated: Kiro with Claude Sonnet 5

---
 scripts/lib/devtool/upgrade.py | 10 ++++++++++

 1 file changed, 10 insertions(+)

Thanks for these. Unfortunately something in here caused the devtool tests to fail:

https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/4274

Cheers,

Richard

#3 lore raw
Quoted 12 lines · Richard Purdie
Quoted 11 lines · Daniel Turull via lists.openembedded.org

From: Daniel Turull <[email protected]>

Some upstream projects bundle vendored dependencies in their source tree (e.g. nghttp2 ships third-party/mruby, which has its own NEWS.md). The changelog extractor could mistake one of these vendored changelogs for the recipe's own, misattributing unrelated upstream changes to the package being upgraded.

Exclude paths under common vendoring directory names (third-party, vendor, external, deps, etc.) from changelog candidates.

Quoted 8 lines
Quoted 5 lines

AI-Generated: Kiro with Claude Sonnet 5

---
 scripts/lib/devtool/upgrade.py | 10 ++++++++++

 1 file changed, 10 insertions(+)

Thanks for these. Unfortunately something in here caused the devtool tests to fail:

Ok. I'll take a look and run the test locally. I just did happy testing with a couple of packages and miss to run the devtool selftests.

Best regards,

Daniel

Quoted 4 lines

Cheers,

Richard