Skip to contentSkip to conversation

devtool: upgrade: extract changelog between versions

PATCH Superseded [email protected] 2 participants 6 messages May 18 – May 22 1 message not in the archive

v4 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

PATCH v4 1/1 devtool: upgrade: extract changelog between versions

From: Daniel Turull <[email protected]>

Automatically extract changelog information when upgrading a recipe. Uses the devtool-base tags created during upgrade to diff known changelog files (NEWS, ChangeLog, CHANGES, etc.) between the old and new versions. For git-based sources, falls back to git log --oneline if no changelog file changed.

Also match per-version release notes files like git's Documentation/RelNotes/2.53.0.adoc or mesa's docs/relnotes/26.0.3.rst.

These are detected by matching filenames that contain a version number with a documentation extension (.txt, .md, .rst, .adoc). Their full content is read via git show since they are new files rather than modifications to an existing changelog.

Collapse runs of multiple blank lines into a single blank line and strip leading/trailing whitespace from the extracted changelog. This produces cleaner output that is suitable for inclusion in commit messages by tools like AUH.

Output is written to workspace/changelogs/<pn>.txt and cleaned up on devtool reset. This allows AUH and other tools to pick up the changelog without implementing their own extraction logic.

Add test cases for existing test recipes

AI-generated: Uses kiro with claude-opus-4.6 model

---

v2:
- squash v1 patches into one commit and keep commit messages
- Integrate test cases with exiting devtool selftests

v3:
- do not silently skip the test if changelog is missing

v4:
- fix typo in commit message and fix AI-generated tag.

---
 .../python3-guessing-game_git.bb.changelog    |  3 +
 .../devtool-upgrade-test1_1.5.3.bb.changelog  | 24 +++++
 .../devtool-upgrade-test2_git.bb.changelog    |  3 +
 .../devtool-upgrade-test3_1.5.3.bb.changelog  | 24 +++++
 .../devtool-upgrade-test4_1.5.3.bb.changelog  | 24 +++++
 .../devtool-upgrade-test5_git.bb.changelog    |  3 +
 meta/lib/oeqa/selftest/cases/devtool.py       | 27 ++++++
 scripts/lib/devtool/standard.py               |  8 ++
 scripts/lib/devtool/upgrade.py                | 95 +++++++++++++++++++
 9 files changed, 211 insertions(+)
 create mode 100644 meta-selftest/recipes-devtools/python/python3-guessing-game_git.bb.changelog
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-upgrade-test1_1.5.3.bb.changelog
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-upgrade-test2_git.bb.changelog
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-upgrade-test3_1.5.3.bb.changelog
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-upgrade-test4_1.5.3.bb.changelog
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-upgrade-test5_git.bb.changelog
meta-selftest/recipes-devtools/python/python3-guessing-game_git.bb.changelog new file +3 -0
new file mode 100644
index 0000000000..571220b2a4
@@ -0,0 +1,3 @@
+Changelog for python3-guessing-game: 0.1.0 -> 0.2.0
+
+40cf004 Sync with maturin tutorial source
meta-selftest/recipes-test/devtool/devtool-upgrade-test1_1.5.3.bb.changelog new file +24 -0
new file mode 100644
index 0000000000..04d9be5103
@@ -0,0 +1,24 @@
+Changelog for devtool-upgrade-test1: 1.5.3 -> 1.6.0
+
+1.6.0 - 15 March 2015
+  - fix lstat64 support when unavailable - separate patches supplied by
+    Ganael Laplanche and Peter Korsgaard
+  - (#1506) new option "-D" / "--delay-start" to only show bar after N
+    seconds (Damon Harper)
+  - new option "--fineta" / "-I" to show ETA as time of day rather than time
+    remaining - patch supplied by Erkki Seppälä (r147)
+  - (#1509) change ETA (--eta / -e) so that days are given if the hours
+    remaining are 24 or more (Jacek Wielemborek)
+  - (#1499) repeat read and write attempts on partial buffer fill/empty to
+    work around post-signal transfer rate drop reported by Ralf Ramsauer
+  - (#1507) do not try to calculate total size in line mode, due to bug
+    reported by Jacek Wielemborek and Michiel Van Herwegen
+  - cleanup: removed defunct RATS comments and unnecessary copyright notices
+  - clean up displayed lines when using --watchfd PID, when PID exits
+  - output errors on a new line to avoid overwriting transfer bar
+
+1.5.7 - 26 August 2014
+  - show KiB instead of incorrect kiB (Debian bug #706175)
+  - (#1284) do not gzip man page, for non-Linux OSes (Bob Friesenhahn)
+  - work around "awk" bug in tests/016-numeric-timer in decimal "," locales
+  - fix "make rpm" and "make srpm", extend "make release" to sign releases
meta-selftest/recipes-test/devtool/devtool-upgrade-test2_git.bb.changelog new file +3 -0
new file mode 100644
index 0000000000..b7f0019d56
@@ -0,0 +1,3 @@
+Changelog for devtool-upgrade-test2: 0.1+git -> 0.1+git
+
+6cc6077 dbus-wait.c: Fix typo
meta-selftest/recipes-test/devtool/devtool-upgrade-test3_1.5.3.bb.changelog new file +24 -0
new file mode 100644
index 0000000000..d9994fe063
@@ -0,0 +1,24 @@
+Changelog for devtool-upgrade-test3: 1.5.3 -> 1.6.0
+
+1.6.0 - 15 March 2015
+  - fix lstat64 support when unavailable - separate patches supplied by
+    Ganael Laplanche and Peter Korsgaard
+  - (#1506) new option "-D" / "--delay-start" to only show bar after N
+    seconds (Damon Harper)
+  - new option "--fineta" / "-I" to show ETA as time of day rather than time
+    remaining - patch supplied by Erkki Seppälä (r147)
+  - (#1509) change ETA (--eta / -e) so that days are given if the hours
+    remaining are 24 or more (Jacek Wielemborek)
+  - (#1499) repeat read and write attempts on partial buffer fill/empty to
+    work around post-signal transfer rate drop reported by Ralf Ramsauer
+  - (#1507) do not try to calculate total size in line mode, due to bug
+    reported by Jacek Wielemborek and Michiel Van Herwegen
+  - cleanup: removed defunct RATS comments and unnecessary copyright notices
+  - clean up displayed lines when using --watchfd PID, when PID exits
+  - output errors on a new line to avoid overwriting transfer bar
+
+1.5.7 - 26 August 2014
+  - show KiB instead of incorrect kiB (Debian bug #706175)
+  - (#1284) do not gzip man page, for non-Linux OSes (Bob Friesenhahn)
+  - work around "awk" bug in tests/016-numeric-timer in decimal "," locales
+  - fix "make rpm" and "make srpm", extend "make release" to sign releases
meta-selftest/recipes-test/devtool/devtool-upgrade-test4_1.5.3.bb.changelog new file +24 -0
new file mode 100644
index 0000000000..3a37271eb7
@@ -0,0 +1,24 @@
+Changelog for devtool-upgrade-test4: 1.5.3 -> 1.6.0
+
+1.6.0 - 15 March 2015
+  - fix lstat64 support when unavailable - separate patches supplied by
+    Ganael Laplanche and Peter Korsgaard
+  - (#1506) new option "-D" / "--delay-start" to only show bar after N
+    seconds (Damon Harper)
+  - new option "--fineta" / "-I" to show ETA as time of day rather than time
+    remaining - patch supplied by Erkki Seppälä (r147)
+  - (#1509) change ETA (--eta / -e) so that days are given if the hours
+    remaining are 24 or more (Jacek Wielemborek)
+  - (#1499) repeat read and write attempts on partial buffer fill/empty to
+    work around post-signal transfer rate drop reported by Ralf Ramsauer
+  - (#1507) do not try to calculate total size in line mode, due to bug
+    reported by Jacek Wielemborek and Michiel Van Herwegen
+  - cleanup: removed defunct RATS comments and unnecessary copyright notices
+  - clean up displayed lines when using --watchfd PID, when PID exits
+  - output errors on a new line to avoid overwriting transfer bar
+
+1.5.7 - 26 August 2014
+  - show KiB instead of incorrect kiB (Debian bug #706175)
+  - (#1284) do not gzip man page, for non-Linux OSes (Bob Friesenhahn)
+  - work around "awk" bug in tests/016-numeric-timer in decimal "," locales
+  - fix "make rpm" and "make srpm", extend "make release" to sign releases
meta-selftest/recipes-test/devtool/devtool-upgrade-test5_git.bb.changelog new file +3 -0
new file mode 100644
index 0000000000..7c5ecf3505
@@ -0,0 +1,3 @@
+Changelog for devtool-upgrade-test5: 0.1+git -> 0.1+git
+
+0a60d6a Add dummy commit on tip for testing
meta/lib/oeqa/selftest/cases/devtool.py +27 -0
index 5ed69aee1b..5a6f38f8d5 100644
@@ -1944,6 +1944,22 @@ class DevtoolUpgradeTests(DevtoolBase):
         except:
             self.skip("Git user.name and user.email must be set")
 
+    def _check_changelog(self, recipe, oldrecipefile):
+        """Compare extracted changelog against reference data."""
+        changelog_ref = oldrecipefile + '.changelog'
+        self.assertExists(changelog_ref, 'Changelog reference file must exist for %s' % recipe)
+        changelog_file = os.path.join(self.workspacedir, 'changelogs', '%s.txt' % recipe)
+        with open(changelog_ref, 'r') as f:
+            expected = f.read()
+        if not expected:
+            self.assertNotExists(changelog_file,
+                'Changelog file should not exist when reference is empty')
+        else:
+            self.assertExists(changelog_file, 'Changelog file should exist after upgrade')
+            with open(changelog_file, 'r') as f:
+                actual = f.read()
+            self.assertEqual(expected, actual)
+
     def test_devtool_upgrade(self):
         # Check preconditions
         self.assertTrue(not os.path.exists(self.workspacedir), 'This test cannot be run with a workspace directory under the build directory')
@@ -1982,6 +1998,8 @@ class DevtoolUpgradeTests(DevtoolBase):
         with open(newrecipefile, 'r') as f:
             newlines = f.readlines()
         self.assertEqual(desiredlines, newlines)
+        # Check changelog
+        self._check_changelog(recipe, oldrecipefile)
         # Check devtool reset recipe
         result = runCmd('devtool reset %s -n' % recipe)
         result = runCmd('devtool status')
@@ -2016,11 +2034,14 @@ class DevtoolUpgradeTests(DevtoolBase):
         with open(newrecipefile, 'r') as f:
             newlines = f.readlines()
         self.assertEqual(desiredlines, newlines)
+        # Check changelog
+        self._check_changelog(recipe, oldrecipefile)
         # Check devtool reset recipe
         result = runCmd('devtool reset %s -n' % recipe)
         result = runCmd('devtool status')
         self.assertNotIn(recipe, result.output)
         self.assertNotExists(os.path.join(self.workspacedir, 'recipes', recipe), 'Recipe directory should not exist after resetting')
+        self.assertNotExists(os.path.join(self.workspacedir, 'changelogs', '%s.txt' % recipe), 'Changelog file should be removed after reset')
 
     def test_devtool_upgrade_git(self):
         self._test_devtool_upgrade_git_by_recipe('devtool-upgrade-test2', '6cc6077a36fe2648a5f993fe7c16c9632f946517')
@@ -2051,6 +2072,8 @@ class DevtoolUpgradeTests(DevtoolBase):
         with open(newrecipefile, 'r') as f:
             newlines = f.readlines()
         self.assertEqual(desiredlines, newlines)
+        # Check changelog
+        self._check_changelog(recipe, oldrecipefile)
 
     def test_devtool_upgrade_all_checksums(self):
         # Check preconditions
@@ -2075,6 +2098,8 @@ class DevtoolUpgradeTests(DevtoolBase):
         with open(newrecipefile, 'r') as f:
             newlines = f.readlines()
         self.assertEqual(desiredlines, newlines)
+        # Check changelog
+        self._check_changelog(recipe, oldrecipefile)
 
     def test_devtool_upgrade_recipe_upgrade_extra_tasks(self):
         # Check preconditions
@@ -2116,6 +2141,8 @@ class DevtoolUpgradeTests(DevtoolBase):
         with open(newcratesincfile, 'r') as f:
             newlines = f.readlines()
         self.assertEqual(desiredlines, newlines)
+        # Check changelog
+        self._check_changelog(recipe, oldrecipefile)
         # Check devtool reset recipe
         result = runCmd('devtool reset %s -n' % recipe)
         result = runCmd('devtool status')
scripts/lib/devtool/standard.py +8 -0
index 42fb13872d..2a3a62d081 100644
@@ -2046,6 +2046,14 @@ def _reset(recipes, no_clean, remove_work, config, basepath, workspace):
 
         clean_preferred_provider(pn, config.workspace_path)
 
+        # Clean up changelog if present
+        changelog_file = os.path.join(config.workspace_path, 'changelogs', '%s.txt' % pn)
+        if os.path.exists(changelog_file):
+            os.remove(changelog_file)
+            changelog_dir = os.path.dirname(changelog_file)
+            if not os.listdir(changelog_dir):
+                os.rmdir(changelog_dir)
+
 def reset(args, config, basepath, workspace):
     """Entry point for the devtool 'reset' subcommand"""
 
scripts/lib/devtool/upgrade.py +95 -0

119 lines not on this page · raw message

#1 lore raw
Quoted 3 lines · On Mon, 18 May 2026 at 15:59, <[email protected]> wrote:

Output is written to workspace/changelogs/<pn>.txt and cleaned up on devtool reset. This allows AUH and other tools to pick up the changelog without implementing their own extraction logic.

The patch review pointed out that this has been tested on a small handful of recipes, but before the change gets accepted, we really need to perform an industrial-strength set of upgrades with it, e.g. a typical AUH run on all of oe-core (possibly rolled back in time to include even more recipes), and then shake out all corner cases, unexpected fails, and pathological changelogs. I was going to make that run, then I realized that the companion patch for AUH has not yet been posted. Can you make that please?

Alex

#2 lore raw
Quoted 16 lines

-----Original Message-----
From: Alexander Kanavin <[email protected]>
Sent: Thursday, 21 May 2026 12:18
To: Daniel Turull <[email protected]>
Cc: [email protected]; [email protected]; Tim Orling <[email protected]>; Richard Purdie <[email protected]>; Ross Burton <[email protected]>
Subject: Re: [PATCH v4] devtool: upgrade: extract changelog between versions

Quoted 3 lines · On Mon, 18 May 2026 at 15:59, <[email protected]> wrote:

Output is written to workspace/changelogs/<pn>.txt and cleaned up on devtool reset. This allows AUH and other tools to pick up the changelog without implementing their own extraction logic.

Quoted 8 lines

The patch review pointed out that this has been tested on a small handful of recipes, but before the change gets accepted, we really need to perform an industrial-strength set of upgrades with it, e.g. a typical AUH run on all of oe-core (possibly rolled back in time to include even more recipes), and then shake out all corner cases, unexpected fails, and pathological changelogs. I was going to make that run, then I realized that the companion patch for AUH has not yet been posted. Can you make that please?

Yes, I can do that. I was waiting to submit the AUH patch after this been merged, but I'll send it and put a note that depends on this patch to get in.

I will take a 1 year old oe-core and run AUH skipping compilation to be faster. I'll push the commits as is into a contrib branch and analyse them for corner cases

Cheers,
Daniel

Quoted 1 line

Alex

#3 lore raw
Quoted 3 lines · Daniel Turull

Yes, I can do that. I was waiting to submit the AUH patch after this been merged, but I'll send it and put a note that depends on this patch to get in.

I will take a 1 year old oe-core and run AUH skipping compilation to be faster. I'll push the commits as is into a contrib branch and analyse them for corner cases

If 'devtool upgrade' fails to rebase the patches (or otherwise fails), will it still be able to extract the changelogs? If you go back 1 year in time, the much larger version delta for most recipes will certainly cause an increase in such failures, so I would probably do an update on current master, and maybe master as of two months ago as well.

Alex

#4 lore raw
Quoted 14 lines

-----Original Message-----
From: Alexander Kanavin <[email protected]>
Sent: Thursday, 21 May 2026 12:31
To: Daniel Turull <[email protected]>
Cc: [email protected]; [email protected]; Tim Orling <[email protected]>; Richard Purdie <[email protected]>; Ross Burton <[email protected]>
Subject: Re: [PATCH v4] devtool: upgrade: extract changelog between versions

Quoted 1 line · Daniel Turull

Yes, I can do that. I was waiting to submit the AUH patch after this been

Quoted 8 lines

merged, but I'll send it and put a note that depends on this patch to get in.

Quoted 4 lines

I will take a 1 year old oe-core and run AUH skipping compilation to be faster. I'll push the commits as is into a contrib branch and analyse them for corner cases

If 'devtool upgrade' fails to rebase the patches (or otherwise fails), will it still be able to extract the changelogs?

It will still run and extract them. The extract new sources is in a try/Exception block.

Quoted 4 lines

If you go back 1 year in time, the much larger
version delta for most recipes will certainly cause an increase in such failures, so I would probably do an update on current master, and maybe master as of two months ago as well.

Ok. One with current master and one with two months old master.

Cheers
Daniel

A message that is not in the archive (sent off-list or not yet received): [email protected]

#6 lore raw

I hope I can send the patches after running all the tests on Monday. I found some small issues and I need to rerun AUH for 216 packages.

I also have the AUH patches ready

Best regards,
Daniel

Quoted 50 lines

-----Original Message-----
From: [email protected] <openembedded- [email protected]> On Behalf Of Daniel Turull via lists.openembedded.org
Sent: Thursday, 21 May 2026 12:47
To: Alexander Kanavin <[email protected]>
Cc: [email protected]; [email protected]; Tim Orling <[email protected]>; Richard Purdie <[email protected]>; Ross Burton <[email protected]>
Subject: Re: [OE-core] [PATCH v4] devtool: upgrade: extract changelog between versions

Quoted 12 lines

-----Original Message-----
From: Alexander Kanavin <[email protected]>
Sent: Thursday, 21 May 2026 12:31
To: Daniel Turull <[email protected]>
Cc: [email protected]; [email protected]; Tim Orling <[email protected]>; Richard Purdie <[email protected]>; Ross Burton <[email protected]>
Subject: Re: [PATCH v4] devtool: upgrade: extract changelog between versions

On Thu, 21 May 2026 at 12:25, Daniel Turull <[email protected]>

Quoted 11 lines · wrote:
Quoted 3 lines

Yes, I can do that. I was waiting to submit the AUH patch after this been

merged, but I'll send it and put a note that depends on this patch to get in.

Quoted 4 lines

I will take a 1 year old oe-core and run AUH skipping compilation to be faster. I'll push the commits as is into a contrib branch and analyse them for corner cases

If 'devtool upgrade' fails to rebase the patches (or otherwise fails), will it still be able to extract the changelogs?

It will still run and extract them. The extract new sources is in a try/Exception block.

Quoted 4 lines

If you go back 1 year in time, the much larger version delta for most recipes will certainly cause an increase in such failures, so I would probably do an update on current master, and maybe master as of two months ago as well.

Ok. One with current master and one with two months old master.

Cheers
Daniel