devtool: upgrade: extract changelog between versions
PATCH v4 1/1 devtool: upgrade: extract changelog between versions
From: Daniel Turull <[email protected]>
Automatically extract changelog information when upgrading a recipe. Uses the devtool-base tags created during upgrade to diff known changelog files (NEWS, ChangeLog, CHANGES, etc.) between the old and new versions. For git-based sources, falls back to git log --oneline if no changelog file changed.
Also match per-version release notes files like git's Documentation/RelNotes/2.53.0.adoc or mesa's docs/relnotes/26.0.3.rst.
These are detected by matching filenames that contain a version number with a documentation extension (.txt, .md, .rst, .adoc). Their full content is read via git show since they are new files rather than modifications to an existing changelog.
Collapse runs of multiple blank lines into a single blank line and strip leading/trailing whitespace from the extracted changelog. This produces cleaner output that is suitable for inclusion in commit messages by tools like AUH.
Output is written to workspace/changelogs/<pn>.txt and cleaned up on devtool reset. This allows AUH and other tools to pick up the changelog without implementing their own extraction logic.
Add test cases for existing test recipes
AI-generated: Uses kiro with claude-opus-4.6 model
---
v2:
- squash v1 patches into one commit and keep commit messages
- Integrate test cases with exiting devtool selftests
v3:
- do not silently skip the test if changelog is missing
v4:
- fix typo in commit message and fix AI-generated tag.
--- .../python3-guessing-game_git.bb.changelog | 3 + .../devtool-upgrade-test1_1.5.3.bb.changelog | 24 +++++ .../devtool-upgrade-test2_git.bb.changelog | 3 + .../devtool-upgrade-test3_1.5.3.bb.changelog | 24 +++++ .../devtool-upgrade-test4_1.5.3.bb.changelog | 24 +++++ .../devtool-upgrade-test5_git.bb.changelog | 3 + meta/lib/oeqa/selftest/cases/devtool.py | 27 ++++++ scripts/lib/devtool/standard.py | 8 ++ scripts/lib/devtool/upgrade.py | 95 +++++++++++++++++++ 9 files changed, 211 insertions(+) create mode 100644 meta-selftest/recipes-devtools/python/python3-guessing-game_git.bb.changelog create mode 100644 meta-selftest/recipes-test/devtool/devtool-upgrade-test1_1.5.3.bb.changelog create mode 100644 meta-selftest/recipes-test/devtool/devtool-upgrade-test2_git.bb.changelog create mode 100644 meta-selftest/recipes-test/devtool/devtool-upgrade-test3_1.5.3.bb.changelog create mode 100644 meta-selftest/recipes-test/devtool/devtool-upgrade-test4_1.5.3.bb.changelog create mode 100644 meta-selftest/recipes-test/devtool/devtool-upgrade-test5_git.bb.changelog
meta-selftest/recipes-devtools/python/python3-guessing-game_git.bb.changelog new file +3 -0
@@ -0,0 +1,3 @@ +Changelog for python3-guessing-game: 0.1.0 -> 0.2.0 + +40cf004 Sync with maturin tutorial source
meta-selftest/recipes-test/devtool/devtool-upgrade-test1_1.5.3.bb.changelog new file +24 -0
@@ -0,0 +1,24 @@ +Changelog for devtool-upgrade-test1: 1.5.3 -> 1.6.0 + +1.6.0 - 15 March 2015 + - fix lstat64 support when unavailable - separate patches supplied by + Ganael Laplanche and Peter Korsgaard + - (#1506) new option "-D" / "--delay-start" to only show bar after N + seconds (Damon Harper) + - new option "--fineta" / "-I" to show ETA as time of day rather than time + remaining - patch supplied by Erkki Seppälä (r147) + - (#1509) change ETA (--eta / -e) so that days are given if the hours + remaining are 24 or more (Jacek Wielemborek) + - (#1499) repeat read and write attempts on partial buffer fill/empty to + work around post-signal transfer rate drop reported by Ralf Ramsauer + - (#1507) do not try to calculate total size in line mode, due to bug + reported by Jacek Wielemborek and Michiel Van Herwegen + - cleanup: removed defunct RATS comments and unnecessary copyright notices + - clean up displayed lines when using --watchfd PID, when PID exits + - output errors on a new line to avoid overwriting transfer bar + +1.5.7 - 26 August 2014 + - show KiB instead of incorrect kiB (Debian bug #706175) + - (#1284) do not gzip man page, for non-Linux OSes (Bob Friesenhahn) + - work around "awk" bug in tests/016-numeric-timer in decimal "," locales + - fix "make rpm" and "make srpm", extend "make release" to sign releases
meta-selftest/recipes-test/devtool/devtool-upgrade-test2_git.bb.changelog new file +3 -0
@@ -0,0 +1,3 @@ +Changelog for devtool-upgrade-test2: 0.1+git -> 0.1+git + +6cc6077 dbus-wait.c: Fix typo
meta-selftest/recipes-test/devtool/devtool-upgrade-test3_1.5.3.bb.changelog new file +24 -0
@@ -0,0 +1,24 @@ +Changelog for devtool-upgrade-test3: 1.5.3 -> 1.6.0 + +1.6.0 - 15 March 2015 + - fix lstat64 support when unavailable - separate patches supplied by + Ganael Laplanche and Peter Korsgaard + - (#1506) new option "-D" / "--delay-start" to only show bar after N + seconds (Damon Harper) + - new option "--fineta" / "-I" to show ETA as time of day rather than time + remaining - patch supplied by Erkki Seppälä (r147) + - (#1509) change ETA (--eta / -e) so that days are given if the hours + remaining are 24 or more (Jacek Wielemborek) + - (#1499) repeat read and write attempts on partial buffer fill/empty to + work around post-signal transfer rate drop reported by Ralf Ramsauer + - (#1507) do not try to calculate total size in line mode, due to bug + reported by Jacek Wielemborek and Michiel Van Herwegen + - cleanup: removed defunct RATS comments and unnecessary copyright notices + - clean up displayed lines when using --watchfd PID, when PID exits + - output errors on a new line to avoid overwriting transfer bar + +1.5.7 - 26 August 2014 + - show KiB instead of incorrect kiB (Debian bug #706175) + - (#1284) do not gzip man page, for non-Linux OSes (Bob Friesenhahn) + - work around "awk" bug in tests/016-numeric-timer in decimal "," locales + - fix "make rpm" and "make srpm", extend "make release" to sign releases
meta-selftest/recipes-test/devtool/devtool-upgrade-test4_1.5.3.bb.changelog new file +24 -0
@@ -0,0 +1,24 @@ +Changelog for devtool-upgrade-test4: 1.5.3 -> 1.6.0 + +1.6.0 - 15 March 2015 + - fix lstat64 support when unavailable - separate patches supplied by + Ganael Laplanche and Peter Korsgaard + - (#1506) new option "-D" / "--delay-start" to only show bar after N + seconds (Damon Harper) + - new option "--fineta" / "-I" to show ETA as time of day rather than time + remaining - patch supplied by Erkki Seppälä (r147) + - (#1509) change ETA (--eta / -e) so that days are given if the hours + remaining are 24 or more (Jacek Wielemborek) + - (#1499) repeat read and write attempts on partial buffer fill/empty to + work around post-signal transfer rate drop reported by Ralf Ramsauer + - (#1507) do not try to calculate total size in line mode, due to bug + reported by Jacek Wielemborek and Michiel Van Herwegen + - cleanup: removed defunct RATS comments and unnecessary copyright notices + - clean up displayed lines when using --watchfd PID, when PID exits + - output errors on a new line to avoid overwriting transfer bar + +1.5.7 - 26 August 2014 + - show KiB instead of incorrect kiB (Debian bug #706175) + - (#1284) do not gzip man page, for non-Linux OSes (Bob Friesenhahn) + - work around "awk" bug in tests/016-numeric-timer in decimal "," locales + - fix "make rpm" and "make srpm", extend "make release" to sign releases
meta-selftest/recipes-test/devtool/devtool-upgrade-test5_git.bb.changelog new file +3 -0
@@ -0,0 +1,3 @@ +Changelog for devtool-upgrade-test5: 0.1+git -> 0.1+git + +0a60d6a Add dummy commit on tip for testing
meta/lib/oeqa/selftest/cases/devtool.py +27 -0
@@ -1944,6 +1944,22 @@ class DevtoolUpgradeTests(DevtoolBase): except: self.skip("Git user.name and user.email must be set") + def _check_changelog(self, recipe, oldrecipefile): + """Compare extracted changelog against reference data.""" + changelog_ref = oldrecipefile + '.changelog' + self.assertExists(changelog_ref, 'Changelog reference file must exist for %s' % recipe) + changelog_file = os.path.join(self.workspacedir, 'changelogs', '%s.txt' % recipe) + with open(changelog_ref, 'r') as f: + expected = f.read() + if not expected: + self.assertNotExists(changelog_file, + 'Changelog file should not exist when reference is empty') + else: + self.assertExists(changelog_file, 'Changelog file should exist after upgrade') + with open(changelog_file, 'r') as f: + actual = f.read() + self.assertEqual(expected, actual) + def test_devtool_upgrade(self): # Check preconditions self.assertTrue(not os.path.exists(self.workspacedir), 'This test cannot be run with a workspace directory under the build directory')
@@ -1982,6 +1998,8 @@ class DevtoolUpgradeTests(DevtoolBase): with open(newrecipefile, 'r') as f: newlines = f.readlines() self.assertEqual(desiredlines, newlines) + # Check changelog + self._check_changelog(recipe, oldrecipefile) # Check devtool reset recipe result = runCmd('devtool reset %s -n' % recipe) result = runCmd('devtool status')
@@ -2016,11 +2034,14 @@ class DevtoolUpgradeTests(DevtoolBase): with open(newrecipefile, 'r') as f: newlines = f.readlines() self.assertEqual(desiredlines, newlines) + # Check changelog + self._check_changelog(recipe, oldrecipefile) # Check devtool reset recipe result = runCmd('devtool reset %s -n' % recipe) result = runCmd('devtool status') self.assertNotIn(recipe, result.output) self.assertNotExists(os.path.join(self.workspacedir, 'recipes', recipe), 'Recipe directory should not exist after resetting') + self.assertNotExists(os.path.join(self.workspacedir, 'changelogs', '%s.txt' % recipe), 'Changelog file should be removed after reset') def test_devtool_upgrade_git(self): self._test_devtool_upgrade_git_by_recipe('devtool-upgrade-test2', '6cc6077a36fe2648a5f993fe7c16c9632f946517')
@@ -2051,6 +2072,8 @@ class DevtoolUpgradeTests(DevtoolBase): with open(newrecipefile, 'r') as f: newlines = f.readlines() self.assertEqual(desiredlines, newlines) + # Check changelog + self._check_changelog(recipe, oldrecipefile) def test_devtool_upgrade_all_checksums(self): # Check preconditions
@@ -2075,6 +2098,8 @@ class DevtoolUpgradeTests(DevtoolBase): with open(newrecipefile, 'r') as f: newlines = f.readlines() self.assertEqual(desiredlines, newlines) + # Check changelog + self._check_changelog(recipe, oldrecipefile) def test_devtool_upgrade_recipe_upgrade_extra_tasks(self): # Check preconditions
@@ -2116,6 +2141,8 @@ class DevtoolUpgradeTests(DevtoolBase): with open(newcratesincfile, 'r') as f: newlines = f.readlines() self.assertEqual(desiredlines, newlines) + # Check changelog + self._check_changelog(recipe, oldrecipefile) # Check devtool reset recipe result = runCmd('devtool reset %s -n' % recipe) result = runCmd('devtool status')
scripts/lib/devtool/standard.py +8 -0
@@ -2046,6 +2046,14 @@ def _reset(recipes, no_clean, remove_work, config, basepath, workspace): clean_preferred_provider(pn, config.workspace_path) + # Clean up changelog if present + changelog_file = os.path.join(config.workspace_path, 'changelogs', '%s.txt' % pn) + if os.path.exists(changelog_file): + os.remove(changelog_file) + changelog_dir = os.path.dirname(changelog_file) + if not os.listdir(changelog_dir): + os.rmdir(changelog_dir) + def reset(args, config, basepath, workspace): """Entry point for the devtool 'reset' subcommand"""
scripts/lib/devtool/upgrade.py +95 -0
119 lines not on this page · raw message
Quoted 3 lines · On Mon, 18 May 2026 at 15:59, <[email protected]> wrote:
Output is written to workspace/changelogs/<pn>.txt and cleaned up on devtool reset. This allows AUH and other tools to pick up the changelog without implementing their own extraction logic.
The patch review pointed out that this has been tested on a small handful of recipes, but before the change gets accepted, we really need to perform an industrial-strength set of upgrades with it, e.g. a typical AUH run on all of oe-core (possibly rolled back in time to include even more recipes), and then shake out all corner cases, unexpected fails, and pathological changelogs. I was going to make that run, then I realized that the companion patch for AUH has not yet been posted. Can you make that please?
Alex
Quoted 16 lines
-----Original Message-----
From: Alexander Kanavin <[email protected]>
Sent: Thursday, 21 May 2026 12:18
To: Daniel Turull <[email protected]>
Cc: [email protected]; [email protected]; Tim Orling <[email protected]>; Richard Purdie <[email protected]>; Ross Burton <[email protected]>
Subject: Re: [PATCH v4] devtool: upgrade: extract changelog between versionsQuoted 3 lines · On Mon, 18 May 2026 at 15:59, <[email protected]> wrote:
Output is written to workspace/changelogs/<pn>.txt and cleaned up on devtool reset. This allows AUH and other tools to pick up the changelog without implementing their own extraction logic.
Quoted 8 lines
The patch review pointed out that this has been tested on a small handful of recipes, but before the change gets accepted, we really need to perform an industrial-strength set of upgrades with it, e.g. a typical AUH run on all of oe-core (possibly rolled back in time to include even more recipes), and then shake out all corner cases, unexpected fails, and pathological changelogs. I was going to make that run, then I realized that the companion patch for AUH has not yet been posted. Can you make that please?
Yes, I can do that. I was waiting to submit the AUH patch after this been merged, but I'll send it and put a note that depends on this patch to get in.
I will take a 1 year old oe-core and run AUH skipping compilation to be faster. I'll push the commits as is into a contrib branch and analyse them for corner cases
Cheers,
Daniel
Quoted 1 line
Alex
Quoted 3 lines · Daniel Turull
Yes, I can do that. I was waiting to submit the AUH patch after this been merged, but I'll send it and put a note that depends on this patch to get in.
I will take a 1 year old oe-core and run AUH skipping compilation to be faster. I'll push the commits as is into a contrib branch and analyse them for corner cases
If 'devtool upgrade' fails to rebase the patches (or otherwise fails), will it still be able to extract the changelogs? If you go back 1 year in time, the much larger version delta for most recipes will certainly cause an increase in such failures, so I would probably do an update on current master, and maybe master as of two months ago as well.
Alex
Quoted 14 lines
-----Original Message-----
From: Alexander Kanavin <[email protected]>
Sent: Thursday, 21 May 2026 12:31
To: Daniel Turull <[email protected]>
Cc: [email protected]; [email protected]; Tim Orling <[email protected]>; Richard Purdie <[email protected]>; Ross Burton <[email protected]>
Subject: Re: [PATCH v4] devtool: upgrade: extract changelog between versionsQuoted 1 line · Daniel Turull
Yes, I can do that. I was waiting to submit the AUH patch after this been
Quoted 8 lines
merged, but I'll send it and put a note that depends on this patch to get in.
Quoted 4 lines
I will take a 1 year old oe-core and run AUH skipping compilation to be faster. I'll push the commits as is into a contrib branch and analyse them for corner cases
If 'devtool upgrade' fails to rebase the patches (or otherwise fails), will it still be able to extract the changelogs?
It will still run and extract them. The extract new sources is in a try/Exception block.
Quoted 4 lines
If you go back 1 year in time, the much larger
version delta for most recipes will certainly cause an increase in such failures, so I would probably do an update on current master, and maybe master as of two months ago as well.
Ok. One with current master and one with two months old master.
Cheers
Daniel
A message that is not in the archive (sent off-list or not yet received): [email protected]
I hope I can send the patches after running all the tests on Monday. I found some small issues and I need to rerun AUH for 216 packages.
I also have the AUH patches ready
Best regards,
Daniel
Quoted 50 lines
-----Original Message-----
From: [email protected] <openembedded- [email protected]> On Behalf Of Daniel Turull via lists.openembedded.org
Sent: Thursday, 21 May 2026 12:47
To: Alexander Kanavin <[email protected]>
Cc: [email protected]; [email protected]; Tim Orling <[email protected]>; Richard Purdie <[email protected]>; Ross Burton <[email protected]>
Subject: Re: [OE-core] [PATCH v4] devtool: upgrade: extract changelog between versionsQuoted 12 lines
-----Original Message-----
From: Alexander Kanavin <[email protected]>
Sent: Thursday, 21 May 2026 12:31
To: Daniel Turull <[email protected]>
Cc: [email protected]; [email protected]; Tim Orling <[email protected]>; Richard Purdie <[email protected]>; Ross Burton <[email protected]>
Subject: Re: [PATCH v4] devtool: upgrade: extract changelog between versionsOn Thu, 21 May 2026 at 12:25, Daniel Turull <[email protected]>
Quoted 11 lines · wrote:
Quoted 3 lines
Yes, I can do that. I was waiting to submit the AUH patch after this been
merged, but I'll send it and put a note that depends on this patch to get in.
Quoted 4 lines
I will take a 1 year old oe-core and run AUH skipping compilation to be faster. I'll push the commits as is into a contrib branch and analyse them for corner cases
If 'devtool upgrade' fails to rebase the patches (or otherwise fails), will it still be able to extract the changelogs?
It will still run and extract them. The extract new sources is in a try/Exception block.
Quoted 4 lines
If you go back 1 year in time, the much larger version delta for most recipes will certainly cause an increase in such failures, so I would probably do an update on current master, and maybe master as of two months ago as well.
Ok. One with current master and one with two months old master.
Cheers
Daniel