rel-html: one fix and update for project renames
cover rel-html: one fix and update for project renames
From: "Luis R. Rodriguez" <[email protected]>
rel-html [0] is used to generate the HTML5 release pages for the backports project but given the input received it was designed to support any project. The backports project was renamed and the release structure was updated as well. This series addresse the project rename and adds support for multiple daily snapshot releases styles. While at it update the other projects this supports making releases for and kick out new pages for them as examples [1].
Note the TODO from Readme.md:
* Figure out how to automatically determine releases from git.
- If we have many stable releases how should we annotate this via git ?
It seems that the way to go is to require a config file for the project with the oldest stable release supported and then annotate eols. We do this right now, so rel-html would just need to be modified to infer newer releases.
* The Linux kernel now (as of 2013-03-10) has json file for releases:
https://www.kernel.org/releases.json
We need to do a few things then:
- Get other projects to use json for releases
- Add json intepreter support to rel-html
If projects don't use json releases files as the Linux kernel does then the current usage of HTMLParser would allow us to parse / infer releases for us.
* See if we can copy the EOL release into an eol/ directory and moving forward instead of parsing the tags use the directory name to automatically determine other release attributes. This is only relevant for the Linux kernel right now.
[0] git://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/rel-html.git
[1] http://drvbp1.linux-foundation.org/~mcgrof/rel-html/backports/
Luis R. Rodriguez (8):
rel_html: add support for project aliases
rel-html: rename compat-drivers.cfg to backports.cfg
rel-html: really listen to ignore_changelogs
rel-html: add RFC3339 / short style next release types support
rel-html: fix next release URLs
rel-html: add v3.10 as supported for Linux
rel-html: update iw release
rel-html: update hostapd releases
projects/backports.cfg | 37 +++++++++++ projects/compat-drivers.cfg | 34 ---------- projects/hostapd.cfg | 1 + projects/iw.cfg | 1 + rel-html.cfg | 1 + rel-html.py | 149 +++++++++++++++++++++++++++---------------- 6 files changed, 134 insertions(+), 89 deletions(-) create mode 100644 projects/backports.cfg delete mode 100644 projects/compat-drivers.cfg
Signature · 1 line
1.7.10.4
PATCH 1/8 rel_html: add support for project aliases
From: "Luis R. Rodriguez" <[email protected]>
If a project changes its name you may still want to print out the older project's releases as well. Add support for this by letting you specify project aliases, you can have however many project aliases as you need. An example of your foo.cfg
[project]rel_html_proj = backportsrel_html_proj_aliases = compat-drivers
Project versioning for figuring out which release is newer or older is independent of the project name and aliases, its all unified.
--- rel-html.py | 82 ++++++++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 58 insertions(+), 24 deletions(-)
rel-html.py +58 -24
@@ -274,6 +274,37 @@ class index_tarball_hunter(HTMLParser): eol_specs['PATCHLEVEL'] == rel_specs['PATCHLEVEL']): return True return False + def get_rel_match(self, value): + index_parser = self.index_parser + rel_match = dict(m = None, rel_name = "") + for rel_name in index_parser.rel_names: + m = re.match(r'' + rel_name + '-+' \ + "v*(?P<VERSION>\d+)\.+" \ + "(?P<PATCHLEVEL>\d+)\.*" \ + "(?P<SUBLEVEL>\w*)[.-]*" \ + "(?P<EXTRAVERSION>\w*)[-]*" \ + "(?P<RELMOD_UPDATE>\d*)[-]*" \ + "(?P<RELMOD_TYPE>[usnpc]*)", \ + value) + if (m): + rel_match['m'] = m + rel_match['rel_name'] = rel_name + return rel_match + return rel_match + def get_rel_match_next(self, value): + index_parser = self.index_parser + rel_match = dict(m = None, rel_name = "") + for rel_name in index_parser.rel_names: + m = re.match(r'' + rel_name + '+' \ + + '\-(?P<DATE_VERSION>' + index_parser.next_rel_date + '+)' \ + + '\-*(?P<EXTRAVERSION>\d*)' \ + + '\-*(?P<RELMOD>\w*)', + value) + if (m): + rel_match['m'] = m + rel_match['rel_name'] = rel_name + return rel_match + return rel_match def update_latest_tarball_stable(self, value): index_parser = self.index_parser if (self.release not in value):
@@ -283,18 +314,11 @@ class index_tarball_hunter(HTMLParser): if (index_parser.release_extension not in value): return - m = re.match(r'' + index_parser.rel_html_proj + '-+' \ - "v*(?P<VERSION>\d+)\.+" \ - "(?P<PATCHLEVEL>\d+)\.*" \ - "(?P<SUBLEVEL>\w*)[.-]*" \ - "(?P<EXTRAVERSION>\w*)[-]*" \ - "(?P<RELMOD_UPDATE>\d*)[-]*" \ - "(?P<RELMOD_TYPE>[usnpc]*)", \ - value) - if (not m): + rel_match = self.get_rel_match(value) + if (not rel_match['m']): return - rel_specifics = m.groupdict() + rel_specifics = rel_match['m'].groupdict() supported = True if (self.is_rel_eol(rel_specifics)):
@@ -303,7 +327,7 @@ class index_tarball_hunter(HTMLParser): p = re.compile(index_parser.release_extension + '$') rel_name = p.sub("", value) - ver = rel_name.lstrip(index_parser.rel_html_proj + '-') + ver = rel_name.lstrip(rel_match['rel_name'] + '-') p = re.compile('-[usnpc]*$') short_ver = p.sub("", ver)
@@ -348,18 +372,13 @@ class index_tarball_hunter(HTMLParser): self.tarball_add_stable(tar) def update_latest_tarball_next(self, value): index_parser = self.index_parser - m = re.match(r'' + index_parser.rel_html_proj + '+' \ - + '\-(?P<DATE_VERSION>' + index_parser.next_rel_date + '+)' \ - + '\-*(?P<EXTRAVERSION>\d*)' \ - + '\-*(?P<RELMOD>\w*)', - value) - - if (not m): + rel_match = self.get_rel_match_next(value) + if (not rel_match['m']): return - rel_specifics = m.groupdict() + rel_specifics = rel_match['m'].groupdict() - rel_name_next = index_parser.rel_html_proj + '-' + rel_specifics['DATE_VERSION'] + rel_name_next = rel_match['rel_name'] + '-' + rel_specifics['DATE_VERSION'] next_version = rel_specifics['DATE_VERSION'] if (rel_specifics['EXTRAVERSION'] != ''):
@@ -451,10 +470,10 @@ class index_rel_inferrer(HTMLParser): self.close() def handle_decl(self, decl): pass - def revise_inference(self, rel, value): + def revise_inference(self, rel, value, rel_name): index_parser = self.index_parser - value = value.lstrip(index_parser.rel_html_proj + "-") + value = value.lstrip(rel_name + "-") p = re.compile(index_parser.release_extension + '$') value = p.sub("", value)
@@ -500,7 +519,8 @@ class index_rel_inferrer(HTMLParser): if tag != 'a': return for name, value in attributes: if name != 'href': return - if (index_parser.rel_html_proj not in value): + rel_name = index_parser.search_rel_name(value) + if (not rel_name): return if (index_parser.release_extension not in value): return
@@ -509,7 +529,7 @@ class index_rel_inferrer(HTMLParser): for rel in index_parser.inferred_releases: if (rel.get('base') not in value): continue - self.revise_inference(rel, value) + self.revise_inference(rel, value, rel_name) def handle_endtag(self, tag): pass def handle_data(self, data):
@@ -618,6 +638,13 @@ class index_parser(HTMLParser): self.config.read(config_file) self.rel_html_proj = self.config.get("project", "rel_html_proj") + if (self.config.has_option("project", "rel_html_proj_aliases")): + self.rel_html_proj_aliases = self.config.get("project", "rel_html_proj_aliases").split() + else: + self.rel_html_proj_aliases = list() + + self.rel_names = self.rel_html_proj_aliases + self.rel_names.insert(0, self.rel_html_proj) self.inferred_releases = []
@@ -692,6 +719,7 @@ class index_parser(HTMLParser): self.changelog = '' self.signed_changelog = False + def get_stable_ext_urls(self, url): url_parser = stable_url_parser(self, url) try:
@@ -701,6 +729,11 @@ class index_parser(HTMLParser): self.stable_urls = url_parser.stable_urls except urllib2.HTTPError, error: return + def search_rel_name(self, value): + for rel_name in self.rel_names: + if (rel_name in value): + return rel_name + return "" def search_stable_tarballs(self, ver, url): try: tarball_hunter = index_tarball_hunter(self, ver, url)
@@ -1133,6 +1166,7 @@ def try_rels(rels): print_rels_weight(col) def debug_rel_tests(): + try_rel_next("20130510-2-u") try_rel_next("2013-01-10-2-u") try_rel_next("20130110-2-u") try_rel_next("2013-03-07-u")
Signature · 1 line
1.7.10.4
PATCH 2/8 rel-html: rename compat-drivers.cfg to backports.cfg
In reply to Luis R. RodriguezFrom: "Luis R. Rodriguez" <[email protected]>
The compat-drivers project was renamed to backports. Rename the project file and also update the config file to add the v3.10 kernel as supported.
We also add the ignore_changelogs for this project given that we want to skip these moving forward and simply provide shortlogs upon releases and refer folks to use git for a full changelog.
--- projects/backports.cfg | 37 +++++++++++++++++++++++++++++++++++++ projects/compat-drivers.cfg | 34 ---------------------------------- 2 files changed, 37 insertions(+), 34 deletions(-) create mode 100644 projects/backports.cfg delete mode 100644 projects/compat-drivers.cfg
projects/backports.cfg new file +37 -0
@@ -0,0 +1,37 @@ +[project] +rel_html_proj = backports +rel_html_proj_aliases = compat-drivers +supported = + 3.9 + 3.8 + 3.7 + 3.10 +rel_html_url_releases = http://www.kernel.org/pub/linux/kernel/projects/backports/stable/ + http://www.kernel.org/pub/linux/kernel/projects/backports/2013/ +rel_license = GPLv2 +ignore_changelogs = True + +[html] +title = backports: Linux kernel backports + +nav_01_url = https://backports.wiki.kernel.org/ +nav_01_txt = Documentation + +# XXX: Note: two % added here to help with what seems to be a python ConfigParser.py bug. +# We should address this eventually. +nav_02_url = https://bugzilla.kernel.org/enter_bug.cgi?product=Backports%%20project +nav_02_txt = Bugs + +nav_03_url = https://backports.wiki.kernel.org/index.php/Mailing_list +nav_03_txt = Mailing List + +release_title = Latest stable releases +release_title_next = Latest development releases + +about_title = About backports +about = We provide drivers released on newer kernels backported for + usage on older kernels. Both daily snapshots based on + <a href="http://git.kernel.org/?p=linux/kernel/git/next/linux-next.git;a=summary">linux-next</a>, + and stable releases based on + <a href="http://git.kernel.org/?p=linux/kernel/git/stable/linux-stable.git;a=summary">Linux's stable releases</a> + are provided. The Linux drivers are <b>automatically backported</b>.
projects/compat-drivers.cfg deleted +0 -34
@@ -1,34 +0,0 @@ -[project] -rel_html_proj = compat-drivers -supported = - 3.9 - 3.8 - 3.7 -rel_html_url_releases = http://www.kernel.org/pub/linux/kernel/projects/backports/stable/ - http://www.kernel.org/pub/linux/kernel/projects/backports/2013/ -rel_license = GPLv2 - -[html] -title = compat-drivers: Linux kernel backports - -nav_01_url = https://backports.wiki.kernel.org/ -nav_01_txt = Documentation - -# XXX: Note: two % added here to help with what seems to be a python ConfigParser.py bug. -# We should address this eventually. -nav_02_url = https://bugzilla.kernel.org/enter_bug.cgi?product=Backports%%20project -nav_02_txt = Bugs - -nav_03_url = https://backports.wiki.kernel.org/index.php/Mailing_list -nav_03_txt = Mailing List - -release_title = Latest stable releases -release_title_next = Latest development releases - -about_title = About compat-drivers -about = We provide drivers released on newer kernels backported for - usage on older kernels. Both daily snapshots based on - <a href="http://git.kernel.org/?p=linux/kernel/git/next/linux-next.git;a=summary">linux-next</a>, - and stable releases based on - <a href="http://git.kernel.org/?p=linux/kernel/git/stable/linux-stable.git;a=summary">Linux's stable releases</a> - are provided. The Linux drivers are <b>automatically backported</b>.
Signature · 1 line
1.7.10.4
PATCH 3/8 rel-html: really listen to ignore_changelogs
In reply to Luis R. RodriguezFrom: "Luis R. Rodriguez" <[email protected]>
When a project was setting ignore_changelogs the changelog was still printed if and only if the changelog happened to be found by chance and it was also signed. In practice if a project wants to abandon changelogs this would then produce an output that would likely make users think the developers forgot to provide a changelog. Simply ignore changelogs even if they are found for older releases when ignore_changelogs is set.
--- rel-html.py | 18 ++---------------- 1 file changed, 2 insertions(+), 16 deletions(-)
rel-html.py +2 -16
@@ -1008,14 +1008,7 @@ class rel_html_gen(HTMLParser): sys.stdout.write('\t\t\t\t<td><a href="%s">%s</a></td>\n' % \ (r.get('changelog_url'), "ChangeLog")) else: - if (r.get('changelog_exists')): - if (r['signed_changelog_exists']): - sys.stdout.write('\t\t\t\t<td><a href="%s">%s</a></td>\n' % \ - (r.get('changelog_url'), "ChangeLog")) - else: - sys.stdout.write('\t\t\t\t<td></td>\n') - else: - sys.stdout.write('\t\t\t\t<td></td>\n') + sys.stdout.write('\t\t\t\t<td></td>\n') sys.stdout.write('\t\t\t\t</tr>') sys.stdout.write('\t\t\t</table>\n')
@@ -1053,14 +1046,7 @@ class rel_html_gen(HTMLParser): sys.stdout.write('\t\t\t\t<td><a href="%s">%s</a></td>\n' % \ (r.get('changelog_url'), "ChangeLog")) else: - if (r.get('changelog_exists')): - if (r['signed_changelog_exists']): - sys.stdout.write('\t\t\t\t<td><a href="%s">%s</a></td>\n' % \ - (r.get('changelog_url'), "ChangeLog")) - else: - sys.stdout.write('\t\t\t\t<td></td>\n') - else: - sys.stdout.write('\t\t\t\t<td></td>\n') + sys.stdout.write('\t\t\t\t<td></td>\n') sys.stdout.write('\t\t\t\t</tr>') sys.stdout.write('\t\t\t</table>\n')
Signature · 1 line
1.7.10.4
PATCH 4/8 rel-html: add RFC3339 / short style next release types support
In reply to Luis R. RodriguezFrom: "Luis R. Rodriguez" <[email protected]>
A project might be using RFC3339 (2013-05-02) style release names for their latest releases and if they switch to the smaller style (20130502) we need to treat both release styles as possible. Add support for this.
Your project need not do anything if you switch, the tarball hunter will everything for for you.
--- rel-html.py | 65 ++++++++++++++++++++++++++++++++++++++--------------------- 1 file changed, 42 insertions(+), 23 deletions(-)
rel-html.py +42 -23
@@ -255,9 +255,10 @@ class index_tarball_hunter(HTMLParser): for t_old in self.tarballs: s_old = t_old.get('specifics') idx = self.tarballs.index(t_old) - if (index_parser.next_rel_date in t_old.get('rel')): - self.tarballs.insert(idx-1, t_new) - return + for next_date in index_parser.next_rel_dates: + if (next_date in t_old.get('rel')): + self.tarballs.insert(idx-1, t_new) + return self.tarballs.append(t_new) def is_rel_eol(self, rel_specs): index_parser = self.index_parser
@@ -295,19 +296,26 @@ class index_tarball_hunter(HTMLParser): index_parser = self.index_parser rel_match = dict(m = None, rel_name = "") for rel_name in index_parser.rel_names: - m = re.match(r'' + rel_name + '+' \ - + '\-(?P<DATE_VERSION>' + index_parser.next_rel_date + '+)' \ - + '\-*(?P<EXTRAVERSION>\d*)' \ - + '\-*(?P<RELMOD>\w*)', - value) - if (m): - rel_match['m'] = m - rel_match['rel_name'] = rel_name - return rel_match + for next_date in index_parser.next_rel_dates: + m = re.match(r'' + rel_name + '+' \ + + '\-(?P<DATE_VERSION>' + next_date + '+)' \ + + '\-*(?P<EXTRAVERSION>\d*)' \ + + '\-*(?P<RELMOD>\w*)', + value) + if (m): + rel_match['m'] = m + rel_match['rel_name'] = rel_name + return rel_match return rel_match + def get_rel_name(self, value): + for rel in self.releases: + if (rel in value): + return rel + return "" def update_latest_tarball_stable(self, value): index_parser = self.index_parser - if (self.release not in value): + release = self.get_rel_name(value) + if (not release): return if ('tar.sign' in value): return
@@ -433,18 +441,24 @@ class index_tarball_hunter(HTMLParser): index_parser = self.index_parser for tar in self.tarballs: index_parser.rel_html_rels.append(tar) + def is_next_rel(self, value): + index_parser = self.index_parser + for next_date in index_parser.next_rel_dates: + if (next_date != '' and + next_date in value and + index_parser.release_extension in value): + return True + return False def handle_starttag(self, tag, attributes): "Process a tags and its 'attributes'." index_parser = self.index_parser if tag != 'a': pass for name, value in attributes: if name != 'href': pass - if (self.release not in value): + release = self.get_rel_name(value) + if (release not in value): pass - - if (index_parser.next_rel_date != '' and - index_parser.next_rel_date in value and - index_parser.release_extension in value): + if (self.is_next_rel(value)): self.update_latest_tarball_next(value) pass
@@ -455,11 +469,11 @@ class index_tarball_hunter(HTMLParser): pass def handle_comment(self, data): pass - def __init__(self, index_parser, release, url): + def __init__(self, index_parser, releases, url): HTMLParser.__init__(self) self.index_parser = index_parser self.base_url = url.rstrip("/") - self.release = release + self.releases = releases self.tarballs = [] class index_rel_inferrer(HTMLParser):
@@ -695,6 +709,8 @@ class index_parser(HTMLParser): self.next_rel_month = 0 self.next_rel_url = '' self.next_rel_date = '' + self.next_rel_date_rfc3339 = '' + self.next_rel_dates = list() self.rel_license = self.config.get("project", "rel_license") self.html_title = self.config.get("html", "title")
@@ -736,7 +752,7 @@ class index_parser(HTMLParser): return "" def search_stable_tarballs(self, ver, url): try: - tarball_hunter = index_tarball_hunter(self, ver, url) + tarball_hunter = index_tarball_hunter(self, [ver], url) f = urllib2.urlopen(url) html = f.read()
@@ -775,11 +791,14 @@ class index_parser(HTMLParser): self.next_rel_day = self.__get_next_rel_page(url + self.next_rel_month) self.next_rel_url = url + self.next_rel_month + '/' + self.next_rel_day # XXX: automatically look for the largest year - self.next_rel_date = '2013' + '-' + self.next_rel_month + '-' + self.next_rel_day + self.next_rel_date_rfc3339 = '2013' + '-' + self.next_rel_month + '-' + self.next_rel_day + self.next_rel_date = self.next_rel_date_rfc3339.replace("-", "") + self.next_rel_dates.append(self.next_rel_date_rfc3339) + self.next_rel_dates.append(self.next_rel_date) def evaluate_next_url(self): try: tarball_hunter = index_tarball_hunter(self, - self.next_rel_date, + self.next_rel_dates, self.next_rel_url) f = urllib2.urlopen(self.next_rel_url)
Signature · 1 line
1.7.10.4
PATCH 5/8 rel-html: fix next release URLs
In reply to Luis R. RodriguezFrom: "Luis R. Rodriguez" <[email protected]>
The tarball URL was completely missing.
--- rel-html.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
rel-html.py +1 -1
@@ -408,7 +408,7 @@ class index_tarball_hunter(HTMLParser): tar_next = dict(version=next_version, weight = w, rel = rel_name_next, - url = '', + url = self.base_url + '/' + tar_next, specifics = rel_specifics, base_url = self.base_url, base_url_validated = False,
Signature · 1 line
1.7.10.4
PATCH 6/8 rel-html: add v3.10 as supported for Linux
In reply to Luis R. RodriguezFrom: "Luis R. Rodriguez" <[email protected]>
Linus has released v3.10, add it to the linux project.
--- rel-html.cfg | 1 + 1 file changed, 1 insertion(+)
rel-html.cfg +1 -0
@@ -1,6 +1,7 @@ [project] rel_html_proj = linux supported = + 3.10 3.9 3.8 3.7
Signature · 1 line
1.7.10.4
PATCH 7/8 rel-html: update iw release
In reply to Luis R. RodriguezFrom: "Luis R. Rodriguez" <[email protected]>
Johannes released the v3.10 release of iw so update the config file for the project to annotate it as supported. Note that there was no v3.9 relase for iw.
Johannes is now following the releases of the kernel for iw release.
--- projects/iw.cfg | 1 + 1 file changed, 1 insertion(+)
projects/iw.cfg +1 -0
@@ -2,6 +2,7 @@ rel_html_proj = iw supported = 3.8 + 3.10 rel_html_testing_ver = latest rel_html_url_releases = https://www.kernel.org/pub/software/network/iw/ rel_license = ISC
Signature · 1 line
1.7.10.4
PATCH 8/8 rel-html: update hostapd releases
In reply to Luis R. RodriguezFrom: "Luis R. Rodriguez" <[email protected]>
Jouni released v2.0 of hostapd a while ago in January, update the project file for it.
--- projects/hostapd.cfg | 1 + 1 file changed, 1 insertion(+)
projects/hostapd.cfg +1 -0
@@ -2,6 +2,7 @@ rel_html_proj = hostapd supported = 1.1 + 2.0 rel_html_url_releases = http://hostap.epitest.fi/releases/ rel_license = BSD ignore_signatures = True
Signature · 1 line
1.7.10.4