Skip to contentSkip to conversation

remotes: automatically add remotes from configuration file

PATCH New Daniel Sangorrin 2 participants 5 messages Jun 17, 2019 – Jun 18, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 remotes: automatically add remotes from configuration file

Currently the user is required to create its own remotes by hand. This should not be necessary, because the information is already collected in conf/remotes.yml. For that reason, if we detect that any remote has not been added to the local repo then we will add those.

---
 README.md                    | 5 +++--
 conf/remotes.yml             | 9 ++++++---
 scripts/import_stable.py     | 9 +++++++++
 scripts/templates/issue.html | 7 ++++---
 4 files changed, 22 insertions(+), 8 deletions(-)
README.md +3 -2
index 4c5808f..dda94a8 100644
@@ -77,8 +77,9 @@ These files, if they exist, contain a mapping where the keys
 are default git remote names.  The values are also mappings,
 with the keys:
 
-* `commit_url_prefix`: URL prefix for browsing a commit on a
-  branch from this remote.
+* `git_repo_url`: URL of the remote repository.
+* `commit_url_suffix`: URL suffix that gets appended to `git_repo_url`
+  for browsing a commit on a branch from this remote.
 * `git_name`: (optional) The name actually used for this git
   remote, if it's different from the default.
 
conf/remotes.yml +6 -3
index 51c523d..446bb15 100644
@@ -1,6 +1,9 @@
 torvalds:
-  commit_url_prefix: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=
+  git_repo_url: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
+  commit_url_suffix: /commit/?id=
 stable:
-  commit_url_prefix: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit?id=
+  git_repo_url: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
+  commit_url_suffix: /commit?id=
 cip:
-  commit_url_prefix: https://git.kernel.org/pub/scm/linux/kernel/git/cip/linux-cip.git/commit?id=
+  git_repo_url: https://git.kernel.org/pub/scm/linux/kernel/git/cip/linux-cip.git
+  commit_url_suffix: /commit?id=
scripts/import_stable.py +9 -0
index b3261cf..26e45a9 100755
@@ -35,6 +35,9 @@ def update(git_repo, remote_name):
     subprocess.check_call(['git', 'remote', 'update', remote_name],
                           cwd=git_repo)
 
+def add(git_repo, remote_name, remote_url):
+    subprocess.check_call(['git', 'remote', 'add', remote_name, remote_url],
+                          cwd=git_repo)
 
 def get_backports(git_repo, remotes, branches, debug=False):
     backports = {}
@@ -140,6 +143,12 @@ def main(git_repo, remotes, debug=False):
     remote_names = set(branch['git_remote'] for branch in branches)
 
     for remote_name in remote_names:
+        import sys
+        current_remotes = subprocess.check_output(['git', 'remote', 'show'],
+            cwd=git_repo).decode(sys.stdout.encoding).strip().split('\n')
+        if remote_name not in current_remotes:
+            add(git_repo, remotes[remote_name]['git_name'],
+                remotes[remote_name]['git_repo_url'])
         update(git_repo, remotes[remote_name]['git_name'])
     backports = get_backports(git_repo, remotes, branches, debug)
     c_b_map = kernel_sec.branch.CommitBranchMap(git_repo, remotes, branches)
scripts/templates/issue.html +4 -3
index a3286e1..de42753 100644
@@ -75,7 +75,8 @@
     <th rowspan={{ branches|length }}>Status</th>
     {% for branch, affected in branches %}
     {% set name = branch.short_name %}
-    {% set url_prefix = remotes[branch.git_remote].commit_url_prefix %}
+    {% set git_repo_url = remotes[branch.git_remote].git_repo_url %}
+    {% set url_suffix = remotes[branch.git_remote].commit_url_suffix %}
     <th>
       <a href="/branch/{{ name }}/">{{ name }}</a>
     </th>
@@ -84,7 +85,7 @@
       {% if issue['fixed-by'] and issue['fixed-by'][name] and issue['fixed-by'][name] != 'never' %}
       <span class="good">fixed</span> by
       {% for commit in issue['fixed-by'][name] %}
-      <a href="{{ url_prefix }}{{ commit }}">{{ commit[:12] }}</a>{% if not loop.last %},{% endif %}
+      <a href="{{ git_repo_url }}{{ url_suffix }}{{ commit }}">{{ commit[:12] }}</a>{% if not loop.last %},{% endif %}
       {% endfor %}
       {% else %}
       <span class="good">never affected</span>
@@ -98,7 +99,7 @@
       {% if issue['introduced-by'] and issue['introduced-by'][name] and issue['introduced-by'][name] != 'never' %}
       - introduced by
       {% for commit in issue['introduced-by'][name] %}
-      <a href="{{ url_prefix }}{{ commit }}">{{ commit[:12] }}</a>{% if not loop.last %},{% endif %}
+      <a href="{{ git_repo_url }}{{ url_suffix }}{{ commit }}">{{ commit[:12] }}</a>{% if not loop.last %},{% endif %}
       {% endfor %}
       {% endif %}
       {% endif %}
Signature · 1 line
2.17.1
#1 lore raw

PATCH 2/2 git_repo: add a check for the git_repo directory

Users are required to prepare a local git repository, by default "../kernel", however there is no check at the moment. Alternatively, we could also create a repository for them (mkdir ../kernel; git init .) but that for now let's just spit an error.

---
 scripts/import_stable.py     |  1 +
 scripts/kernel_sec/branch.py | 11 +++++++++++
 scripts/report_affected.py   |  1 +
 scripts/webview.py           |  1 +
 4 files changed, 14 insertions(+)
scripts/import_stable.py +1 -0
index 26e45a9..2d9b66b 100755
@@ -179,6 +179,7 @@ if __name__ == '__main__':
                         dest='git_repo', default='../kernel',
                         help=('git repository from which to read commit logs '
                               '(default: ../kernel)'),
+                        type=kernel_sec.branch.check_git_repo,
                         metavar='DIRECTORY')
     parser.add_argument('--remote-name',
                         dest='remote_name', action='append', default=[],
scripts/kernel_sec/branch.py +11 -0
index 3ede64a..0ea5086 100644
@@ -219,3 +219,14 @@ def get_remotes(mappings, mainline=None, stable=None):
     if stable:
         remotes['stable']['git_name'] = stable
     return remotes
+
+
+def check_git_repo(git_repo):
+    import argparse
+    if not os.path.isdir(git_repo):
+        msg = "directory %r not present" % git_repo
+        raise argparse.ArgumentTypeError(msg)
+    if not os.path.isdir(os.path.join(git_repo, '.git')):
+        msg = "directory %r is not a git repository" % git_repo
+        raise argparse.ArgumentTypeError(msg)
+    return git_repo
scripts/report_affected.py +1 -0
index fac0885..bcb8168 100755
@@ -72,6 +72,7 @@ if __name__ == '__main__':
                         dest='git_repo', default='../kernel',
                         help=('git repository from which to read commit logs '
                               '(default: ../kernel)'),
+                        type=kernel_sec.branch.check_git_repo,
                         metavar='DIRECTORY')
     parser.add_argument('--remote-name',
                         dest='remote_name', action='append', default=[],
scripts/webview.py +1 -0
index 9e16a0c..e334371 100755
@@ -202,6 +202,7 @@ if __name__ == '__main__':
                         dest='git_repo', default='../kernel',
                         help=('git repository from which to read commit logs '
                               '(default: ../kernel)'),
+                        type=kernel_sec.branch.check_git_repo,
                         metavar='DIRECTORY')
     parser.add_argument('--remote-name',
                         dest='remote_name', action='append', default=[],
Signature · 1 line
2.17.1
#2 lore raw

PATCH 1/2 remotes: automatically add remotes from configuration file

In reply to Daniel Sangorrin
Quoted 5 lines · Daniel Sangorrin

Currently the user is required to create its own remotes by hand. This should not be necessary, because the information is already collected in conf/remotes.yml. For that reason, if we detect that any remote has not been added to the local repo then we will add those.

I'm not yet convinced this is a good idea.

After this change, we would still be assuming that there is a kernel repository or working tree next to kernel-config. If we wanted to be really helpful we could create a new repository if it's not there.

But realistically, a developer using this probably already has a kernel git repository, just not in the expected place. Isn't it likely that the developer also already has the remotes we want, but under different names?

Aside from that, I have some specific issues with the implementation:

Quoted 16 lines

---

?README.md????????????????????| 5 +++--
?conf/remotes.yml?????????????| 9 ++++++---
?scripts/import_stable.py?????| 9 +++++++++
?scripts/templates/issue.html | 7 ++++---

?4 files changed, 22 insertions(+), 8 deletions(-)

README.md +0 -0
index 4c5808f..dda94a8 100644
@@ -77,8 +77,9 @@ These files, if they exist, contain a mapping where the keys
 ?are default git remote names.??The values are also mappings,

?with the keys:
?

README.md These files, if they exist, contain a mapping where the keys

Quoted 5 lines
README.md +3 -2
-* `commit_url_prefix`: URL prefix for browsing a commit on a
-??branch from this remote.
+* `git_repo_url`: URL of the remote repository.
+* `commit_url_suffix`: URL suffix that gets appended to `git_repo_url`
+??for browsing a commit on a branch from this remote.

Although it's now common to have a single https: URL that works as both a git remote and a web view, there are plenty of git hosts that don't work that way.

So these two things should be kept independent.

[...]

Quoted 5 lines
scripts/import_stable.py +0 -0
@@ -35,6 +35,9 @@ def update(git_repo, remote_name):
 ?????subprocess.check_call(['git', 'remote', 'update', remote_name],
 ???????????????????????????cwd=git_repo)
Quoted 7 lines

?
+def add(git_repo, remote_name, remote_url):
+????subprocess.check_call(['git', 'remote', 'add', remote_name, remote_url], +??????????????????????????cwd=git_repo)
?
?def get_backports(git_repo, remotes, branches, debug=False):
?????backports = {}

There should be two blank lines between top-level definitions, according to PEP-8.

scripts/import_stable.py def main(git_repo, remotes, debug=False): · line 146

Quoted 5 lines
scripts/import_stable.py +1 -0
@@ -140,6 +143,12 @@ def main(git_repo, remotes, debug=False):
     remote_names = set(branch['git_remote'] for branch in branches)
 
     for remote_name in remote_names:
+        import sys

This import belongs at the top level with the other imports.

scripts/import_stable.py def main(git_repo, remotes, debug=False): · line 148

Quoted 2 lines
scripts/import_stable.py +2 -0
+        current_remotes = subprocess.check_output(['git', 'remote', 'show'],
+            cwd=git_repo).decode(sys.stdout.encoding).strip().split('\n')

This doesn't belong inside the loop.

scripts/import_stable.py def main(git_repo, remotes, debug=False): · line 154

Quoted 6 lines
scripts/import_stable.py +3 -0
+        if remote_name not in current_remotes:
+            add(git_repo, remotes[remote_name]['git_name'],
+                remotes[remote_name]['git_repo_url'])
         update(git_repo, remotes[remote_name]['git_name'])
     backports = get_backports(git_repo, remotes, branches, debug)
     c_b_map = kernel_sec.branch.CommitBranchMap(git_repo, remotes, branches)

Ben.

Signature · 3 lines
Ben Hutchings, Software Developer                ?        Codethink Ltd
https://www.codethink.co.uk/                 Dale House, 35 Dale Street
                                     Manchester, M1 2HF, United Kingdom
#3 lore raw

PATCH 1/2 remotes: automatically add remotes from configuration file

Quoted 9 lines

From: Ben Hutchings <[email protected]>

Quoted 5 lines · Daniel Sangorrin

Currently the user is required to create its own remotes by hand. This should not be necessary, because the information is already collected in conf/remotes.yml. For that reason, if we detect that any remote has not been added to the local repo then we will add those.

I'm not yet convinced this is a good idea.

I sent a new patch that takes a different approach. The new patch will only check that the necessary remotes are there. I also prepared a new separate script, for new users to prepare the local repository.

Quoted 3 lines

After this change, we would still be assuming that there is a kernel repository or working tree next to kernel-config. If we wanted to be really helpful we could create a new repository if it's not there.

I didn't understand the kernel-config thing. Maybe I am missing something important.

Quoted 6 lines

But realistically, a developer using this probably already has a kernel git repository, just not in the expected place. Isn't it likely that the developer also already has the remotes we want, but under different names?

Aside from that, I have some specific issues with the implementation:

Thanks the new patch has taken those issues into account.

Kind regards,
Daniel

Quoted 72 lines
Quoted 17 lines

---

?README.md????????????????????| 5 +++--
?conf/remotes.yml?????????????| 9 ++++++---
?scripts/import_stable.py?????| 9 +++++++++
?scripts/templates/issue.html | 7 ++++---

?4 files changed, 22 insertions(+), 8 deletions(-)

README.md +0 -0
index 4c5808f..dda94a8 100644
@@ -77,8 +77,9 @@ These files, if they exist, contain a mapping where the keys
 ?are default git remote names.??The values are also mappings,

?with the keys:

README.md These files, if they exist, contain a mapping where the keys

Quoted 5 lines
README.md +3 -2
-* `commit_url_prefix`: URL prefix for browsing a commit on a
-??branch from this remote.
+* `git_repo_url`: URL of the remote repository.
+* `commit_url_suffix`: URL suffix that gets appended to `git_repo_url`
+??for browsing a commit on a branch from this remote.

Although it's now common to have a single https: URL that works as both a git remote and a web view, there are plenty of git hosts that don't work that way.

So these two things should be kept independent.

[...]

Quoted 10 lines
scripts/import_stable.py +3 -0
@@ -35,6 +35,9 @@ def update(git_repo, remote_name):
 ?????subprocess.check_call(['git', 'remote', 'update', remote_name],
 ???????????????????????????cwd=git_repo)
 
+def add(git_repo, remote_name, remote_url):
+????subprocess.check_call(['git', 'remote', 'add', remote_name, remote_url],
+??????????????????????????cwd=git_repo)
 
Quoted 2 lines

?def get_backports(git_repo, remotes, branches, debug=False):
?????backports = {}

There should be two blank lines between top-level definitions, according to PEP-8.

scripts/import_stable.py def main(git_repo, remotes, debug=False): · line 146

Quoted 5 lines
scripts/import_stable.py +1 -0
@@ -140,6 +143,12 @@ def main(git_repo, remotes, debug=False):
     remote_names = set(branch['git_remote'] for branch in branches)
 
     for remote_name in remote_names:
+        import sys

This import belongs at the top level with the other imports.

scripts/import_stable.py def main(git_repo, remotes, debug=False): · line 148

Quoted 2 lines
scripts/import_stable.py +2 -0
+        current_remotes = subprocess.check_output(['git', 'remote', 'show'],
+            cwd=git_repo).decode(sys.stdout.encoding).strip().split('\n')

This doesn't belong inside the loop.

scripts/import_stable.py def main(git_repo, remotes, debug=False): · line 154

Quoted 6 lines
scripts/import_stable.py +3 -0
+        if remote_name not in current_remotes:
+            add(git_repo, remotes[remote_name]['git_name'],
+                remotes[remote_name]['git_repo_url'])
         update(git_repo, remotes[remote_name]['git_name'])
     backports = get_backports(git_repo, remotes, branches, debug)
     c_b_map = kernel_sec.branch.CommitBranchMap(git_repo, remotes, branches)

Ben.

--
Ben Hutchings, Software Developer ? Codethink Ltd
https://www.codethink.co.uk/ Dale House, 35 Dale Street
Manchester, M1 2HF, United Kingdom

#4 lore raw

PATCH 1/2 remotes: automatically add remotes from configuration file

Quoted 14 lines · daniel.sangorrin at toshiba.co.jp
Quoted 9 lines
Quoted 1 line
Quoted 1 line

From: Ben Hutchings <[email protected]>

Quoted 5 lines · Daniel Sangorrin

Currently the user is required to create its own remotes by hand. This should not be necessary, because the information is already collected in conf/remotes.yml. For that reason, if we detect that any remote has not been added to the local repo then we will add those.

I'm not yet convinced this is a good idea.

I sent a new patch that takes a different approach. The new patch will only check that the necessary remotes are there. I also prepared a new separate script, for new users to prepare the local repository.

Quoted 5 lines
Quoted 3 lines

After this change, we would still be assuming that there is a kernel repository or working tree next to kernel-config.??If we wanted to be really helpful we could create a new repository if it's not there.

I didn't understand the kernel-config thing. Maybe I am missing something important.

[...]

Sorry, I meant "next to kernel-sec".

Ben.

Signature · 3 lines
Ben Hutchings, Software Developer                ?        Codethink Ltd
https://www.codethink.co.uk/                 Dale House, 35 Dale Street
                                     Manchester, M1 2HF, United Kingdom