autobuild-run: add --repo option

This option allows to specify which Buildroot git repository to clone. This can be useful in several situations:
- to use a different mirror in case you don't have a good connection to github;
- for debugging this script;
- to point to a local, patched repository you want to test.

Note that the clone/pull will use the currently checked out branch of the repository if it is non-bare, which is ideal for testing.

Since switching repositories may also switch branches, we use a git fetch/checkout sequence instead of doing a git pull. With git pull, the branches would be merged instead of switched. To avoid polluting the log with the long git message about a detached head, while still getting some useful output from git, pass the --detach option to git checkout.

Signed-off-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>

---
Ideally the contents of this commit message would be put somewhere as documentation, but I couldn't find a good spot for it.

---
 scripts/autobuild-run | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)

scripts/autobuild-run +12 -4

index a7d7d4f..523e382 100755
@@ -68,6 +68,7 @@ defaults = {
     '--pid-file': '/tmp/buildroot-autobuild.pid',
     '--http-url': 'http://autobuild.buildroot.org/submit/',
     '--toolchains-url': 'http://autobuild.buildroot.org/toolchains/configs/toolchain-configs.csv',
+    '--repo': 'https://github.com/buildroot/buildroot.git',
 }
 
 doc = """autobuild-run - run Buildroot autobuilder

@@ -103,6 +104,8 @@ Options:
                                  Not set by default.
   -d, --debug                    Send log output to stdout instead of log file
   --toolchains-url URL           URL of toolchain configuration file
+  -r, --repo URL                 URL of Buildroot repository to clone
+                                 Defaults to %(--repo)s
 
 Format of the configuration file:
 

@@ -329,7 +332,7 @@ def prepare_build(**kwargs):
     # didn't exist already.
     srcdir = os.path.join(idir, "buildroot")
     if not os.path.exists(srcdir):
-        ret = subprocess.call(["git", "clone", "https://github.com/buildroot/buildroot.git", srcdir],
+        ret = subprocess.call(["git", "clone", kwargs['repo'], srcdir],
                               stdout=log, stderr=log)
         if ret != 0:
             log_write(log, "ERROR: could not clone Buildroot sources")

@@ -337,9 +340,14 @@ def prepare_build(**kwargs):
 
     # Update the Buildroot sources.
     abssrcdir = os.path.abspath(srcdir)
-    ret = subprocess.call(["git", "pull"], cwd=abssrcdir, stdout=log, stderr=log)
+    ret = subprocess.call(["git", "fetch", kwargs['repo']], cwd=abssrcdir, stdout=log, stderr=log)
     if ret != 0:
-        log_write(log, "ERROR: could not pull Buildroot sources")
+        log_write(log, "ERROR: could not fetch Buildroot sources")
+        return -1
+
+    ret = subprocess.call(["git", "checkout", "--detach", "FETCH_HEAD"], cwd=abssrcdir, stdout=log, stderr=log)
+    if ret != 0:
+        log_write(log, "ERROR: could not check out Buildroot sources")
         return -1
 
     # Create an empty output directory. We remove it first, in case a previous build was aborted.

@@ -948,6 +956,7 @@ def main():
                 make_opts = (args['--make-opts'] or ''),
                 nice = (args['--nice'] or 0),
                 toolchains_url = args['--toolchains-url'],
+                repo = args['--repo'],
                 upload = upload,
                 buildpid = buildpid,
                 debug = args['--debug']
--

2.13.2

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Arnout,

Quoted 17 lines · Arnout Vandecappelle (Essensium/Mind)

This option allows to specify which Buildroot git repository to clone. This can be useful in several situations:
- to use a different mirror in case you don't have a good connection to github;
- for debugging this script;
- to point to a local, patched repository you want to test.

Note that the clone/pull will use the currently checked out branch of the repository if it is non-bare, which is ideal for testing.

Since switching repositories may also switch branches, we use a git fetch/checkout sequence instead of doing a git pull. With git pull, the branches would be merged instead of switched. To avoid polluting the log with the long git message about a detached head, while still getting some useful output from git, pass the --detach option to git checkout.

Quoted 1 line

Signed-off-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>

I carry some local changes for tailoring the repo for similar reasons (I while back I submitted something similar before the input parsing was cleaned up). What do you think about also making it a config file option?

This option allows to specify which Buildroot git repository to clone. This can be useful in several situations:
- to use a different mirror in case you don't have a good connection to github;
- for debugging this script;
- to point to a local, patched repository you want to test.

Note that the clone/pull will use the currently checked out branch of the repository if it is non-bare, which is ideal for testing.

Since switching repositories may also switch branches, we use a git fetch/checkout sequence instead of doing a git pull. With git pull, the branches would be merged instead of switched. To avoid polluting the log with the long git message about a detached head, while still getting some useful output from git, pass the --detach option to git checkout.

Signed-off-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>

Hello,

Quoted 17 lines · Arnout Vandecappelle (Essensium/Mind)

This option allows to specify which Buildroot git repository to clone. This can be useful in several situations:
- to use a different mirror in case you don't have a good connection to github;
- for debugging this script;
- to point to a local, patched repository you want to test.

Note that the clone/pull will use the currently checked out branch of the repository if it is non-bare, which is ideal for testing.

Since switching repositories may also switch branches, we use a git fetch/checkout sequence instead of doing a git pull. With git pull, the branches would be merged instead of switched. To avoid polluting the log with the long git message about a detached head, while still getting some useful output from git, pass the --detach option to git checkout.

Quoted 1 line

Signed-off-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>

My concern about such change (which I've already expressed in the past) is to see build results submitted to the public autobuild.b.o instance that do not correspond to an upstream Buildroot version.

Would it be reasonable to request that if a custom repo is used, then uploading the results to autobuild.b.o is not possible ?

Or am I too zealous about this ?

Quoted 2 lines

Would it be reasonable to request that if a custom repo is used, then uploading the results to autobuild.b.o is not possible ?

But Matt's use case is exactly that he doesn't have a good connection to the
upstream repo so he wants to use a mirror, but still upload to a.b.o.

Admittedly, this is exactly what lead to the recent problems with Matt's
autobuilder...

Quoted 1 line

Or am I too zealous about this ?

Yes you are. Without this change, someone like Matt will just edit the
autobuild-run script and you still have the same problem :-)

Perhaps we can make the problem at least detectable, by pushing *kwargs to
a.b.o in addition to the BR git ID. But that obviously requires modifications on a.b.o itself, which is a bit more difficult to organize I guess.

Since you're the BDFL for autobuild-run, feel free to follow up this patch with
something that disables uploading if --repo is non-default. And let Matt solve his own problems :-)

Or I could add a server-side check that the git commit being reported exists in the upstream Buildroot, and is not more than 1 or 2 days old. It probably makes more sense to do this server-side.

Which means I could merge your --repo option (even if I still find its use cases very, very dubious).

On 18-07-17 10:19, Thomas Petazzoni wrote:
[snip]

Quoted 3 lines

Or I could add a server-side check that the git commit being reported exists in the upstream Buildroot, and is not more than 1 or 2 days old. It probably makes more sense to do this server-side.

Probably yes, because indeed, something like Matt's problem could occur for
other reasons as well.

Quoted 2 lines

Which means I could merge your --repo option (even if I still find its use cases very, very dubious).

Well, my own use case was to be able to test the autobuild-run script with the
in-tree config generator. When that happens, it does make a lot of sense to be able to test any change to the config generator with autobuild-run. Currently you can test the change by just running the modified autobuild-run, but when the config generator comes from the Buildroot tree that's not possible anymore.

I was going to post all those changes together, but since Matt was mentioning
he had local changes to use a different mirror, I thought I'd post this one individually already.

This option allows to specify which Buildroot git repository to clone. This can be useful in several situations:
- to use a different mirror in case you don't have a good connection to github;
- for debugging this script;
- to point to a local, patched repository you want to test.

Note that the clone/pull will use the currently checked out branch of the repository if it is non-bare, which is ideal for testing.

Since switching repositories may also switch branches, we use a git fetch/checkout sequence instead of doing a git pull. With git pull, the branches would be merged instead of switched. To avoid polluting the log with the long git message about a detached head, while still getting some useful output from git, pass the --detach option to git checkout.

Signed-off-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>

---
Ideally the contents of this commit message would be put somewhere as documentation, but I couldn't find a good spot for it.

---
 scripts/autobuild-run | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)