Skip to contentSkip to conversation

autogen.sh: Restore --no-git (avoid git submodule update)

PATCH In review Ian Jackson 4 participants 8 messages Jun 2, 2020 – Jun 3, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 autogen.sh: Restore --no-git (avoid git submodule update)

Prior to 2621d48f005a "gnulib: delete all gnulib integration", one could pass ./autogen.sh --no-git to prevent the libvirt build system from running git submodule update.

This feature is needed by systems like the Xen Project CI which want to explicitly control the revisions of every tree. These will typically arrange to initialise the submodules check out the right version of everything, and then expect the build system not to mess with it any more.

Despite to the old documentation comments referring only to gnulib, the --no-git feature is required not only because of gnulib but also because of the other submodule, src/keycodemapdb.

(And in any case, even if it were no longer required because all the submodules were removed, it ought ideally to have been retained as a no-op for compaibility reasons.)

So restore the --no-git feature.

Because of the way the argument parsing of autogen.sh works, it is easiest to recognise this option only if it comes first. This works for the Xen Project CI, which has always passed this option first.

If something else is using this option (and hasn't introduced a different workaround in the meantime), not in the first position, then perhaps a more sophisticated approach will be needed. But I think this will do for now.

---
 autogen.sh | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
autogen.sh +10 -1
index 4e1bbceb0a..1c98273452 100755
@@ -1,5 +1,10 @@
 #!/bin/sh
 # Run this to generate all the initial makefiles, etc.
+#
+# THe following options must come first.  All other or subsequent
+# arguments are passed to configure:
+#   --no-git   skip `git submodule update --init`
+
 test -n "$srcdir" || srcdir=$(dirname "$0")
 test -n "$srcdir" || srcdir=.
 
@@ -13,7 +18,11 @@ cd "$srcdir"
     exit 1
 }
 
-git submodule update --init || exit 1
+if [ "x$1" = x--no-git ]; then
+	shift
+else
+	git submodule update --init || exit 1
+fi
 
 autoreconf --verbose --force --install || exit 1
 
Signature · 1 line
2.11.0
#1 lore raw
Quoted 9 lines · Ian Jackson

Prior to 2621d48f005a "gnulib: delete all gnulib integration", one could pass ./autogen.sh --no-git to prevent the libvirt build system from running git submodule update.

This feature is needed by systems like the Xen Project CI which want to explicitly control the revisions of every tree. These will typically arrange to initialise the submodules check out the right version of everything, and then expect the build system not to mess with it any more.

To be honest I don't understand why would anyone want to keep track of all submodules of all projects for any CI and update it manually every time the upstream project changes these submodules. Sounds like a lot of unnecessary work but maybe I'm missing something.

Quoted 7 lines

Despite to the old documentation comments referring only to gnulib, the --no-git feature is required not only because of gnulib but also because of the other submodule, src/keycodemapdb.

(And in any case, even if it were no longer required because all the submodules were removed, it ought ideally to have been retained as a no-op for compaibility reasons.)

Well, we will break a lot more by switching to Meson build system where everyone building libvirt will have to change their scripts as it will not be compatible at all.

Pavel

Quoted 47 lines

So restore the --no-git feature.

Because of the way the argument parsing of autogen.sh works, it is easiest to recognise this option only if it comes first. This works for the Xen Project CI, which has always passed this option first.

If something else is using this option (and hasn't introduced a different workaround in the meantime), not in the first position, then perhaps a more sophisticated approach will be needed. But I think this will do for now.

---
 autogen.sh | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
autogen.sh +10 -1
index 4e1bbceb0a..1c98273452 100755
@@ -1,5 +1,10 @@
 #!/bin/sh
 # Run this to generate all the initial makefiles, etc.
+#
+# THe following options must come first.  All other or subsequent
+# arguments are passed to configure:
+#   --no-git   skip `git submodule update --init`
+
 test -n "$srcdir" || srcdir=$(dirname "$0")
 test -n "$srcdir" || srcdir=.
 
@@ -13,7 +18,11 @@ cd "$srcdir"
     exit 1
 }
 
-git submodule update --init || exit 1
+if [ "x$1" = x--no-git ]; then
+	shift
+else
+	git submodule update --init || exit 1
+fi
 
 autoreconf --verbose --force --install || exit 1
 
Signature · 1 line
2.11.0
#2 lore raw
Quoted 11 lines · Pavel Hrdina
Quoted 9 lines · Ian Jackson

Prior to 2621d48f005a "gnulib: delete all gnulib integration", one could pass ./autogen.sh --no-git to prevent the libvirt build system from running git submodule update.

This feature is needed by systems like the Xen Project CI which want to explicitly control the revisions of every tree. These will typically arrange to initialise the submodules check out the right version of everything, and then expect the build system not to mess with it any more.

Quoted 4 lines

To be honest I don't understand why would anyone want to keep track of all submodules of all projects for any CI and update it manually every time the upstream project changes these submodules. Sounds like a lot of unnecessary work but maybe I'm missing something.

Two possible scenarios that I think are valid

- The CI job script does not have network access
- The CI job script sees the source dir as read-only

In both cases the CI harness would have to initialize the submodule before runing the CI job.

I don't know if this is what Xen CI is hitting, but I think the request is reasonable none the less.

Both Jenkins and GitLab CI have option to make the harness init submodules prior to the job running.

Quoted 11 lines
Quoted 7 lines

Despite to the old documentation comments referring only to gnulib, the --no-git feature is required not only because of gnulib but also because of the other submodule, src/keycodemapdb.

(And in any case, even if it were no longer required because all the submodules were removed, it ought ideally to have been retained as a no-op for compaibility reasons.)

Well, we will break a lot more by switching to Meson build system where everyone building libvirt will have to change their scripts as it will not be compatible at all.

Yes, but I think that's tangential, as the two above reasons will still apply, and Meson will cope with having submodules pre-initialized.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#3 lore raw
Quoted 22 lines · Daniel P. Berrangé
Quoted 11 lines · Pavel Hrdina
Quoted 9 lines · Ian Jackson

Prior to 2621d48f005a "gnulib: delete all gnulib integration", one could pass ./autogen.sh --no-git to prevent the libvirt build system from running git submodule update.

This feature is needed by systems like the Xen Project CI which want to explicitly control the revisions of every tree. These will typically arrange to initialise the submodules check out the right version of everything, and then expect the build system not to mess with it any more.

Quoted 4 lines

To be honest I don't understand why would anyone want to keep track of all submodules of all projects for any CI and update it manually every time the upstream project changes these submodules. Sounds like a lot of unnecessary work but maybe I'm missing something.

Two possible scenarios that I think are valid

- The CI job script does not have network access
- The CI job script sees the source dir as read-only

Quoted 8 lines

In both cases the CI harness would have to initialize the submodule before runing the CI job.

I don't know if this is what Xen CI is hitting, but I think the request is reasonable none the less.

Both Jenkins and GitLab CI have option to make the harness init submodules prior to the job running.

My point was that running 'git submodule update --init' will not change anything if all submodules are updated to the correct revision and if the repository was pre-created with submodules and is read-only when the test is executed the git command will not fail as well and everything will be fine.

If the submodules are not updated to the correct revision then it will fail and notify the CI users that something is broken.

There should not be any need to disable this explicitly unless you want to build libvirt with different revisions of submodules.

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

Despite to the old documentation comments referring only to gnulib, the --no-git feature is required not only because of gnulib but also because of the other submodule, src/keycodemapdb.

(And in any case, even if it were no longer required because all the submodules were removed, it ought ideally to have been retained as a no-op for compaibility reasons.)

Quoted 6 lines
Quoted 3 lines

Well, we will break a lot more by switching to Meson build system where everyone building libvirt will have to change their scripts as it will not be compatible at all.

Yes, but I think that's tangential, as the two above reasons will still apply, and Meson will cope with having submodules pre-initialized.

Yes, these are unrelated and I just wanted to point out that compaibility reasons are in most cases not valid to changes to build system or moving files around in git repository and so on.

Pavel

#4 lore raw

Pavel Hrdina writes ("Re: [PATCH] autogen.sh: Restore --no-git (avoid git submodule update)"):

Quoted 2 lines

There should not be any need to disable this explicitly unless you want to build libvirt with different revisions of submodules.

The Xen Project CI has a cross-tree bisector. It is capable of automatically selecting revisions, including revisions of submodules, for testing.

It therefore needs to be able to build with different revisions of submodules, indeed.

Thanks,
Ian.

#5 lore raw
In reply to Pavel Hrdina

Pavel Hrdina writes ("Re: [PATCH] autogen.sh: Restore --no-git (avoid git submodule update)"):

Quoted 4 lines

To be honest I don't understand why would anyone want to keep track of all submodules of all projects for any CI and update it manually every time the upstream project changes these submodules. Sounds like a lot of unnecessary work but maybe I'm missing something.

Maybe I should answer this. The short answer is that this can be done entirely automatically.

Quoted 3 lines

Well, we will break a lot more by switching to Meson build system where everyone building libvirt will have to change their scripts as it will not be compatible at all.

When that occurs we'll have to change our build rune, of course.

Our CI system (which does bisection and stable tracking and so on) will wants to build old versions, so it'll have to have both build runes and look for something in-tree to tell the two methods apart.

Thanks,
Ian.

Reviewed-by on v1 1/1
#6 lore raw
In reply to Ian Jackson
Quoted 50 lines · Ian Jackson

Prior to 2621d48f005a "gnulib: delete all gnulib integration", one could pass ./autogen.sh --no-git to prevent the libvirt build system from running git submodule update.

This feature is needed by systems like the Xen Project CI which want to explicitly control the revisions of every tree. These will typically arrange to initialise the submodules check out the right version of everything, and then expect the build system not to mess with it any more.

Despite to the old documentation comments referring only to gnulib, the --no-git feature is required not only because of gnulib but also because of the other submodule, src/keycodemapdb.

(And in any case, even if it were no longer required because all the submodules were removed, it ought ideally to have been retained as a no-op for compaibility reasons.)

So restore the --no-git feature.

Because of the way the argument parsing of autogen.sh works, it is easiest to recognise this option only if it comes first. This works for the Xen Project CI, which has always passed this option first.

If something else is using this option (and hasn't introduced a different workaround in the meantime), not in the first position, then perhaps a more sophisticated approach will be needed. But I think this will do for now.

---
 autogen.sh | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
autogen.sh +5 -0
index 4e1bbceb0a..1c98273452 100755
@@ -1,5 +1,10 @@
 #!/bin/sh
 # Run this to generate all the initial makefiles, etc.
+#
+# THe following options must come first.  All other or subsequent
+# arguments are passed to configure:
+#   --no-git   skip `git submodule update --init`
+
 test -n "$srcdir" || srcdir=$(dirname "$0")
 test -n "$srcdir" || srcdir=.
 

autogen.sh cd "$srcdir" · line 24

Quoted 8 lines
autogen.sh +4 -1
@@ -13,7 +18,11 @@ cd "$srcdir"
     exit 1
 }
 
-git submodule update --init || exit 1
+if [ "x$1" = x--no-git ]; then
+	shift
+else
+	git submodule update --init || exit 1

I changed the TAB into spaces.

autogen.sh cd "$srcdir" · line 27

Quoted 3 lines
autogen.sh +1 -0
+fi
 
 autoreconf --verbose --force --install || exit 1

and pushed.

#7 lore raw

Pavel Hrdina writes ("Re: [PATCH] autogen.sh: Restore --no-git (avoid git submodule update)"):

Quoted 8 lines · Ian Jackson

autogen.sh cd "$srcdir" · line 24

Quoted 5 lines
autogen.sh +4 -1
-git submodule update --init || exit 1
+if [ "x$1" = x--no-git ]; then
+	shift
+else
+	git submodule update --init || exit 1

I changed the TAB into spaces.

Ah, sorry for not following the right style.

Quoted 3 lines

and pushed.

Thanks :-).

Ian.