Skip to contentSkip to conversation

git: add BB_FETCH_PREMIRROR_READONLY option

RFC In review Martin Jansa 3 participants 9 messages Apr 30, 2013 – May 10, 2013

lore
Share & tools
lore
#0 lore raw

RFC 1/1 git: add BB_FETCH_PREMIRROR_READONLY option

* just RFC, because I haven't even tested this change, use it only to describe the issue and discuss it

* remove symlink to ud.fullmirror when BB_FETCH_PREMIRROR_READONLY is set

* with read-only PREMIRROR (e.g. mounted over NFS or CIFS and referenced as file:///mnt/premirror) we cannot use BB_GENERATE_MIRROR_TARBALLS because all git2_abc.git.tar.gz files later became just symlinks to read-only location in PREMIRROR (it works fine on first build and for new components, because at that time there isn't tarball on PREMIRROR yet).

ERROR: Fetcher failure: Fetch command failed with exit code 141, output:
tar (child): /build/downloads/git2_abc.git.tar.gz: Cannot open: Read-only file system tar (child): Error is not recoverable: exiting now

* maybe we can change the default behavior and always remove symlink without introducing new option

---
 lib/bb/fetch2/git.py | 4 ++++
 1 file changed, 4 insertions(+)
lib/bb/fetch2/git.py +4 -0
index 052802e..14a8bc2 100644
@@ -217,6 +217,10 @@ class Git(FetchMethod):
     def build_mirror_data(self, url, ud, d):
         # Generate a mirror tarball if needed
         if ud.write_tarballs and (ud.repochanged or not os.path.exists(ud.fullmirror)):
+            premirror_readonly = (d.getVar("BB_FETCH_PREMIRROR_READONLY", True) == "1")
+            if premirror_readonly and os.path.islink(ud.fullmirror):
+                os.unlink(origud.localpath)
+
             os.chdir(ud.clonedir)
             logger.info("Creating tarball of git repository")
             runfetchcmd("tar -czf %s %s" % (ud.fullmirror, os.path.join(".") ), d)
Signature · 1 line
1.8.2.1
#1 lore raw
Quoted 12 lines · On Tuesday 30 April 2013 12:52:40 Martin Jansa wrote:

* just RFC, because I haven't even tested this change, use it only to describe the issue and discuss it

* remove symlink to ud.fullmirror when BB_FETCH_PREMIRROR_READONLY is set

* with read-only PREMIRROR (e.g. mounted over NFS or CIFS and referenced as file:///mnt/premirror) we cannot use BB_GENERATE_MIRROR_TARBALLS because all git2_abc.git.tar.gz files later became just symlinks to read-only location in PREMIRROR (it works fine on first build and for new components, because at that time there isn't tarball on PREMIRROR yet).

Quoted 6 lines

ERROR: Fetcher failure: Fetch command failed with exit code 141, output:
tar (child): /build/downloads/git2_abc.git.tar.gz: Cannot open: Read-only
file system tar (child): Error is not recoverable: exiting now

* maybe we can change the default behavior and always remove symlink without introducing new option

Personally I think it would be better to do the latter.

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#2 lore raw
Quoted 13 lines · Paul Eggleton
Quoted 12 lines · On Tuesday 30 April 2013 12:52:40 Martin Jansa wrote:

* just RFC, because I haven't even tested this change, use it only to describe the issue and discuss it

* remove symlink to ud.fullmirror when BB_FETCH_PREMIRROR_READONLY is set

* with read-only PREMIRROR (e.g. mounted over NFS or CIFS and referenced as file:///mnt/premirror) we cannot use BB_GENERATE_MIRROR_TARBALLS because all git2_abc.git.tar.gz files later became just symlinks to read-only location in PREMIRROR (it works fine on first build and for new components, because at that time there isn't tarball on PREMIRROR yet).

Quoted 8 lines
Quoted 6 lines

ERROR: Fetcher failure: Fetch command failed with exit code 141, output:
tar (child): /build/downloads/git2_abc.git.tar.gz: Cannot open: Read-only
file system tar (child): Error is not recoverable: exiting now

* maybe we can change the default behavior and always remove symlink without introducing new option

Personally I think it would be better to do the latter.

Only case when keeping old behavior is IMHO desirable is when someone is using file:// and wants tar.gz updated in original location (e.g. to rsync less from downloads directory after build).

But it won't update all (e.g. new components which don't have symlinks yet) so rsync is still needed and usefulness of such requirement is quite low.

Cheers,

Quoted 8 lines

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
Signature · 1 line
Martin 'JaMa' Jansa     jabber: [email protected]
#3 lore raw

PATCH v2 1/1 git: remove symling before updating mirror tarball

In reply to Paul Eggleton

* with read-only PREMIRROR (e.g. mounted over NFS or CIFS and referenced as file:///mnt/premirror) we cannot use BB_GENERATE_MIRROR_TARBALLS because all git2_abc.git.tar.gz files later became just symlinks to read-only location in PREMIRROR (it works fine on first build and for new components, because at that time there isn't tarball on PREMIRROR yet).

ERROR: Fetcher failure: Fetch command failed with exit code 141, output:
tar (child): /build/downloads/git2_abc.git.tar.gz: Cannot open: Read-only file system tar (child): Error is not recoverable: exiting now

---
 lib/bb/fetch2/git.py | 4 ++++
 1 file changed, 4 insertions(+)
lib/bb/fetch2/git.py +4 -0
index 052802e..1ec8908 100644
@@ -217,6 +217,10 @@ class Git(FetchMethod):
     def build_mirror_data(self, url, ud, d):
         # Generate a mirror tarball if needed
         if ud.write_tarballs and (ud.repochanged or not os.path.exists(ud.fullmirror)):
+            # it's possible that this symlink points to read-only filesystem with PREMIRROR
+            if os.path.islink(ud.fullmirror):
+                os.unlink(origud.localpath)
+
             os.chdir(ud.clonedir)
             logger.info("Creating tarball of git repository")
             runfetchcmd("tar -czf %s %s" % (ud.fullmirror, os.path.join(".") ), d)
Signature · 1 line
1.8.2.1
#4 lore raw

PATCH v3 1/1 git: remove symling before updating mirror tarball

In reply to Paul Eggleton

* with read-only PREMIRROR (e.g. mounted over NFS or CIFS and referenced as file:///mnt/premirror) we cannot use BB_GENERATE_MIRROR_TARBALLS because all git2_abc.git.tar.gz files later became just symlinks to read-only location in PREMIRROR (it works fine on first build and for new components, because at that time there isn't tarball on PREMIRROR yet).

ERROR: Fetcher failure: Fetch command failed with exit code 141, output:
tar (child): /build/downloads/git2_abc.git.tar.gz: Cannot open: Read-only file system tar (child): Error is not recoverable: exiting now

---
 lib/bb/fetch2/git.py | 4 ++++
 1 file changed, 4 insertions(+)
lib/bb/fetch2/git.py +4 -0
index 052802e..6175e4c 100644
@@ -217,6 +217,10 @@ class Git(FetchMethod):
     def build_mirror_data(self, url, ud, d):
         # Generate a mirror tarball if needed
         if ud.write_tarballs and (ud.repochanged or not os.path.exists(ud.fullmirror)):
+            # it's possible that this symlink points to read-only filesystem with PREMIRROR
+            if os.path.islink(ud.fullmirror):
+                os.unlink(ud.fullmirror)
+
             os.chdir(ud.clonedir)
             logger.info("Creating tarball of git repository")
             runfetchcmd("tar -czf %s %s" % (ud.fullmirror, os.path.join(".") ), d)
Signature · 1 line
1.8.2.1
#5 lore raw

Re: [PATCHv3] git: remove symling before updating mirror tarball

Quoted 10 lines · Martin Jansa

* with read-only PREMIRROR (e.g. mounted over NFS or CIFS and referenced as file:///mnt/premirror) we cannot use BB_GENERATE_MIRROR_TARBALLS because all git2_abc.git.tar.gz files later became just symlinks to read-only location in PREMIRROR (it works fine on first build and for new components, because at that time there isn't tarball on PREMIRROR yet).

ERROR: Fetcher failure: Fetch command failed with exit code 141, output:
tar (child): /build/downloads/git2_abc.git.tar.gz: Cannot open: Read-only file system tar (child): Error is not recoverable: exiting now

This one is tested and I would like to get it also in 1.16 and 1.18

Steps to reproduce the issue (without redownloading anything big)

# mv downloads downloads-ro
# mkdir downloads
# mkdir tmp-eglibc/downloads
# sudo mount -o ro,bind downloads-ro tmp-eglibc/downloads
# sudo mount -o remount,ro tmp-eglibc/downloads

conf/local.conf:
SOURCE_MIRROR_URL = "file:///OE/oe-core/tmp-eglibc/downloads" INHERIT += "own-mirrors"

# bitbake -c fetch qtbase
# ls -lah downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz 
  should be a symlink

bump SRCREV in qtbase_git.bb to some revision not included in git2_qt.gitorious.org.qt.qtbase.git.tar.gz

# bitbake -c fetch qtbase
WARNING: Failed to fetch URL git://qt.gitorious.org/qt/qtbase.git;branch=stable, attempting MIRRORS if available
DEBUG: Fetcher failure: Fetch command failed with exit code 141, output:
tar (child): /OE/oe-core/downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz: Cannot open: Read-only file system
tar (child): Error is not recoverable: exiting now

apply this patch

# bitbake -c fetch qtbase
works fine
# ls -lah downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz
-rw-r--r-- 1 bitbake bitbake 176M May  7 23:41 downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz
is now proper file instead of symlink
Quoted 23 lines
---
 lib/bb/fetch2/git.py | 4 ++++
 1 file changed, 4 insertions(+)
lib/bb/fetch2/git.py +4 -0
index 052802e..6175e4c 100644
@@ -217,6 +217,10 @@ class Git(FetchMethod):
     def build_mirror_data(self, url, ud, d):
         # Generate a mirror tarball if needed
         if ud.write_tarballs and (ud.repochanged or not os.path.exists(ud.fullmirror)):
+            # it's possible that this symlink points to read-only filesystem with PREMIRROR
+            if os.path.islink(ud.fullmirror):
+                os.unlink(ud.fullmirror)
+
             os.chdir(ud.clonedir)
             logger.info("Creating tarball of git repository")
             runfetchcmd("tar -czf %s %s" % (ud.fullmirror, os.path.join(".") ), d)
Signature · 1 line
1.8.2.1
Signature · 1 line
Martin 'JaMa' Jansa     jabber: [email protected]
#6 lore raw

Re: [PATCHv3] git: remove symling before updating mirror tarball

Quoted 39 lines · Martin Jansa
Quoted 10 lines · Martin Jansa

* with read-only PREMIRROR (e.g. mounted over NFS or CIFS and referenced as file:///mnt/premirror) we cannot use BB_GENERATE_MIRROR_TARBALLS because all git2_abc.git.tar.gz files later became just symlinks to read-only location in PREMIRROR (it works fine on first build and for new components, because at that time there isn't tarball on PREMIRROR yet).

ERROR: Fetcher failure: Fetch command failed with exit code 141, output:
tar (child): /build/downloads/git2_abc.git.tar.gz: Cannot open: Read-only file system tar (child): Error is not recoverable: exiting now

This one is tested and I would like to get it also in 1.16 and 1.18

Steps to reproduce the issue (without redownloading anything big)

# mv downloads downloads-ro
# mkdir downloads
# mkdir tmp-eglibc/downloads
# sudo mount -o ro,bind downloads-ro tmp-eglibc/downloads
# sudo mount -o remount,ro tmp-eglibc/downloads

conf/local.conf:
SOURCE_MIRROR_URL = "file:///OE/oe-core/tmp-eglibc/downloads" INHERIT += "own-mirrors"

# bitbake -c fetch qtbase
# ls -lah downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz 
  should be a symlink

bump SRCREV in qtbase_git.bb to some revision not included in git2_qt.gitorious.org.qt.qtbase.git.tar.gz

# bitbake -c fetch qtbase
WARNING: Failed to fetch URL git://qt.gitorious.org/qt/qtbase.git;branch=stable, attempting MIRRORS if available
DEBUG: Fetcher failure: Fetch command failed with exit code 141, output:
tar (child): /OE/oe-core/downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz: Cannot open: Read-only file system
tar (child): Error is not recoverable: exiting now
Quoted 6 lines

apply this patch

# bitbake -c fetch qtbase
works fine
# ls -lah downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz
-rw-r--r-- 1 bitbake bitbake 176M May  7 23:41 downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz
is now proper file instead of symlink

There is one more interesting behavior with such setup where file:// fetcher is used in PREMIRROR

AFAIK it's completely harmless and also not influenced by this change,
but .done stamps for files downloaded from premirror are stored in ${DL_DIR}<path-to-local-premirror>/file.done, e.g. downloads/OE/oe-core/tmp-eglibc/downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz.done

qtbase was recreated locally thanks to this patch, that's why it has 2 .done stamps, one in downloads, 2nd in downloads/OE/oe-core/tmp-eglibc/downloads/

qtdeclarative has only symlink and .done stamp in downloads/OE/oe-core/tmp-eglibc/downloads/

OE @ ~/oe-core $ ll downloads/*qt{base,declarative}.git*

-rw-r--r-- 1 bitbake bitbake 184015350 May  7 23:41 downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz
-rw-r--r-- 1 bitbake bitbake         0 May  7 23:41 downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz.done
lrwxrwxrwx 1 bitbake bitbake        82 May  8 00:00 downloads/git2_qt.gitorious.org.qt.qtdeclarative.git.tar.gz -> /OE/oe-core/tmp-eglibc/downloads/git2_qt.gitorious.org.qt.qtdeclarative.git.tar.gz

OE @ ~/oe-core $ ll downloads/OE/oe-core/tmp-eglibc/downloads/*qt{base,declarative}.git*
-rw-r--r-- 1 bitbake bitbake 0 May 7 23:41 downloads/OE/oe-core/tmp-eglibc/downloads/git2_qt.gitorious.org.qt.qtbase.git.tar.gz.done -rw-r--r-- 1 bitbake bitbake 0 May 8 00:00 downloads/OE/oe-core/tmp-eglibc/downloads/git2_qt.gitorious.org.qt.qtdeclarative.git.tar.gz.done

Quoted 26 lines
Quoted 23 lines
---
 lib/bb/fetch2/git.py | 4 ++++
 1 file changed, 4 insertions(+)
lib/bb/fetch2/git.py +4 -0
index 052802e..6175e4c 100644
@@ -217,6 +217,10 @@ class Git(FetchMethod):
     def build_mirror_data(self, url, ud, d):
         # Generate a mirror tarball if needed
         if ud.write_tarballs and (ud.repochanged or not os.path.exists(ud.fullmirror)):
+            # it's possible that this symlink points to read-only filesystem with PREMIRROR
+            if os.path.islink(ud.fullmirror):
+                os.unlink(ud.fullmirror)
+
             os.chdir(ud.clonedir)
             logger.info("Creating tarball of git repository")
             runfetchcmd("tar -czf %s %s" % (ud.fullmirror, os.path.join(".") ), d)
Signature · 1 line
1.8.2.1
Signature · 1 line
Martin 'JaMa' Jansa     jabber: [email protected]
Signature · 1 line
Martin 'JaMa' Jansa     jabber: [email protected]
#7 lore raw

Re: [PATCHv3] git: remove symling before updating mirror tarball

In reply to Martin Jansa
Quoted 8 lines · Martin Jansa
Quoted 7 lines · Martin Jansa

* with read-only PREMIRROR (e.g. mounted over NFS or CIFS and referenced as file:///mnt/premirror) we cannot use BB_GENERATE_MIRROR_TARBALLS because all git2_abc.git.tar.gz files later became just symlinks to read-only location in PREMIRROR (it works fine on first build and for new components, because at that time there isn't tarball on PREMIRROR yet).

Quoted 5 lines
Quoted 3 lines

ERROR: Fetcher failure: Fetch command failed with exit code 141, output:
tar (child): /build/downloads/git2_abc.git.tar.gz: Cannot open: Read-only file system tar (child): Error is not recoverable: exiting now

This one is tested and I would like to get it also in 1.16 and 1.18

cherry-picking 3627b02f77c78beedadadd77c619b9e5edaae076 works fine for both, do you prefer to send patches with [1.16]/[1.18] in subject or should I just wait a bit more with patch in master and then ping you later?

Signature · 1 line
Martin 'JaMa' Jansa     jabber: [email protected]
#8 lore raw

Re: [PATCHv3] git: remove symling before updating mirror tarball

Quoted 13 lines · Martin Jansa
Quoted 12 lines · Martin Jansa
Quoted 10 lines · Martin Jansa

* with read-only PREMIRROR (e.g. mounted over NFS or CIFS and referenced as file:///mnt/premirror) we cannot use BB_GENERATE_MIRROR_TARBALLS because all git2_abc.git.tar.gz files later became just symlinks to read-only location in PREMIRROR (it works fine on first build and for new components, because at that time there isn't tarball on PREMIRROR yet).

ERROR: Fetcher failure: Fetch command failed with exit code 141, output:
tar (child): /build/downloads/git2_abc.git.tar.gz: Cannot open: Read-only file system tar (child): Error is not recoverable: exiting now

Quoted 6 lines
Quoted 1 line

This one is tested and I would like to get it also in 1.16 and 1.18

cherry-picking 3627b02f77c78beedadadd77c619b9e5edaae076 works fine for both, do you prefer to send patches with [1.16]/[1.18] in subject or should I just wait a bit more with patch in master and then ping you later?

Just give this a short while, then we can backport. I don't like instantly backporting to other releases...

Cheers,

Richard