sstate: Add a two character subdirectory to the sstate directory layout

Currently all sstate files are placed into one directory. This does not scale and causes a variety of filesystem issues. This patch adds a two character subdirectory to the layout (based on the first two characters of the hash) so that files can be split into several directories.

This should help performance of sstate in most cases by avoding creating directories with huge numbers of files.

The SSTATE_MIRRORS syntax needs updating to account for the extra path element by the addition of a PATH item, for example:

SSTATE_MIRRORS = "file://.* file:///some/path/to/sstate-cache/PATH" SSTATE_MIRRORS = "file://.* http://192.168.1.23/sstate-cache/PATH"

This change also sets the scene for using things like lsb-release in the

Signed-off-by Richard Purdie <[email protected]>

---

meta/classes/sstate.bbclass +15 -12

index 570b371..d00779a 100644
@@ -4,16 +4,21 @@ SSTATE_MANIFESTS ?= "${TMPDIR}/sstate-control"
 SSTATE_MANFILEBASE = "${SSTATE_MANIFESTS}/manifest-${SSTATE_MANMACH}-"
 SSTATE_MANFILEPREFIX = "${SSTATE_MANFILEBASE}${PN}"
 
+def generate_sstatefn(spec, hash, d):
+    if not hash:
+        hash = "INVALID"
+    return hash[:2] + "/" + spec + hash
 
 SSTATE_PKGARCH    = "${PACKAGE_ARCH}"
 SSTATE_PKGSPEC    = "sstate-${PN}-${PACKAGE_ARCH}${TARGET_VENDOR}-${TARGET_OS}-${PV}-${PR}-${SSTATE_PKGARCH}-${SSTATE_VERSION}-"
-SSTATE_PKGNAME    = "${SSTATE_PKGSPEC}${BB_TASKHASH}"
+SSTATE_PKGNAME    = "${@generate_sstatefn(d.getVar('SSTATE_PKGSPEC', True), d.getVar('BB_TASKHASH', True), d)}"
 SSTATE_PKG        = "${SSTATE_DIR}/${SSTATE_PKGNAME}"
+SSTATE_PATHSPEC   = "${SSTATE_DIR}/*/${SSTATE_PKGSPEC}"
 
 SSTATE_SCAN_FILES ?= "*.la *-config *_config"
 SSTATE_SCAN_CMD ?= 'find ${SSTATE_BUILDDIR} \( -name "${@"\" -o -name \"".join(d.getVar("SSTATE_SCAN_FILES", True).split())}" \) -type f'
 
-BB_HASHFILENAME = "${SSTATE_PKGNAME}"
+BB_HASHFILENAME = "${SSTATE_PKGSPEC}"
 
 SSTATE_MANMACH ?= "${SSTATE_PKGARCH}"
 

@@ -158,10 +163,11 @@ def sstate_installpkg(ss, d):
         oe.path.remove(dir)
 
     sstateinst = d.expand("${WORKDIR}/sstate-install-%s/" % ss['name'])
+    sstatefetch = d.getVar('SSTATE_PKGNAME', True) + '_' + ss['name'] + ".tgz"
     sstatepkg = d.getVar('SSTATE_PKG', True) + '_' + ss['name'] + ".tgz"
 
     if not os.path.exists(sstatepkg):
-       pstaging_fetch(sstatepkg, d)
+       pstaging_fetch(sstatefetch, sstatepkg, d)
 
     if not os.path.isfile(sstatepkg):
         bb.note("Staging package %s does not exist" % sstatepkg)

@@ -223,8 +229,7 @@ def sstate_installpkg(ss, d):
 def sstate_clean_cachefile(ss, d):
     import oe.path
 
-    sstatepkgdir = d.getVar('SSTATE_DIR', True)
-    sstatepkgfile = sstatepkgdir + '/' + d.getVar('SSTATE_PKGSPEC', True) + "*_" + ss['name'] + ".tgz*"
+    sstatepkgfile = d.getVar('SSTATE_PATHSPEC', True) + "*_" + ss['name'] + ".tgz*"
     bb.note("Removing %s" % sstatepkgfile)
     oe.path.remove(sstatepkgfile)
 

@@ -417,7 +422,7 @@ def sstate_package(ss, d):
 
     return
 
-def pstaging_fetch(sstatepkg, d):
+def pstaging_fetch(sstatefetch, sstatepkg, d):
     import bb.fetch2
 
     # Only try and fetch if the user has configured a mirror

@@ -430,7 +435,7 @@ def pstaging_fetch(sstatepkg, d):
     bb.data.update_data(localdata)
 
     dldir = localdata.expand("${SSTATE_DIR}")
-    srcuri = "file://" + os.path.basename(sstatepkg)
+    srcuri = "file://" + sstatefetch
 
     bb.mkdirhier(dldir)
 

@@ -519,8 +524,7 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d):
     }
 
     for task in range(len(sq_fn)):
-        sstatefile = d.expand("${SSTATE_DIR}/" + sq_hashfn[task] + "_" + mapping[sq_task[task]] + ".tgz")
-        sstatefile = sstatefile.replace("${BB_TASKHASH}", sq_hash[task])
+        sstatefile = d.expand("${SSTATE_DIR}/" + generate_sstatefn(sq_hashfn[task], sq_hash[task], d) + "_" + mapping[sq_task[task]] + ".tgz")
         if os.path.exists(sstatefile):
             bb.debug(2, "SState: Found valid sstate file %s" % sstatefile)
             ret.append(task)

@@ -544,10 +548,9 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d):
             if task in ret:
                 continue
 
-            sstatefile = d.expand("${SSTATE_DIR}/" + sq_hashfn[task] + "_" + mapping[sq_task[task]] + ".tgz")
-            sstatefile = sstatefile.replace("${BB_TASKHASH}", sq_hash[task])
+            sstatefile = d.expand(generate_sstatefn(sq_hashfn[task], sq_hash[task], d) + "_" + mapping[sq_task[task]] + ".tgz")
 
-            srcuri = "file://" + os.path.basename(sstatefile)
+            srcuri = "file://" + sstatefile
             localdata.setVar('SRC_URI', srcuri)
             bb.debug(2, "SState: Attempting to fetch %s" % srcuri)
 

_______________________________________________
Openembedded-devel mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel

Currently all sstate files are placed into one directory. This does not scale and causes a variety of filesystem issues. This patch adds a two character subdirectory to the layout (based on the first two characters of the hash) so that files can be split into several directories.

This should help performance of sstate in most cases by avoding creating directories with huge numbers of files.

The SSTATE_MIRRORS syntax needs updating to account for the extra path element by the addition of a PATH item, for example:

SSTATE_MIRRORS = "file://.* file:///some/path/to/sstate-cache/PATH" SSTATE_MIRRORS = "file://.* http://192.168.1.23/sstate-cache/PATH"

This change also sets the scene for using things like lsb-release in the

Signed-off-by Richard Purdie <[email protected]>

---

meta/classes/sstate.bbclass +15 -12

index 570b371..d00779a 100644
@@ -4,16 +4,21 @@ SSTATE_MANIFESTS ?= "${TMPDIR}/sstate-control"
 SSTATE_MANFILEBASE = "${SSTATE_MANIFESTS}/manifest-${SSTATE_MANMACH}-"
 SSTATE_MANFILEPREFIX = "${SSTATE_MANFILEBASE}${PN}"
 
+def generate_sstatefn(spec, hash, d):
+    if not hash:
+        hash = "INVALID"
+    return hash[:2] + "/" + spec + hash
 
 SSTATE_PKGARCH    = "${PACKAGE_ARCH}"
 SSTATE_PKGSPEC    = "sstate-${PN}-${PACKAGE_ARCH}${TARGET_VENDOR}-${TARGET_OS}-${PV}-${PR}-${SSTATE_PKGARCH}-${SSTATE_VERSION}-"
-SSTATE_PKGNAME    = "${SSTATE_PKGSPEC}${BB_TASKHASH}"
+SSTATE_PKGNAME    = "${@generate_sstatefn(d.getVar('SSTATE_PKGSPEC', True), d.getVar('BB_TASKHASH', True), d)}"
 SSTATE_PKG        = "${SSTATE_DIR}/${SSTATE_PKGNAME}"
+SSTATE_PATHSPEC   = "${SSTATE_DIR}/*/${SSTATE_PKGSPEC}"
 
 SSTATE_SCAN_FILES ?= "*.la *-config *_config"
 SSTATE_SCAN_CMD ?= 'find ${SSTATE_BUILDDIR} \( -name "${@"\" -o -name \"".join(d.getVar("SSTATE_SCAN_FILES", True).split())}" \) -type f'
 
-BB_HASHFILENAME = "${SSTATE_PKGNAME}"
+BB_HASHFILENAME = "${SSTATE_PKGSPEC}"
 
 SSTATE_MANMACH ?= "${SSTATE_PKGARCH}"
 

@@ -158,10 +163,11 @@ def sstate_installpkg(ss, d):
         oe.path.remove(dir)
 
     sstateinst = d.expand("${WORKDIR}/sstate-install-%s/" % ss['name'])
+    sstatefetch = d.getVar('SSTATE_PKGNAME', True) + '_' + ss['name'] + ".tgz"
     sstatepkg = d.getVar('SSTATE_PKG', True) + '_' + ss['name'] + ".tgz"
 
     if not os.path.exists(sstatepkg):
-       pstaging_fetch(sstatepkg, d)
+       pstaging_fetch(sstatefetch, sstatepkg, d)
 
     if not os.path.isfile(sstatepkg):
         bb.note("Staging package %s does not exist" % sstatepkg)

@@ -223,8 +229,7 @@ def sstate_installpkg(ss, d):
 def sstate_clean_cachefile(ss, d):
     import oe.path
 
-    sstatepkgdir = d.getVar('SSTATE_DIR', True)
-    sstatepkgfile = sstatepkgdir + '/' + d.getVar('SSTATE_PKGSPEC', True) + "*_" + ss['name'] + ".tgz*"
+    sstatepkgfile = d.getVar('SSTATE_PATHSPEC', True) + "*_" + ss['name'] + ".tgz*"
     bb.note("Removing %s" % sstatepkgfile)
     oe.path.remove(sstatepkgfile)
 

@@ -417,7 +422,7 @@ def sstate_package(ss, d):
 
     return
 
-def pstaging_fetch(sstatepkg, d):
+def pstaging_fetch(sstatefetch, sstatepkg, d):
     import bb.fetch2
 
     # Only try and fetch if the user has configured a mirror

@@ -430,7 +435,7 @@ def pstaging_fetch(sstatepkg, d):
     bb.data.update_data(localdata)
 
     dldir = localdata.expand("${SSTATE_DIR}")
-    srcuri = "file://" + os.path.basename(sstatepkg)
+    srcuri = "file://" + sstatefetch
 
     bb.mkdirhier(dldir)
 

@@ -519,8 +524,7 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d):
     }
 
     for task in range(len(sq_fn)):
-        sstatefile = d.expand("${SSTATE_DIR}/" + sq_hashfn[task] + "_" + mapping[sq_task[task]] + ".tgz")
-        sstatefile = sstatefile.replace("${BB_TASKHASH}", sq_hash[task])
+        sstatefile = d.expand("${SSTATE_DIR}/" + generate_sstatefn(sq_hashfn[task], sq_hash[task], d) + "_" + mapping[sq_task[task]] + ".tgz")
         if os.path.exists(sstatefile):
             bb.debug(2, "SState: Found valid sstate file %s" % sstatefile)
             ret.append(task)

@@ -544,10 +548,9 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d):
             if task in ret:
                 continue
 
-            sstatefile = d.expand("${SSTATE_DIR}/" + sq_hashfn[task] + "_" + mapping[sq_task[task]] + ".tgz")
-            sstatefile = sstatefile.replace("${BB_TASKHASH}", sq_hash[task])
+            sstatefile = d.expand(generate_sstatefn(sq_hashfn[task], sq_hash[task], d) + "_" + mapping[sq_task[task]] + ".tgz")
 
-            srcuri = "file://" + os.path.basename(sstatefile)
+            srcuri = "file://" + sstatefile
             localdata.setVar('SRC_URI', srcuri)
             bb.debug(2, "SState: Attempting to fetch %s" % srcuri)
 

_______________________________________________
Openembedded-devel mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel

Quoted 8 lines · Richard Purdie

Currently all sstate files are placed into one directory. This does not scale and causes a variety of filesystem issues. This patch adds a two character subdirectory to the layout (based on the first two characters of the hash) so that files can be split into several directories.

This should help performance of sstate in most cases by avoding creating directories with huge numbers of files.

Quoted 8 lines

The SSTATE_MIRRORS syntax needs updating to account for the extra path element by the addition of a PATH item, for example:

SSTATE_MIRRORS = "file://.* file:///some/path/to/sstate-cache/PATH" SSTATE_MIRRORS = "file://.* http://192.168.1.23/sstate-cache/PATH"

This change also sets the scene for using things like lsb-release in the

Is it possible to create 2nd level cache with this?

I have some server with slow upload but fully populated sstate-cache.

So on server with faster upload which could be used as offical SSTATE_MIRROR for SHR distro I would like to add

SSTATE_MIRRORS ?= "file://.* http://slow-server/sstate-cache/PATH"

And then sync my sstate-cache directory to public accessible web root (with rsync).

Problem is that now sstate-cache has all files in slightly different layout then original sstate-cache on slow server. From what I see I guess it finds URL with correct prefix "sstate-cache/Gentoo-2.1/0d" and downloads it directly to sstate-cache dir (and adds .done)

OE @ ~/oe-core $ ll sstate-cache/sstate-apr-native-x86_64-linux-1.4.6-r1-x86_64-2-*populate-lic*

(quoted diff) +0 -2

-rw-r--r-- 1 bitbake bitbake 9257 Jul 30 12:31 sstate-cache/sstate-apr-native-x86_64-linux-1.4.6-r1-x86_64-2-0d2ed24b90d50bf83e5fe94536596e50_populate-lic.tgz
-rw-r--r-- 1 bitbake bitbake    0 Aug  2 15:40 sstate-cache/sstate-apr-native-x86_64-linux-1.4.6-r1-x86_64-2-0d2ed24b90d50bf83e5fe94536596e50_populate-lic.tgz.done

And then creates symlink in right prefix back to absolute path of sstate-cache/file:
OE @ ~/oe-core $ ll sstate-cache/Gentoo-2.1/0d/sstate-apr-native-x86_64-linux-1.4.6-r1-x86_64-2-*populate-lic*
lrwxrwxrwx 1 bitbake bitbake 123 Aug 2 15:40 sstate-cache/Gentoo-2.1/0d/sstate-apr-native-x86_64-linux-1.4.6-r1-x86_64-2-0d2ed24b90d50bf83e5fe94536596e50_populate-lic.tgz -> /OE/oe-core/sstate-cache/sstate-apr-native-x86_64-linux-1.4.6-r1-x86_64-2-0d2ed24b90d50bf83e5fe94536596e50_populate-lic.tgz

But after sstate-cache directory is rsynced somewhere else and oe-core/sstate-cache is removed, all those symlinks point nowhere and public sstate-cache is unusable.

Can we have relative paths used in symlinks or even instruct fetcher to download that file directly to right prefix?

Quoted 43 lines · Martin Jansa

Quoted 8 lines · Richard Purdie

Currently all sstate files are placed into one directory. This does not scale and causes a variety of filesystem issues. This patch adds a two character subdirectory to the layout (based on the first two characters of the hash) so that files can be split into several directories.

This should help performance of sstate in most cases by avoding creating directories with huge numbers of files.

Quoted 8 lines

The SSTATE_MIRRORS syntax needs updating to account for the extra path element by the addition of a PATH item, for example:

SSTATE_MIRRORS = "file://.* file:///some/path/to/sstate-cache/PATH" SSTATE_MIRRORS = "file://.* http://192.168.1.23/sstate-cache/PATH"

This change also sets the scene for using things like lsb-release in the

Is it possible to create 2nd level cache with this?

I have some server with slow upload but fully populated sstate-cache.

So on server with faster upload which could be used as offical SSTATE_MIRROR for SHR distro I would like to add

SSTATE_MIRRORS ?= "file://.* http://slow-server/sstate-cache/PATH"

And then sync my sstate-cache directory to public accessible web root (with rsync).

Problem is that now sstate-cache has all files in slightly different layout then original sstate-cache on slow server. From what I see I guess it finds URL with correct prefix "sstate-cache/Gentoo-2.1/0d" and downloads it directly to sstate-cache dir (and adds .done)

OE @ ~/oe-core $ ll sstate-cache/sstate-apr-native-x86_64-linux-1.4.6-r1-x86_64-2-*populate-lic*

(quoted diff) +0 -2

-rw-r--r-- 1 bitbake bitbake 9257 Jul 30 12:31 sstate-cache/sstate-apr-native-x86_64-linux-1.4.6-r1-x86_64-2-0d2ed24b90d50bf83e5fe94536596e50_populate-lic.tgz
-rw-r--r-- 1 bitbake bitbake    0 Aug  2 15:40 sstate-cache/sstate-apr-native-x86_64-linux-1.4.6-r1-x86_64-2-0d2ed24b90d50bf83e5fe94536596e50_populate-lic.tgz.done

And then creates symlink in right prefix back to absolute path of sstate-cache/file:
OE @ ~/oe-core $ ll sstate-cache/Gentoo-2.1/0d/sstate-apr-native-x86_64-linux-1.4.6-r1-x86_64-2-*populate-lic*
lrwxrwxrwx 1 bitbake bitbake 123 Aug 2 15:40 sstate-cache/Gentoo-2.1/0d/sstate-apr-native-x86_64-linux-1.4.6-r1-x86_64-2-0d2ed24b90d50bf83e5fe94536596e50_populate-lic.tgz -> /OE/oe-core/sstate-cache/sstate-apr-native-x86_64-linux-1.4.6-r1-x86_64-2-0d2ed24b90d50bf83e5fe94536596e50_populate-lic.tgz

Quoted 5 lines

But after sstate-cache directory is rsynced somewhere else and oe-core/sstate-cache is removed, all those symlinks point nowhere and public sstate-cache is unusable.

Can we have relative paths used in symlinks or even instruct fetcher to download that file directly to right prefix?

2 more ideas:

1) would be great to also download file.sigdata if it exists, to be able to compare them when they change even on machine which downloaded older sstate file from remote url
2) if the reason for this patch was number of files in shared sstate-cache directory, then fetcher creating .done files makes number double too (would be fine if fetcher stores all 3 files (.tgz, .tgz.sigdata, .tgz.done) in right prefix, or moves them to right prefix instead of symlinks.

Quoted 52 lines · Martin Jansa

Quoted 43 lines · Martin Jansa

Quoted 8 lines · Richard Purdie

Currently all sstate files are placed into one directory. This does not scale and causes a variety of filesystem issues. This patch adds a two character subdirectory to the layout (based on the first two characters of the hash) so that files can be split into several directories.

This should help performance of sstate in most cases by avoding creating directories with huge numbers of files.

Quoted 8 lines

The SSTATE_MIRRORS syntax needs updating to account for the extra path element by the addition of a PATH item, for example:

SSTATE_MIRRORS = "file://.* file:///some/path/to/sstate-cache/PATH" SSTATE_MIRRORS = "file://.* http://192.168.1.23/sstate-cache/PATH"

This change also sets the scene for using things like lsb-release in the

Is it possible to create 2nd level cache with this?

I have some server with slow upload but fully populated sstate-cache.

So on server with faster upload which could be used as offical SSTATE_MIRROR for SHR distro I would like to add

SSTATE_MIRRORS ?= "file://.* http://slow-server/sstate-cache/PATH"

And then sync my sstate-cache directory to public accessible web root (with rsync).

Problem is that now sstate-cache has all files in slightly different layout then original sstate-cache on slow server. From what I see I guess it finds URL with correct prefix "sstate-cache/Gentoo-2.1/0d" and downloads it directly to sstate-cache dir (and adds .done)

OE @ ~/oe-core $ ll sstate-cache/sstate-apr-native-x86_64-linux-1.4.6-r1-x86_64-2-*populate-lic*

(quoted diff) +0 -2

-rw-r--r-- 1 bitbake bitbake 9257 Jul 30 12:31 sstate-cache/sstate-apr-native-x86_64-linux-1.4.6-r1-x86_64-2-0d2ed24b90d50bf83e5fe94536596e50_populate-lic.tgz
-rw-r--r-- 1 bitbake bitbake    0 Aug  2 15:40 sstate-cache/sstate-apr-native-x86_64-linux-1.4.6-r1-x86_64-2-0d2ed24b90d50bf83e5fe94536596e50_populate-lic.tgz.done

And then creates symlink in right prefix back to absolute path of sstate-cache/file:
OE @ ~/oe-core $ ll sstate-cache/Gentoo-2.1/0d/sstate-apr-native-x86_64-linux-1.4.6-r1-x86_64-2-*populate-lic*
lrwxrwxrwx 1 bitbake bitbake 123 Aug 2 15:40 sstate-cache/Gentoo-2.1/0d/sstate-apr-native-x86_64-linux-1.4.6-r1-x86_64-2-0d2ed24b90d50bf83e5fe94536596e50_populate-lic.tgz -> /OE/oe-core/sstate-cache/sstate-apr-native-x86_64-linux-1.4.6-r1-x86_64-2-0d2ed24b90d50bf83e5fe94536596e50_populate-lic.tgz

Quoted 5 lines

But after sstate-cache directory is rsynced somewhere else and oe-core/sstate-cache is removed, all those symlinks point nowhere and public sstate-cache is unusable.

Can we have relative paths used in symlinks or even instruct fetcher to download that file directly to right prefix?

2 more ideas:

Quoted 8 lines

1) would be great to also download file.sigdata if it exists, to be able to compare them when they change even on machine which downloaded older sstate file from remote url
2) if the reason for this patch was number of files in shared sstate-cache directory, then fetcher creating .done files makes number double too (would be fine if fetcher stores all 3 files (.tgz, .tgz.sigdata, .tgz.done) in right prefix, or moves them to right prefix instead of symlinks.

I'm aware of the problem. The main issue is that we probably need to start enforcing complete paths for all downloads in DL_DIR, including
http:// urls. This would resolve conflicts like:

SRC_URI = "http://server1.org/somefile.patch \
http://server2.org/somefile.patch"

where the two files are different. The trouble is it will pretty much break all the source mirrors :(.

Quoted 19 lines · Richard Purdie

Quoted 10 lines · Martin Jansa

2 more ideas:

1) would be great to also download file.sigdata if it exists, to be able to compare them when they change even on machine which downloaded older sstate file from remote url
2) if the reason for this patch was number of files in shared sstate-cache directory, then fetcher creating .done files makes number double too (would be fine if fetcher stores all 3 files (.tgz, .tgz.sigdata, .tgz.done) in right prefix, or moves them to right prefix instead of symlinks.

I'm aware of the problem. The main issue is that we probably need to start enforcing complete paths for all downloads in DL_DIR, including
http:// urls. This would resolve conflicts like:

SRC_URI = "http://server1.org/somefile.patch \
http://server2.org/somefile.patch"

Quoted 2 lines

where the two files are different. The trouble is it will pretty much break all the source mirrors :(.

I think we need to stop the tendency to use DL_DIR as is as a mirror, and instead create a task or something to populate a mirror directory from the DL_DIR. This would avoid potential issues with licensing if it uses license filtering to control what gets populated, as well.

Quoted 52 lines · Martin Jansa

Quoted 43 lines · Martin Jansa

Quoted 8 lines · Richard Purdie

Currently all sstate files are placed into one directory. This does not scale and causes a variety of filesystem issues. This patch adds a two character subdirectory to the layout (based on the first two characters of the hash) so that files can be split into several directories.

This should help performance of sstate in most cases by avoding creating directories with huge numbers of files.

Quoted 8 lines

The SSTATE_MIRRORS syntax needs updating to account for the extra path element by the addition of a PATH item, for example:

SSTATE_MIRRORS = "file://.* file:///some/path/to/sstate-cache/PATH" SSTATE_MIRRORS = "file://.* http://192.168.1.23/sstate-cache/PATH"

This change also sets the scene for using things like lsb-release in the

Is it possible to create 2nd level cache with this?

I have some server with slow upload but fully populated sstate-cache.

So on server with faster upload which could be used as offical SSTATE_MIRROR for SHR distro I would like to add

SSTATE_MIRRORS ?= "file://.* http://slow-server/sstate-cache/PATH"

And then sync my sstate-cache directory to public accessible web root (with rsync).

Problem is that now sstate-cache has all files in slightly different layout then original sstate-cache on slow server. From what I see I guess it finds URL with correct prefix "sstate-cache/Gentoo-2.1/0d" and downloads it directly to sstate-cache dir (and adds .done)

OE @ ~/oe-core $ ll sstate-cache/sstate-apr-native-x86_64-linux-1.4.6-r1-x86_64-2-*populate-lic*

(quoted diff) +0 -2

-rw-r--r-- 1 bitbake bitbake 9257 Jul 30 12:31 sstate-cache/sstate-apr-native-x86_64-linux-1.4.6-r1-x86_64-2-0d2ed24b90d50bf83e5fe94536596e50_populate-lic.tgz
-rw-r--r-- 1 bitbake bitbake    0 Aug  2 15:40 sstate-cache/sstate-apr-native-x86_64-linux-1.4.6-r1-x86_64-2-0d2ed24b90d50bf83e5fe94536596e50_populate-lic.tgz.done

And then creates symlink in right prefix back to absolute path of sstate-cache/file:
OE @ ~/oe-core $ ll sstate-cache/Gentoo-2.1/0d/sstate-apr-native-x86_64-linux-1.4.6-r1-x86_64-2-*populate-lic*
lrwxrwxrwx 1 bitbake bitbake 123 Aug 2 15:40 sstate-cache/Gentoo-2.1/0d/sstate-apr-native-x86_64-linux-1.4.6-r1-x86_64-2-0d2ed24b90d50bf83e5fe94536596e50_populate-lic.tgz -> /OE/oe-core/sstate-cache/sstate-apr-native-x86_64-linux-1.4.6-r1-x86_64-2-0d2ed24b90d50bf83e5fe94536596e50_populate-lic.tgz

Quoted 5 lines

But after sstate-cache directory is rsynced somewhere else and oe-core/sstate-cache is removed, all those symlinks point nowhere and public sstate-cache is unusable.

Can we have relative paths used in symlinks or even instruct fetcher to download that file directly to right prefix?

2 more ideas:

Quoted 8 lines

1) would be great to also download file.sigdata if it exists, to be able to compare them when they change even on machine which downloaded older sstate file from remote url
2) if the reason for this patch was number of files in shared sstate-cache directory, then fetcher creating .done files makes number double too (would be fine if fetcher stores all 3 files (.tgz, .tgz.sigdata, .tgz.done) in right prefix, or moves them to right prefix instead of symlinks.

I'm aware of the problem. The main issue is that we probably need to

start enforcing complete paths for all downloads in DL_DIR, including
http:// urls. This would resolve conflicts like:

SRC_URI = "http://server1.org/somefile.patch \
http://server2.org/somefile.patch"

where the two files are different. The trouble is it will pretty much break all the source mirrors :(.

Quoted 62 lines · Richard Purdie

Quoted 52 lines · Martin Jansa

Quoted 43 lines · Martin Jansa

Quoted 8 lines · Richard Purdie

Currently all sstate files are placed into one directory. This does not scale and causes a variety of filesystem issues. This patch adds a two character subdirectory to the layout (based on the first two characters of the hash) so that files can be split into several directories.

This should help performance of sstate in most cases by avoding creating directories with huge numbers of files.

Quoted 8 lines

The SSTATE_MIRRORS syntax needs updating to account for the extra path element by the addition of a PATH item, for example:

SSTATE_MIRRORS = "file://.* file:///some/path/to/sstate-cache/PATH" SSTATE_MIRRORS = "file://.* http://192.168.1.23/sstate-cache/PATH"

This change also sets the scene for using things like lsb-release in the

Is it possible to create 2nd level cache with this?

I have some server with slow upload but fully populated sstate-cache.

So on server with faster upload which could be used as offical SSTATE_MIRROR for SHR distro I would like to add

SSTATE_MIRRORS ?= "file://.* http://slow-server/sstate-cache/PATH"

And then sync my sstate-cache directory to public accessible web root (with rsync).

Problem is that now sstate-cache has all files in slightly different layout then original sstate-cache on slow server. From what I see I guess it finds URL with correct prefix "sstate-cache/Gentoo-2.1/0d" and downloads it directly to sstate-cache dir (and adds .done)

OE @ ~/oe-core $ ll sstate-cache/sstate-apr-native-x86_64-linux-1.4.6-r1-x86_64-2-*populate-lic*

(quoted diff) +0 -2

-rw-r--r-- 1 bitbake bitbake 9257 Jul 30 12:31 sstate-cache/sstate-apr-native-x86_64-linux-1.4.6-r1-x86_64-2-0d2ed24b90d50bf83e5fe94536596e50_populate-lic.tgz
-rw-r--r-- 1 bitbake bitbake    0 Aug  2 15:40 sstate-cache/sstate-apr-native-x86_64-linux-1.4.6-r1-x86_64-2-0d2ed24b90d50bf83e5fe94536596e50_populate-lic.tgz.done

And then creates symlink in right prefix back to absolute path of sstate-cache/file:
OE @ ~/oe-core $ ll sstate-cache/Gentoo-2.1/0d/sstate-apr-native-x86_64-linux-1.4.6-r1-x86_64-2-*populate-lic*
lrwxrwxrwx 1 bitbake bitbake 123 Aug 2 15:40 sstate-cache/Gentoo-2.1/0d/sstate-apr-native-x86_64-linux-1.4.6-r1-x86_64-2-0d2ed24b90d50bf83e5fe94536596e50_populate-lic.tgz -> /OE/oe-core/sstate-cache/sstate-apr-native-x86_64-linux-1.4.6-r1-x86_64-2-0d2ed24b90d50bf83e5fe94536596e50_populate-lic.tgz

Quoted 5 lines

But after sstate-cache directory is rsynced somewhere else and oe-core/sstate-cache is removed, all those symlinks point nowhere and public sstate-cache is unusable.

Can we have relative paths used in symlinks or even instruct fetcher to download that file directly to right prefix?

2 more ideas:

Quoted 8 lines

1) would be great to also download file.sigdata if it exists, to be able to compare them when they change even on machine which downloaded older sstate file from remote url
2) if the reason for this patch was number of files in shared sstate-cache directory, then fetcher creating .done files makes number double too (would be fine if fetcher stores all 3 files (.tgz, .tgz.sigdata, .tgz.done) in right prefix, or moves them to right prefix instead of symlinks.

Quoted 1 line

I'm aware of the problem. The main issue is that we probably need to

And what about .sigdata files?

I have sort shell script to replace symlinks with real files in prefixed dirs, would it be worth it integrating to openembedded-core/scripts/sstate-cache-management.sh
which doesn't work with new layout anyway?

Quoted 5 lines

start enforcing complete paths for all downloads in DL_DIR, including
http:// urls. This would resolve conflicts like:

SRC_URI = "http://server1.org/somefile.patch \
http://server2.org/somefile.patch"

In two separate recipes right?

Quoted 2 lines

where the two files are different. The trouble is it will pretty much break all the source mirrors :(.

So you would store them in DL_DIR/server1.org/somefile.patch path?

That would make oposite scenario where the BIG.tgz is available (or even requested by different recipes) from different location less efficient.

And not creating .done files for local files fetched from file:// whould also help for:

foo.bb: SRC_URI = "file://somefile.patch" bar.bb: SRC_URI = "http://server2.org/somefile.patch"

Which now ignores checksums for samefile.patch downloaded for bar.bb if foo.bb was built before.

Quoted 63 lines · Martin Jansa

Quoted 62 lines · Richard Purdie

Quoted 60 lines · Martin Jansa

Quoted 43 lines · Martin Jansa

Quoted 8 lines · Richard Purdie

Currently all sstate files are placed into one directory. This does not scale and causes a variety of filesystem issues. This patch adds a two character subdirectory to the layout (based on the first two characters of the hash) so that files can be split into several directories.

This should help performance of sstate in most cases by avoding creating directories with huge numbers of files.

Quoted 8 lines

The SSTATE_MIRRORS syntax needs updating to account for the extra path element by the addition of a PATH item, for example:

SSTATE_MIRRORS = "file://.* file:///some/path/to/sstate-cache/PATH" SSTATE_MIRRORS = "file://.* http://192.168.1.23/sstate-cache/PATH"

This change also sets the scene for using things like lsb-release in the

Is it possible to create 2nd level cache with this?

I have some server with slow upload but fully populated sstate-cache.

So on server with faster upload which could be used as offical SSTATE_MIRROR for SHR distro I would like to add

SSTATE_MIRRORS ?= "file://.* http://slow-server/sstate-cache/PATH"

And then sync my sstate-cache directory to public accessible web root (with rsync).

Problem is that now sstate-cache has all files in slightly different layout then original sstate-cache on slow server. From what I see I guess it finds URL with correct prefix "sstate-cache/Gentoo-2.1/0d" and downloads it directly to sstate-cache dir (and adds .done)

OE @ ~/oe-core $ ll sstate-cache/sstate-apr-native-x86_64-linux-1.4.6-r1-x86_64-2-*populate-lic*

(quoted diff) +0 -2

-rw-r--r-- 1 bitbake bitbake 9257 Jul 30 12:31 sstate-cache/sstate-apr-native-x86_64-linux-1.4.6-r1-x86_64-2-0d2ed24b90d50bf83e5fe94536596e50_populate-lic.tgz
-rw-r--r-- 1 bitbake bitbake    0 Aug  2 15:40 sstate-cache/sstate-apr-native-x86_64-linux-1.4.6-r1-x86_64-2-0d2ed24b90d50bf83e5fe94536596e50_populate-lic.tgz.done

And then creates symlink in right prefix back to absolute path of sstate-cache/file:
OE @ ~/oe-core $ ll sstate-cache/Gentoo-2.1/0d/sstate-apr-native-x86_64-linux-1.4.6-r1-x86_64-2-*populate-lic*
lrwxrwxrwx 1 bitbake bitbake 123 Aug 2 15:40 sstate-cache/Gentoo-2.1/0d/sstate-apr-native-x86_64-linux-1.4.6-r1-x86_64-2-0d2ed24b90d50bf83e5fe94536596e50_populate-lic.tgz -> /OE/oe-core/sstate-cache/sstate-apr-native-x86_64-linux-1.4.6-r1-x86_64-2-0d2ed24b90d50bf83e5fe94536596e50_populate-lic.tgz

Quoted 5 lines

But after sstate-cache directory is rsynced somewhere else and oe-core/sstate-cache is removed, all those symlinks point nowhere and public sstate-cache is unusable.

Can we have relative paths used in symlinks or even instruct fetcher to download that file directly to right prefix?

2 more ideas:

1) would be great to also download file.sigdata if it exists, to be able to compare them when they change even on machine which downloaded older sstate file from remote url
2) if the reason for this patch was number of files in shared sstate-cache directory, then fetcher creating .done files makes number double too (would be fine if fetcher stores all 3 files (.tgz, .tgz.sigdata, .tgz.done) in right prefix, or moves them to right prefix instead of symlinks.

Quoted 3 lines

Quoted 1 line

I'm aware of the problem. The main issue is that we probably need to

And what about .sigdata files?

Quoted 4 lines

I have sort shell script to replace symlinks with real files in prefixed dirs, would it be worth it integrating to openembedded-core/scripts/sstate-cache-management.sh
which doesn't work with new layout anyway?

Quoted 6 lines

Quoted 5 lines

start enforcing complete paths for all downloads in DL_DIR, including
http:// urls. This would resolve conflicts like:

SRC_URI = "http://server1.org/somefile.patch \
http://server2.org/somefile.patch"

Quoted 6 lines

In two separate recipes right?

Quoted 2 lines

where the two files are different. The trouble is it will pretty much break all the source mirrors :(.

So you would store them in DL_DIR/server1.org/somefile.patch path?

I've wondered about:

DL_DIR/server1.org/somepath/somefile.patch

Quoted 3 lines

That would make oposite scenario where the BIG.tgz is available (or even requested by different recipes) from different location less efficient.

Not necessarily with the right mirror/premirror configuration.

Quoted 8 lines

And not creating .done files for local files fetched from file:// whould also help for:

foo.bb: SRC_URI = "file://somefile.patch" bar.bb: SRC_URI = "http://server2.org/somefile.patch"

Which now ignores checksums for samefile.patch downloaded for bar.bb if foo.bb was built before.

That is a pain but we've basically always assumed no namespace collision. I'm not saying that is a good thing, just the way it is.

Not creating done files for local urls causes a variety of problems, not least that you then have to special case local urls in the generic fetcher code, it also hits performance. I've been trying to get the fetcher away from a set of special cases...

Cheers,

Richard

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core