Shared state for all !

The optional parameter should be an octal file mode representing the desired permissions of the created directory. If mode isn;t specified use 0777. This is the default of os.makedirs() yet is not often what the created directory ends up with due to umask affecting the call.

The use of chmod() in this patch ensures that whatever octal is passed are the permissions the directory will have after creation. This is a behaviour change...

Signed-off-by Joshua Lock <[email protected]>

---
 bitbake/lib/bb/utils.py |    7 +++++--
 1 files changed, 5 insertions(+), 2 deletions(-)

bitbake/lib/bb/utils.py +1 -1

index 7a73419..33206fe 100644
@@ -531,13 +531,16 @@ def prune_suffix(var, suffixes, d):
             return var.replace(suffix, "")
     return var
 
-def mkdirhier(directory):
+def mkdirhier(directory, mode=0777):
     """Create a directory like 'mkdir -p', but does not complain if
     directory already exists like os.makedirs
     """

Create signature files, and the directories which contain them, with rwx for everyone so that they are easily shared with different users of the same machine.

Signed-off-by Joshua Lock <[email protected]>

---
 bitbake/lib/bb/siggen.py |    7 ++++++-
 1 files changed, 6 insertions(+), 1 deletions(-)

bitbake/lib/bb/siggen.py +0 -0

index 8c79b17..bd6d59b 100644
@@ -201,7 +201,12 @@ class SignatureGeneratorBasic(SignatureGenerator):
             for dep in data['runtaskdeps']:

Quoted 17 lines · Joshua Lock

Create signature files, and the directories which contain them, with rwx for everyone so that they are easily shared with different users of the same machine.

Signed-off-by Joshua Lock<[email protected]>

---
  bitbake/lib/bb/siggen.py |    7 ++++++-
  1 files changed, 6 insertions(+), 1 deletions(-)

bitbake/lib/bb/siggen.py +0 -0

index 8c79b17..bd6d59b 100644
@@ -201,7 +201,12 @@ class SignatureGeneratorBasic(SignatureGenerator):
              for dep in data['runtaskdeps']:
                  data['runtaskhashes'][dep] = self.taskhash[dep]
 

bitbake/lib/bb/siggen.py class SignatureGeneratorBasic(SignatureGenerator): · line 210

Quoted 8 lines

bitbake/lib/bb/siggen.py +6 -1

-        p = pickle.Pickler(file(sigfile, "wb"), -1)
+        # Create file with permissive (0777) read/write for easier sharing
+        f = os.fdopen(os.open(sigfile, os.O_RDWR|os.O_CREAT), "wb")
+        # os.open() and os.fdopen() are affected by the users umask so brute force
+        # the permissions with a call to chmod
+        os.chmod(sigfile, 0777)
+        p = pickle.Pickler(f, -1)
          p.dump(data)

Why not just run the chmod after the original pickle code? Its not as if you avoid a race this way :/.

In order to make sstate cache's more easily shared ensure any user of the system has rwx permission by calling chown on sstate files after they're created.

Signed-off-by Joshua Lock<[email protected]>

---
  meta/classes/sstate.bbclass |    1 +
  1 files changed, 1 insertions(+), 0 deletions(-)

meta/classes/sstate.bbclass +0 -0

index a8c98e5..6707ecf 100644

Quoted 20 lines · Joshua Lock

In order to make sstate cache's more easily shared ensure any user of the system has rwx permission by calling chown on sstate files after they're created.

Signed-off-by Joshua Lock<[email protected]>

---
meta/classes/sstate.bbclass | 1 +

1 files changed, 1 insertions(+), 0 deletions(-)

meta/classes/sstate.bbclass +0 -0

index a8c98e5..6707ecf 100644
@@ -454,6 +454,7 @@ sstate_create_package () {

else
tar -cz --file=$TFILE --files-from=/dev/null
fi
+ chmod 0777 $TFILE
mv $TFILE ${SSTATE_PKG}

Quoted 1 line

cd ${WORKDIR}

Why execute permission, and should it not be restricted to 664 for group level write access? Why would multiple users be writing to the same sstate file anyways once it's there it could be read-only since a change will trigger a new sstate file, not a re-write of the existing one.

In Yocto #2041[2]  Mark reported an issue with reusing shared state as a different user on the same machine.

Since the whole purpose of shared state is that it be shared I decided to dig into this issue. I wanted to at least be able to use the shared-state cache of a different user without error, even if all of the objects aren't actually used (i.e. native, at least on the Edison branch I did most of the testing with).

This is an RFC mainly because it changes the permissions of created directories, sstate files and siginfo files from what they have traditionally been.

There is more of the rhyme an reason in the patch commit headers and comments but tl;dr bb.mkdirhier directories will be 0777 (rwxrwxrwx) with this patch, as will all of the contents of sstate-cache (siginfo and tgz) files.

This is actually what one would expect from reading the Python API docs for os.makedirs "The default mode is 0777 (octal)."[1] but not what actually happens on most modern Linux systems thanks to umask.

Please review the following changes for suitability for inclusion. If you have any objections or suggestions for improvement, please respond to the patches. If you agree with the changes, please provide your Acked-by.

Quoted 19 lines · Joshua Lock

In Yocto #2041[2] Mark reported an issue with reusing shared state as a different user on the same machine.

Since the whole purpose of shared state is that it be shared I decided to dig into this issue. I wanted to at least be able to use the shared-state cache of a different user without error, even if all of the objects aren't actually used (i.e. native, at least on the Edison branch I did most of the testing with).

This is an RFC mainly because it changes the permissions of created directories, sstate files and siginfo files from what they have traditionally been.

There is more of the rhyme an reason in the patch commit headers and comments but tl;dr bb.mkdirhier directories will be 0777 (rwxrwxrwx) with this patch, as will all of the contents of sstate-cache (siginfo and tgz) files.

This is actually what one would expect from reading the Python API docs for os.makedirs "The default mode is 0777 (octal)."[1] but not what actually happens on most modern Linux systems thanks to umask.

Quoted 3 lines

Please review the following changes for suitability for inclusion. If you have any objections or suggestions for improvement, please respond to the patches. If you agree with the changes, please provide your Acked-by.

777 seems questionable to me, personally. Generally collaboration happens amongst folks within a group, and chmod g+s makes that easier. I'd expect 775 to be a more sane value, myself.

Quoted 38 lines · Chris Larson

Quoted 29 lines · Joshua Lock

In Yocto #2041[2]  Mark reported an issue with reusing shared state as a different user on the same machine.

Since the whole purpose of shared state is that it be shared I decided to dig
into this issue. I wanted to at least be able to use the shared-state cache of
a different user without error, even if all of the objects aren't actually used
(i.e. native, at least on the Edison branch I did most of the testing with).

This is an RFC mainly because it changes the permissions of created directories,
sstate files and siginfo files from what they have traditionally been.

There is more of the rhyme an reason in the patch commit headers and comments
but tl;dr bb.mkdirhier directories will be 0777 (rwxrwxrwx) with this patch, as
will all of the contents of sstate-cache (siginfo and tgz) files.

This is actually what one would expect from reading the Python API docs for
os.makedirs "The default mode is 0777 (octal)."[1] but not what actually happens
on most modern Linux systems thanks to umask.

Quoted 5 lines

Please review the following changes for suitability for inclusion. If you have
any objections or suggestions for improvement, please respond to the patches. If
you agree with the changes, please provide your Acked-by.

Quoted 3 lines

777 seems questionable to me, personally. Generally collaboration happens amongst folks within a group, and chmod g+s makes that easier. I'd expect 775 to be a more sane value, myself.

Do you mean for bb.mkdirhier calls, the tgz files, the siginfo files or everything?

I went with 777 for mkdirhier as that's the default of os.makedirs before umask is involved. I would likely have picked rw-rw-r-- (664) if I weren't trying to request comments.

Quoted 46 lines · Joshua Lock

Quoted 38 lines · Chris Larson

Quoted 29 lines · Joshua Lock

In Yocto #2041[2] Mark reported an issue with reusing shared state as a different user on the same machine.

Since the whole purpose of shared state is that it be shared I decided to dig
into this issue. I wanted to at least be able to use the shared-state cache of
a different user without error, even if all of the objects aren't actually used
(i.e. native, at least on the Edison branch I did most of the testing with).

This is an RFC mainly because it changes the permissions of created directories,
sstate files and siginfo files from what they have traditionally been.

There is more of the rhyme an reason in the patch commit headers and comments
but tl;dr bb.mkdirhier directories will be 0777 (rwxrwxrwx) with this patch, as
will all of the contents of sstate-cache (siginfo and tgz) files.

This is actually what one would expect from reading the Python API docs for
os.makedirs "The default mode is 0777 (octal)."[1] but not what actually happens
on most modern Linux systems thanks to umask.

Quoted 5 lines

Please review the following changes for suitability for inclusion. If you have
any objections or suggestions for improvement, please respond to the patches. If
you agree with the changes, please provide your Acked-by.

Quoted 3 lines

777 seems questionable to me, personally. Generally collaboration happens amongst folks within a group, and chmod g+s makes that easier. I'd expect 775 to be a more sane value, myself.

Quoted 6 lines

Do you mean for bb.mkdirhier calls, the tgz files, the siginfo files or everything?

I went with 777 for mkdirhier as that's the default of os.makedirs before umask is involved. I would likely have picked rw-rw-r-- (664) if I weren't trying to request comments.

Gotcha.

I'm concerned about the behavior change and potential implications of changing the default behavior of mkdirhier. I'm inclined to say that when you don't pass mode, let it use the current behavior of obeying the umask.

default behavior, then I think 777 is the wrong/questionable default. Beyond that, 777 is certainly the wrong mode to be using for the shared state package in sstate.bbclass.

Quoted 62 lines · Chris Larson

Quoted 52 lines · Joshua Lock

Quoted 44 lines · Chris Larson

Quoted 33 lines · Joshua Lock

In Yocto #2041[2] Mark reported an issue with reusing shared state as a
different user on the same machine.

Since the whole purpose of shared state is that it be shared I decided to
dig
into this issue. I wanted to at least be able to use the shared-state cache of
a different user without error, even if all of the objects aren't actually used
(i.e. native, at least on the Edison branch I did most of the testing with).

This is an RFC mainly because it changes the permissions of created directories,
sstate files and siginfo files from what they have traditionally been.

There is more of the rhyme an reason in the patch commit headers and comments
but tl;dr bb.mkdirhier directories will be 0777 (rwxrwxrwx) with this patch, as
will all of the contents of sstate-cache (siginfo and tgz) files.

This is actually what one would expect from reading the Python API docs
for
os.makedirs "The default mode is 0777 (octal)."[1] but not what actually
happens
on most modern Linux systems thanks to umask.

Quoted 6 lines

Please review the following changes for suitability for inclusion. If you
have
any objections or suggestions for improvement, please respond to the patches. If
you agree with the changes, please provide your Acked-by.

Quoted 3 lines

777 seems questionable to me, personally. Generally collaboration happens amongst folks within a group, and chmod g+s makes that easier. I'd expect 775 to be a more sane value, myself.

Quoted 8 lines

Do you mean for bb.mkdirhier calls, the tgz files, the siginfo files or everything?

I went with 777 for mkdirhier as that's the default of os.makedirs before
umask is involved. I would likely have picked rw-rw-r-- (664) if I weren't
trying to request comments.

Quoted 6 lines

Gotcha.

I'm concerned about the behavior change and potential implications of changing the default behavior of mkdirhier. I'm inclined to say that when you don't pass mode, let it use the current behavior of obeying the umask.

An earlier version of the series did this and I'm happy to add that behaviour back in.

If we're not going to do that, and want to change the

Quoted 3 lines

default behavior, then I think 777 is the wrong/questionable default. Beyond that, 777 is certainly the wrong mode to be using for the shared state package in sstate.bbclass.

Do you have a strong preference on 664 vs. 775 ?

Quoted 79 lines · Joshua Lock

Quoted 63 lines · Chris Larson

Quoted 53 lines · Joshua Lock

Quoted 45 lines · Chris Larson

Quoted 34 lines · Joshua Lock

In Yocto #2041[2] Mark reported an issue with reusing shared state as a
different user on the same machine.

Since the whole purpose of shared state is that it be shared I decided to
dig
into this issue. I wanted to at least be able to use the shared-state cache of
a different user without error, even if all of the objects aren't actually used
(i.e. native, at least on the Edison branch I did most of the testing with).

This is an RFC mainly because it changes the permissions of created directories,
sstate files and siginfo files from what they have traditionally been.

There is more of the rhyme an reason in the patch commit headers and comments
but tl;dr bb.mkdirhier directories will be 0777 (rwxrwxrwx) with this patch, as
will all of the contents of sstate-cache (siginfo and tgz) files.

This is actually what one would expect from reading the Python API docs
for
os.makedirs "The default mode is 0777 (octal)."[1] but not what actually
happens
on most modern Linux systems thanks to umask.

Quoted 6 lines

Please review the following changes for suitability for inclusion. If you
have
any objections or suggestions for improvement, please respond to the patches. If
you agree with the changes, please provide your Acked-by.

Quoted 3 lines

777 seems questionable to me, personally. Generally collaboration happens amongst folks within a group, and chmod g+s makes that easier. I'd expect 775 to be a more sane value, myself.

Quoted 8 lines

Do you mean for bb.mkdirhier calls, the tgz files, the siginfo files or everything?

I went with 777 for mkdirhier as that's the default of os.makedirs before
umask is involved. I would likely have picked rw-rw-r-- (664) if I weren't
trying to request comments.

Quoted 6 lines

Gotcha.

I'm concerned about the behavior change and potential implications of changing the default behavior of mkdirhier. I'm inclined to say that when you don't pass mode, let it use the current behavior of obeying the umask.

An earlier version of the series did this and I'm happy to add that behaviour back in.

If we're not going to do that, and want to change the

Quoted 3 lines

default behavior, then I think 777 is the wrong/questionable default. Beyond that, 777 is certainly the wrong mode to be using for the shared state package in sstate.bbclass.

Quoted 1 line

Do you have a strong preference on 664 vs. 775 ?

I'm leaning towards 664 (rw-rw-r--) for the files and 775 (rwxrwxr-x) for directories, these are the defaults for file and directory creation on Ubuntu 12.04 and Fedora 16.

Quoted 95 lines · Joshua Lock

Quoted 91 lines · Joshua Lock

Quoted 70 lines · Chris Larson

Quoted 58 lines · Joshua Lock

Quoted 48 lines · Chris Larson

Quoted 35 lines · Joshua Lock

In Yocto #2041[2] Mark reported an issue with reusing shared state as a
different user on the same machine.

Since the whole purpose of shared state is that it be shared I decided to
dig
into this issue. I wanted to at least be able to use the shared-state cache of
a different user without error, even if all of the objects aren't actually used
(i.e. native, at least on the Edison branch I did most of the testing with).

This is an RFC mainly because it changes the permissions of created directories,
sstate files and siginfo files from what they have traditionally been.

There is more of the rhyme an reason in the patch commit headers and comments
but tl;dr bb.mkdirhier directories will be 0777 (rwxrwxrwx) with this patch, as
will all of the contents of sstate-cache (siginfo and tgz) files.

This is actually what one would expect from reading the Python API docs
for
os.makedirs "The default mode is 0777 (octal)."[1] but not what actually
happens
on most modern Linux systems thanks to umask.

Quoted 6 lines

Please review the following changes for suitability for inclusion. If you
have
any objections or suggestions for improvement, please respond to the patches. If
you agree with the changes, please provide your Acked-by.

Quoted 3 lines

777 seems questionable to me, personally. Generally collaboration happens amongst folks within a group, and chmod g+s makes that easier. I'd expect 775 to be a more sane value, myself.

Quoted 8 lines

Do you mean for bb.mkdirhier calls, the tgz files, the siginfo files or everything?

I went with 777 for mkdirhier as that's the default of os.makedirs before
umask is involved. I would likely have picked rw-rw-r-- (664) if I weren't
trying to request comments.

Quoted 6 lines

Gotcha.

I'm concerned about the behavior change and potential implications of changing the default behavior of mkdirhier. I'm inclined to say that when you don't pass mode, let it use the current behavior of obeying the umask.

An earlier version of the series did this and I'm happy to add that behaviour back in.

If we're not going to do that, and want to change the

Quoted 4 lines

default behavior, then I think 777 is the wrong/questionable default. Beyond that, 777 is certainly the wrong mode to be using for the shared state package in sstate.bbclass.

Do you have a strong preference on 664 vs. 775 ?

Quoted 3 lines

I'm leaning towards 664 (rw-rw-r--) for the files and 775 (rwxrwxr-x) for directories, these are the defaults for file and directory creation on Ubuntu 12.04 and Fedora 16.

At which point, the obvious "solution" to the bug report is a sanity check whether the user can read and write to the sstate directory.

Quoted 96 lines · Joshua Lock

Quoted 95 lines · Joshua Lock

Quoted 91 lines · Joshua Lock

Quoted 70 lines · Chris Larson

Quoted 58 lines · Joshua Lock

Quoted 48 lines · Chris Larson

Quoted 35 lines · Joshua Lock

In Yocto #2041[2] Mark reported an issue with reusing shared state as a
different user on the same machine.

Since the whole purpose of shared state is that it be shared I decided to
dig
into this issue. I wanted to at least be able to use the shared-state cache of
a different user without error, even if all of the objects aren't actually used
(i.e. native, at least on the Edison branch I did most of the testing with).

This is an RFC mainly because it changes the permissions of created directories,
sstate files and siginfo files from what they have traditionally been.

There is more of the rhyme an reason in the patch commit headers and comments
but tl;dr bb.mkdirhier directories will be 0777 (rwxrwxrwx) with this patch, as
will all of the contents of sstate-cache (siginfo and tgz) files.

This is actually what one would expect from reading the Python API docs
for
os.makedirs "The default mode is 0777 (octal)."[1] but not what actually
happens
on most modern Linux systems thanks to umask.

Quoted 6 lines

Please review the following changes for suitability for inclusion. If you
have
any objections or suggestions for improvement, please respond to the patches. If
you agree with the changes, please provide your Acked-by.

Quoted 3 lines

777 seems questionable to me, personally. Generally collaboration happens amongst folks within a group, and chmod g+s makes that easier. I'd expect 775 to be a more sane value, myself.

Quoted 8 lines

Do you mean for bb.mkdirhier calls, the tgz files, the siginfo files or everything?

I went with 777 for mkdirhier as that's the default of os.makedirs before
umask is involved. I would likely have picked rw-rw-r-- (664) if I weren't
trying to request comments.

Quoted 6 lines

Gotcha.

I'm concerned about the behavior change and potential implications of changing the default behavior of mkdirhier. I'm inclined to say that when you don't pass mode, let it use the current behavior of obeying the umask.

An earlier version of the series did this and I'm happy to add that behaviour back in.

If we're not going to do that, and want to change the

Quoted 4 lines

default behavior, then I think 777 is the wrong/questionable default. Beyond that, 777 is certainly the wrong mode to be using for the shared state package in sstate.bbclass.

Do you have a strong preference on 664 vs. 775 ?

Quoted 7 lines

Quoted 3 lines

I'm leaning towards 664 (rw-rw-r--) for the files and 775 (rwxrwxr-x) for directories, these are the defaults for file and directory creation on Ubuntu 12.04 and Fedora 16.

At which point, the obvious "solution" to the bug report is a sanity check whether the user can read and write to the sstate directory.

may be its already taken care of but I will ask anyway, if shared state is writable for all
how is contention resolved when someone tries to update the shared state since he/she rebuilt the package for some reason. I think having a global shared cache to read from but not update it would be something interesting where shared state is updated by say an autobuilder and used by developers in their local builds.

SSTATE_MIRRORS does this nicely and since it uses the fetcher code that can be a file:// URI or other.

This is an RFC series, and for ease of review sent as against the Poky tree

Op 10 mei 2012, om 02:22 heeft Joshua Lock het volgende geschreven:

Quoted 1 line

This is an RFC series, and for ease of review sent as against the Poky tree

It requires an extra step to apply and thus makes review harder for me since my motivation to test them drops below zero. Please send patches to bitbake and oe-core to this list and reserve poky patches for the poky list. As for ease of review by glancing over them: 3 emails are 3 emails.