Skip to contentSkip to conversation

bitbake.conf: Add BB_TASK_NETWORK to enable task network globally

PATCH In review Robert Yang 4 participants 8 messages Jan 20, 2022 – Jan 21, 2022

lore
Share & tools
lore
#0 lore raw

cover bitbake.conf: Add BB_TASK_NETWORK to enable task network globally

The following changes since commit ecb61b36938754cf925bf58aad3edf7346deced0:

build-appliance-image: Update to master head revision (2022-01-19 23:04:05 +0000)

are available in the Git repository at:

git://git.openembedded.org/openembedded-core-contrib rbt/network
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=rbt/network

Robert Yang (1):
bitbake.conf: Add BB_TASK_NETWORK to enable task network globally

 meta/classes/icecc.bbclass |  2 ++
 meta/conf/bitbake.conf     |  3 +++
 meta/lib/oe/utils.py       | 15 +++++++++++++++
 3 files changed, 20 insertions(+)
Signature · 1 line
2.31.1
#1 lore raw

PATCH 1/1 bitbake.conf: Add BB_TASK_NETWORK to enable task network globally

The NIS or icecc can't work when task network is dissable, add BB_TASK_NETWORK to enable network globally for such exceptions.

Note, enable nscd on the build machine might be a solution, but that isn't reliable since it depends on whether the network function has been cached or not.

---
 meta/classes/icecc.bbclass |  2 ++
 meta/conf/bitbake.conf     |  3 +++
 meta/lib/oe/utils.py       | 15 +++++++++++++++
 3 files changed, 20 insertions(+)
meta/classes/icecc.bbclass +2 -0
index 794e9930ad9..c39c86458a4 100644
@@ -41,6 +41,8 @@ ICECC_ENV_EXEC ?= "${STAGING_BINDIR_NATIVE}/icecc-create-env"
 
 HOSTTOOLS_NONFATAL += "icecc patchelf"
 
+BB_TASK_NETWORK ? = "1"
+
 # This version can be incremented when changes are made to the environment that
 # invalidate the version on the compile nodes. Changing it will cause a new
 # environment to be created.
meta/conf/bitbake.conf +3 -0
index fba99e8f0cd..bf5bcd55519 100644
@@ -946,3 +946,6 @@ MULTILIB_VARIANTS ??= ""
 # what it would be anyway if the signature generator (e.g. OEEquivHash) doesn't
 # support unihashes.
 BB_UNIHASH ?= "${BB_TASKHASH}"
+
+# Enable task network for remote user such as NIS.
+BB_TASK_NETWORK ??= "${@['1', '0'][oe.utils.is_local_uid()]}"
meta/lib/oe/utils.py +15 -0
index 136650e6f74..c21f034aafc 100644
@@ -595,3 +595,18 @@ def directory_size(root, blocksize=4096):
         total += sum(roundup(getsize(os.path.join(root, name))) for name in files)
         total += roundup(getsize(root))
     return total
+
+def is_local_uid(uid=''):
+    """
+    Check whether uid is a local one or not.
+    Can't use pwd module since it gets all UIDs, not local ones only.
+    """
+    if not uid:
+        uid = os.getuid()
+    local_uids = set()
+    with open('/etc/passwd', 'r') as f:
+        for line in f.readlines():
+            if not ':' in line:
+                continue
+            local_uids.add(line.split(':')[2])
+    return uid in local_uids
Signature · 1 line
2.31.1
#2 lore raw

Hi Robert,

Quoted 52 lines · Robert Yang

The NIS or icecc can't work when task network is dissable, add BB_TASK_NETWORK to enable network globally for such exceptions.

Note, enable nscd on the build machine might be a solution, but that isn't reliable since it depends on whether the network function has been cached or not.

---
  meta/classes/icecc.bbclass |  2 ++
  meta/conf/bitbake.conf     |  3 +++
  meta/lib/oe/utils.py       | 15 +++++++++++++++
  3 files changed, 20 insertions(+)
meta/classes/icecc.bbclass +2 -0
index 794e9930ad9..c39c86458a4 100644
@@ -41,6 +41,8 @@ ICECC_ENV_EXEC ?= "${STAGING_BINDIR_NATIVE}/icecc-create-env"
 
  HOSTTOOLS_NONFATAL += "icecc patchelf"
 
+BB_TASK_NETWORK ? = "1"
+
  # This version can be incremented when changes are made to the environment that
  # invalidate the version on the compile nodes. Changing it will cause a new
  # environment to be created.
meta/conf/bitbake.conf +3 -0
index fba99e8f0cd..bf5bcd55519 100644
@@ -946,3 +946,6 @@ MULTILIB_VARIANTS ??= ""
  # what it would be anyway if the signature generator (e.g. OEEquivHash) doesn't
  # support unihashes.
  BB_UNIHASH ?= "${BB_TASKHASH}"
+
+# Enable task network for remote user such as NIS.
+BB_TASK_NETWORK ??= "${@['1', '0'][oe.utils.is_local_uid()]}"
meta/lib/oe/utils.py +5 -0
index 136650e6f74..c21f034aafc 100644
@@ -595,3 +595,18 @@ def directory_size(root, blocksize=4096):
          total += sum(roundup(getsize(os.path.join(root, name))) for name in files)
          total += roundup(getsize(root))
      return total
+
+def is_local_uid(uid=''):
+    """
+    Check whether uid is a local one or not.
+    Can't use pwd module since it gets all UIDs, not local ones only.

meta/lib/oe/utils.py def directory_size(root, blocksize=4096): · line 612

Quoted 10 lines
meta/lib/oe/utils.py +10 -0
+    """
+    if not uid:
+        uid = os.getuid()
+    local_uids = set()
+    with open('/etc/passwd', 'r') as f:
+        for line in f.readlines():
+            if not ':' in line:
+                continue
+            local_uids.add(line.split(':')[2])
+    return uid in local_uids

You can return ASAP instead of iterating over all users:

with open('/etc/passwd', 'r') as f:
for line in f.readlines():
if not ':' in line:
continue
if line.split(':')[2] == uid:
return True
return False

Otherwise, since we explicitly add icecc functions to existing tasks, can't you just add [network] = "1" to all those tasks in icecc.bbclass? e.g. do_configure[network] = "1" ?

Also, even if icecc is globally inherited and enabled, recipes can still disable its use on a per-recipe basis in which case I don't think we should have the network enabled?

I can't comment on NIS support.

Cheers,
Quentin

#3 lore raw
Quoted 66 lines · Quentin Schulz

Hi Robert,

Quoted 50 lines · Robert Yang

The NIS or icecc can't work when task network is dissable, add BB_TASK_NETWORK to enable network globally for such exceptions.

Note, enable nscd on the build machine might be a solution, but that isn't reliable since it depends on whether the network function has been cached or not.

---
  meta/classes/icecc.bbclass |  2 ++
  meta/conf/bitbake.conf     |  3 +++
  meta/lib/oe/utils.py       | 15 +++++++++++++++

  3 files changed, 20 insertions(+)
meta/classes/icecc.bbclass +0 -0
index 794e9930ad9..c39c86458a4 100644
@@ -41,6 +41,8 @@ ICECC_ENV_EXEC ?= "${STAGING_BINDIR_NATIVE}/icecc-create-env"

  HOSTTOOLS_NONFATAL += "icecc patchelf"

meta/classes/icecc.bbclass +2 -0
+BB_TASK_NETWORK ? = "1"
+

  # This version can be incremented when changes are made to the environment that

  # invalidate the version on the compile nodes. Changing it will cause a new
  # environment to be created.
meta/conf/bitbake.conf +0 -0
index fba99e8f0cd..bf5bcd55519 100644
@@ -946,3 +946,6 @@ MULTILIB_VARIANTS ??= ""
   # what it would be anyway if the signature generator (e.g. OEEquivHash) doesn't
   # support unihashes.

  BB_UNIHASH ?= "${BB_TASKHASH}"

meta/conf/bitbake.conf +3 -0
+
+# Enable task network for remote user such as NIS.
+BB_TASK_NETWORK ??= "${@['1', '0'][oe.utils.is_local_uid()]}"
meta/lib/oe/utils.py +0 -0
index 136650e6f74..c21f034aafc 100644
@@ -595,3 +595,18 @@ def directory_size(root, blocksize=4096):
           total += sum(roundup(getsize(os.path.join(root, name))) for name in files)
           total += roundup(getsize(root))

      return total

meta/lib/oe/utils.py +5 -0
+
+def is_local_uid(uid=''):
+    """
+    Check whether uid is a local one or not.
+    Can't use pwd module since it gets all UIDs, not local ones only.

meta/lib/oe/utils.py def directory_size(root, blocksize=4096): · line 612

Quoted 10 lines
meta/lib/oe/utils.py +10 -0
+    """
+    if not uid:
+        uid = os.getuid()
+    local_uids = set()
+    with open('/etc/passwd', 'r') as f:
+        for line in f.readlines():
+            if not ':' in line:
+                continue
+            local_uids.add(line.split(':')[2])
+    return uid in local_uids

You can return ASAP instead of iterating over all users:

Quoted 7 lines

with open('/etc/passwd', 'r') as f:
    for line in f.readlines():
        if not ':' in line:
            continue
        if line.split(':')[2] == uid:
            return True
return False

Yes, you're right, I renamed the func from get_all_local_uids to is_local_uid(), and forgot to change the logical.

Quoted 2 lines

Otherwise, since we explicitly add icecc functions to existing tasks, can't you just add [network] = "1" to all  those tasks in icecc.bbclass? e.g. do_configure[network] = "1" ?

The icecc issue is reported by Jose, I'm not sure whether that works for him or not.

Quoted 2 lines

Also, even if icecc is globally inherited and enabled, recipes can still disable its use on a per-recipe basis in which case I don't think we should have the network enabled?

The network is disabled by default unless set task[network] = "1", and BB_TASK_NETWORK is for preventing the disable, so you can't disable network per-recipe basis, otherwise, it won't work for the case like NIS.

Maybe a better name should be FORBIDDEN_DISABLE_TASK_NETWORK.

// Robert

Quoted 4 lines

I can't comment on NIS support. >
Cheers,
Quentin

#4 lore raw
In reply to Robert Yang
Quoted 19 lines · Robert Yang

The NIS or icecc can't work when task network is dissable, add BB_TASK_NETWORK to enable network globally for such exceptions.

Note, enable nscd on the build machine might be a solution, but that isn't reliable since it depends on whether the network function has been cached or not.

---
 meta/classes/icecc.bbclass |  2 ++
 meta/conf/bitbake.conf     |  3 +++
 meta/lib/oe/utils.py       | 15 +++++++++++++++
 3 files changed, 20 insertions(+)
meta/classes/icecc.bbclass +0 -0
index 794e9930ad9..c39c86458a4 100644

meta/classes/icecc.bbclass ICECC_ENV_EXEC ?= "${STAGING_BINDIR_NATIVE}/icecc-create-env" · line 45

Quoted 5 lines
meta/classes/icecc.bbclass +2 -0
@@ -41,6 +41,8 @@ ICECC_ENV_EXEC ?= "${STAGING_BINDIR_NATIVE}/icecc-create-env"
 
 HOSTTOOLS_NONFATAL += "icecc patchelf"
 
+BB_TASK_NETWORK ? = "1"
+

As Quentin mentions, I'd much rather this was done on the tasks using icecc using the flag.

Quoted 27 lines
meta/classes/icecc.bbclass +0 -0
 # This version can be incremented when changes are made to the environment that
 # invalidate the version on the compile nodes. Changing it will cause a new
 # environment to be created.
meta/conf/bitbake.conf +3 -0
index fba99e8f0cd..bf5bcd55519 100644
@@ -946,3 +946,6 @@ MULTILIB_VARIANTS ??= ""
 # what it would be anyway if the signature generator (e.g. OEEquivHash) doesn't
 # support unihashes.
 BB_UNIHASH ?= "${BB_TASKHASH}"
+
+# Enable task network for remote user such as NIS.
+BB_TASK_NETWORK ??= "${@['1', '0'][oe.utils.is_local_uid()]}"
meta/lib/oe/utils.py +5 -0
index 136650e6f74..c21f034aafc 100644
@@ -595,3 +595,18 @@ def directory_size(root, blocksize=4096):
         total += sum(roundup(getsize(os.path.join(root, name))) for name in files)
         total += roundup(getsize(root))
     return total
+
+def is_local_uid(uid=''):
+    """
+    Check whether uid is a local one or not.
+    Can't use pwd module since it gets all UIDs, not local ones only.

meta/lib/oe/utils.py def directory_size(root, blocksize=4096): · line 612

Quoted 10 lines
meta/lib/oe/utils.py +10 -0
+    """
+    if not uid:
+        uid = os.getuid()
+    local_uids = set()
+    with open('/etc/passwd', 'r') as f:
+        for line in f.readlines():
+            if not ':' in line:
+                continue
+            local_uids.add(line.split(':')[2])
+    return uid in local_uids

I think we should move this code into bitbake as I really dislike having it in bitbake.conf and that [0][1] syntax is horrible too...

I'm also not convinced about the name "BB_TASK_NETWORK" and think we need something better. If we're moving the code to bitbake we might not need a name right now though.

Cheers,

Richard

#5 lore raw
Quoted 71 lines · Richard Purdie
Quoted 19 lines · Robert Yang

The NIS or icecc can't work when task network is dissable, add BB_TASK_NETWORK to enable network globally for such exceptions.

Note, enable nscd on the build machine might be a solution, but that isn't reliable since it depends on whether the network function has been cached or not.

---
  meta/classes/icecc.bbclass |  2 ++
  meta/conf/bitbake.conf     |  3 +++
  meta/lib/oe/utils.py       | 15 +++++++++++++++
  3 files changed, 20 insertions(+)
meta/classes/icecc.bbclass +0 -0
index 794e9930ad9..c39c86458a4 100644

meta/classes/icecc.bbclass ICECC_ENV_EXEC ?= "${STAGING_BINDIR_NATIVE}/icecc-create-env" · line 45

Quoted 5 lines
meta/classes/icecc.bbclass +2 -0
@@ -41,6 +41,8 @@ ICECC_ENV_EXEC ?= "${STAGING_BINDIR_NATIVE}/icecc-create-env"
 
  HOSTTOOLS_NONFATAL += "icecc patchelf"
 
+BB_TASK_NETWORK ? = "1"
+

As Quentin mentions, I'd much rather this was done on the tasks using icecc using the flag.

Quoted 27 lines
meta/classes/icecc.bbclass +0 -0
  # This version can be incremented when changes are made to the environment that
  # invalidate the version on the compile nodes. Changing it will cause a new
  # environment to be created.
meta/conf/bitbake.conf +3 -0
index fba99e8f0cd..bf5bcd55519 100644
@@ -946,3 +946,6 @@ MULTILIB_VARIANTS ??= ""
  # what it would be anyway if the signature generator (e.g. OEEquivHash) doesn't
  # support unihashes.
  BB_UNIHASH ?= "${BB_TASKHASH}"
+
+# Enable task network for remote user such as NIS.
+BB_TASK_NETWORK ??= "${@['1', '0'][oe.utils.is_local_uid()]}"
meta/lib/oe/utils.py +5 -0
index 136650e6f74..c21f034aafc 100644
@@ -595,3 +595,18 @@ def directory_size(root, blocksize=4096):
          total += sum(roundup(getsize(os.path.join(root, name))) for name in files)
          total += roundup(getsize(root))
      return total
+
+def is_local_uid(uid=''):
+    """
+    Check whether uid is a local one or not.
+    Can't use pwd module since it gets all UIDs, not local ones only.

meta/lib/oe/utils.py def directory_size(root, blocksize=4096): · line 612

Quoted 10 lines
meta/lib/oe/utils.py +10 -0
+    """
+    if not uid:
+        uid = os.getuid()
+    local_uids = set()
+    with open('/etc/passwd', 'r') as f:
+        for line in f.readlines():
+            if not ':' in line:
+                continue
+            local_uids.add(line.split(':')[2])
+    return uid in local_uids

I think we should move this code into bitbake as I really dislike having it in bitbake.conf and that [0][1] syntax is horrible too...

Quoted 3 lines

I'm also not convinced about the name "BB_TASK_NETWORK" and think we need something better. If we're moving the code to bitbake we might not need a name right now though.

Yes, it's enough for NIS without a name, it seems that NIS is the only case atm, I will send a V2 about this.

// Robert

Quoted 7 lines

Cheers,

Richard

#6 lore raw
In reply to Robert Yang
Quoted 45 lines

-----Original Message-----
From: [email protected] <openembedded- [email protected]> On Behalf Of Robert Yang
Sent: den 20 januari 2022 08:09
To: [email protected]
Cc: [email protected]
Subject: [OE-core] [PATCH 1/1] bitbake.conf: Add BB_TASK_NETWORK to enable task network globally

The NIS or icecc can't work when task network is dissable, add BB_TASK_NETWORK to enable network globally for such exceptions.

Note, enable nscd on the build machine might be a solution, but that isn't reliable since it depends on whether the network function has been cached or not.

---
 meta/classes/icecc.bbclass |  2 ++
 meta/conf/bitbake.conf     |  3 +++
 meta/lib/oe/utils.py       | 15 +++++++++++++++
 3 files changed, 20 insertions(+)
meta/classes/icecc.bbclass +2 -0
index 794e9930ad9..c39c86458a4 100644
@@ -41,6 +41,8 @@ ICECC_ENV_EXEC ?= "${STAGING_BINDIR_NATIVE}/icecc-create-env"
 
 HOSTTOOLS_NONFATAL += "icecc patchelf"
 
+BB_TASK_NETWORK ? = "1"
+
 # This version can be incremented when changes are made to the environment that
 # invalidate the version on the compile nodes. Changing it will cause a new
 # environment to be created.
meta/conf/bitbake.conf +0 -0
index fba99e8f0cd..bf5bcd55519 100644
@@ -946,3 +946,6 @@ MULTILIB_VARIANTS ??= ""
 # what it would be anyway if the signature generator (e.g. OEEquivHash)

doesn't

 # support unihashes.
 BB_UNIHASH ?= "${BB_TASKHASH}"

meta/conf/bitbake.conf MULTILIB_VARIANTS ??= "" · line 951

Quoted 3 lines
meta/conf/bitbake.conf +3 -0
+
+# Enable task network for remote user such as NIS.
+BB_TASK_NETWORK ??= "${@['1', '0'][oe.utils.is_local_uid()]}"

I think this is more readable:

BB_TASK_NETWORK ??= "${@0 if oe.utils.is_local_uid() else 1}"

An even more readable solution would be to rename the function to is_nonlocal_uid() and make it return the inverse of what it returns now.

You probably also want to change it to use := instead of ??= so that /etc/passwd is only parsed once. This might require to introduce another variable:

UID_IS_NONLOCAL := "${@0 if oe.utils.is_local_uid() else 1}" BB_TASK_NETWORK ??= "${UID_IS_NONLOCAL}"

Quoted 24 lines
meta/lib/oe/utils.py +15 -1
index 136650e6f74..c21f034aafc 100644
@@ -595,3 +595,18 @@ def directory_size(root, blocksize=4096):
         total += sum(roundup(getsize(os.path.join(root, name))) for name in files)
         total += roundup(getsize(root))
     return total
+
+def is_local_uid(uid=''):
+    """
+    Check whether uid is a local one or not.
+    Can't use pwd module since it gets all UIDs, not local ones only.
+    """
+    if not uid:
+        uid = os.getuid()
+    local_uids = set()
+    with open('/etc/passwd', 'r') as f:
+        for line in f.readlines():
+            if not ':' in line:
+                continue
+            local_uids.add(line.split(':')[2])
+    return uid in local_uids
--
Quoted 1 line

2.31.1

//Peter

#7 lore raw
Quoted 57 lines · Peter Kjellerstedt
Quoted 45 lines

-----Original Message-----
From: [email protected] <openembedded- [email protected]> On Behalf Of Robert Yang
Sent: den 20 januari 2022 08:09
To: [email protected]
Cc: [email protected]
Subject: [OE-core] [PATCH 1/1] bitbake.conf: Add BB_TASK_NETWORK to enable task network globally

The NIS or icecc can't work when task network is dissable, add BB_TASK_NETWORK to enable network globally for such exceptions.

Note, enable nscd on the build machine might be a solution, but that isn't reliable since it depends on whether the network function has been cached or not.

---
  meta/classes/icecc.bbclass |  2 ++
  meta/conf/bitbake.conf     |  3 +++
  meta/lib/oe/utils.py       | 15 +++++++++++++++
  3 files changed, 20 insertions(+)
meta/classes/icecc.bbclass +2 -0
index 794e9930ad9..c39c86458a4 100644
@@ -41,6 +41,8 @@ ICECC_ENV_EXEC ?= "${STAGING_BINDIR_NATIVE}/icecc-create-env"
 
  HOSTTOOLS_NONFATAL += "icecc patchelf"
 
+BB_TASK_NETWORK ? = "1"
+
  # This version can be incremented when changes are made to the environment that
  # invalidate the version on the compile nodes. Changing it will cause a new
  # environment to be created.
meta/conf/bitbake.conf +0 -0
index fba99e8f0cd..bf5bcd55519 100644
@@ -946,3 +946,6 @@ MULTILIB_VARIANTS ??= ""
  # what it would be anyway if the signature generator (e.g. OEEquivHash)

doesn't

  # support unihashes.
  BB_UNIHASH ?= "${BB_TASKHASH}"

meta/conf/bitbake.conf MULTILIB_VARIANTS ??= "" · line 951

Quoted 3 lines
meta/conf/bitbake.conf +3 -0
+
+# Enable task network for remote user such as NIS.
+BB_TASK_NETWORK ??= "${@['1', '0'][oe.utils.is_local_uid()]}"

I think this is more readable:

BB_TASK_NETWORK ??= "${@0 if oe.utils.is_local_uid() else 1}"

An even more readable solution would be to rename the function to is_nonlocal_uid() and make it return the inverse of what it returns now.

Quoted 6 lines

You probably also want to change it to use := instead of ??= so that /etc/passwd is only parsed once. This might require to introduce another variable:

UID_IS_NONLOCAL := "${@0 if oe.utils.is_local_uid() else 1}" BB_TASK_NETWORK ??= "${UID_IS_NONLOCAL}"

We may not need this any more since NIS seems that is the only case which doesn't work, and can't be fixed (or not easy to fix) by set network per task.

// Robert

Quoted 29 lines
Quoted 25 lines
meta/lib/oe/utils.py +15 -1
index 136650e6f74..c21f034aafc 100644
@@ -595,3 +595,18 @@ def directory_size(root, blocksize=4096):
          total += sum(roundup(getsize(os.path.join(root, name))) for name in files)
          total += roundup(getsize(root))
      return total
+
+def is_local_uid(uid=''):
+    """
+    Check whether uid is a local one or not.
+    Can't use pwd module since it gets all UIDs, not local ones only.
+    """
+    if not uid:
+        uid = os.getuid()
+    local_uids = set()
+    with open('/etc/passwd', 'r') as f:
+        for line in f.readlines():
+            if not ':' in line:
+                continue
+            local_uids.add(line.split(':')[2])
+    return uid in local_uids
--
Quoted 1 line

2.31.1

//Peter