bitbake.conf: Add BB_TASK_NETWORK to enable task network globally
cover bitbake.conf: Add BB_TASK_NETWORK to enable task network globally
The following changes since commit ecb61b36938754cf925bf58aad3edf7346deced0:
build-appliance-image: Update to master head revision (2022-01-19 23:04:05 +0000)
are available in the Git repository at:
git://git.openembedded.org/openembedded-core-contrib rbt/network
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=rbt/network
Robert Yang (1):
bitbake.conf: Add BB_TASK_NETWORK to enable task network globally
meta/classes/icecc.bbclass | 2 ++ meta/conf/bitbake.conf | 3 +++ meta/lib/oe/utils.py | 15 +++++++++++++++ 3 files changed, 20 insertions(+)
Signature · 1 line
2.31.1
PATCH 1/1 bitbake.conf: Add BB_TASK_NETWORK to enable task network globally
The NIS or icecc can't work when task network is dissable, add BB_TASK_NETWORK to enable network globally for such exceptions.
Note, enable nscd on the build machine might be a solution, but that isn't reliable since it depends on whether the network function has been cached or not.
--- meta/classes/icecc.bbclass | 2 ++ meta/conf/bitbake.conf | 3 +++ meta/lib/oe/utils.py | 15 +++++++++++++++ 3 files changed, 20 insertions(+)
meta/classes/icecc.bbclass +2 -0
@@ -41,6 +41,8 @@ ICECC_ENV_EXEC ?= "${STAGING_BINDIR_NATIVE}/icecc-create-env" HOSTTOOLS_NONFATAL += "icecc patchelf" +BB_TASK_NETWORK ? = "1" + # This version can be incremented when changes are made to the environment that # invalidate the version on the compile nodes. Changing it will cause a new # environment to be created.
meta/conf/bitbake.conf +3 -0
@@ -946,3 +946,6 @@ MULTILIB_VARIANTS ??= "" # what it would be anyway if the signature generator (e.g. OEEquivHash) doesn't # support unihashes. BB_UNIHASH ?= "${BB_TASKHASH}" + +# Enable task network for remote user such as NIS. +BB_TASK_NETWORK ??= "${@['1', '0'][oe.utils.is_local_uid()]}"
meta/lib/oe/utils.py +15 -0
@@ -595,3 +595,18 @@ def directory_size(root, blocksize=4096): total += sum(roundup(getsize(os.path.join(root, name))) for name in files) total += roundup(getsize(root)) return total + +def is_local_uid(uid=''): + """ + Check whether uid is a local one or not. + Can't use pwd module since it gets all UIDs, not local ones only. + """ + if not uid: + uid = os.getuid() + local_uids = set() + with open('/etc/passwd', 'r') as f: + for line in f.readlines(): + if not ':' in line: + continue + local_uids.add(line.split(':')[2]) + return uid in local_uids
Signature · 1 line
2.31.1
Hi Robert,
Quoted 52 lines · Robert Yang
The NIS or icecc can't work when task network is dissable, add
BB_TASK_NETWORKto enable network globally for such exceptions.Note, enable nscd on the build machine might be a solution, but that isn't reliable since it depends on whether the network function has been cached or not.
Signed-off-by Robert Yang <[email protected]>--- meta/classes/icecc.bbclass | 2 ++ meta/conf/bitbake.conf | 3 +++ meta/lib/oe/utils.py | 15 +++++++++++++++ 3 files changed, 20 insertions(+)meta/classes/icecc.bbclass +2
-0@@ -41,6 +41,8 @@ ICECC_ENV_EXEC ?= "${STAGING_BINDIR_NATIVE}/icecc-create-env" HOSTTOOLS_NONFATAL += "icecc patchelf" +BB_TASK_NETWORK ? = "1" + # This version can be incremented when changes are made to the environment that # invalidate the version on the compile nodes. Changing it will cause a new # environment to be created.meta/conf/bitbake.conf +3
-0@@ -946,3 +946,6 @@ MULTILIB_VARIANTS ??= "" # what it would be anyway if the signature generator (e.g. OEEquivHash) doesn't # support unihashes. BB_UNIHASH ?= "${BB_TASKHASH}" + +# Enable task network for remote user such as NIS. +BB_TASK_NETWORK ??= "${@['1', '0'][oe.utils.is_local_uid()]}"meta/lib/oe/utils.py +5
-0@@ -595,3 +595,18 @@ def directory_size(root, blocksize=4096): total += sum(roundup(getsize(os.path.join(root, name))) for name in files) total += roundup(getsize(root)) return total + +def is_local_uid(uid=''): + """ + Check whether uid is a local one or not. + Can't use pwd module since it gets all UIDs, not local ones only.
meta/lib/oe/utils.py def directory_size(root, blocksize=4096): · line 612
Quoted 10 lines
meta/lib/oe/utils.py +10
-0+ """ + if not uid: + uid = os.getuid() + local_uids = set() + with open('/etc/passwd', 'r') as f: + for line in f.readlines(): + if not ':' in line: + continue + local_uids.add(line.split(':')[2]) + return uid in local_uids
You can return ASAP instead of iterating over all users:
with open('/etc/passwd', 'r') as f:
for line in f.readlines():
if not ':' in line:
continue
if line.split(':')[2] == uid:
return True
return False
Otherwise, since we explicitly add icecc functions to existing tasks, can't you just add [network] = "1" to all those tasks in icecc.bbclass? e.g. do_configure[network] = "1" ?
Also, even if icecc is globally inherited and enabled, recipes can still disable its use on a per-recipe basis in which case I don't think we should have the network enabled?
I can't comment on NIS support.
Cheers,
Quentin
Quoted 66 lines · Quentin Schulz
Hi Robert,
Quoted 50 lines · Robert Yang
The NIS or icecc can't work when task network is dissable, add
BB_TASK_NETWORKto enable network globally for such exceptions.Note, enable nscd on the build machine might be a solution, but that isn't reliable since it depends on whether the network function has been cached or not.
Signed-off-by Robert Yang <[email protected]>---
meta/classes/icecc.bbclass | 2 ++
meta/conf/bitbake.conf | 3 +++
meta/lib/oe/utils.py | 15 +++++++++++++++3 files changed, 20 insertions(+)meta/classes/icecc.bbclass +0
-0@@ -41,6 +41,8 @@ ICECC_ENV_EXEC ?= "${STAGING_BINDIR_NATIVE}/icecc-create-env"
HOSTTOOLS_NONFATAL+= "icecc patchelf"meta/classes/icecc.bbclass +2
-0+BB_TASK_NETWORK ? = "1" +# This version can be incremented when changes are made to the environment that
# invalidate the version on the compile nodes. Changing it will cause a new # environment to be created.meta/conf/bitbake.conf +0
-0@@ -946,3 +946,6 @@ MULTILIB_VARIANTS ??= "" # what it would be anyway if the signature generator (e.g. OEEquivHash) doesn't # support unihashes.
BB_UNIHASH?= "${BB_TASKHASH}"meta/conf/bitbake.conf +3
-0+ +# Enable task network for remote user such as NIS. +BB_TASK_NETWORK ??= "${@['1', '0'][oe.utils.is_local_uid()]}"meta/lib/oe/utils.py +0
-0@@ -595,3 +595,18 @@ def directory_size(root, blocksize=4096): total += sum(roundup(getsize(os.path.join(root, name))) for name in files) total += roundup(getsize(root))return total
meta/lib/oe/utils.py +5
-0+ +def is_local_uid(uid=''): + """ + Check whether uid is a local one or not. + Can't use pwd module since it gets all UIDs, not local ones only.meta/lib/oe/utils.py def directory_size(root, blocksize=4096): · line 612
Quoted 10 lines
meta/lib/oe/utils.py +10
-0+ """ + if not uid: + uid = os.getuid() + local_uids = set() + with open('/etc/passwd', 'r') as f: + for line in f.readlines(): + if not ':' in line: + continue + local_uids.add(line.split(':')[2]) + return uid in local_uidsYou can return ASAP instead of iterating over all users:
Quoted 7 lines
with
open('/etc/passwd', 'r')as f:
for line inf.readlines():
if not ':' in line:
continue
if line.split(':')[2] == uid:
return True
return False
Yes, you're right, I renamed the func from get_all_local_uids to is_local_uid(), and forgot to change the logical.
Quoted 2 lines
Otherwise, since we explicitly add icecc functions to existing tasks, can't you just add [network] = "1" to all those tasks in icecc.bbclass? e.g. do_configure[network] = "1" ?
The icecc issue is reported by Jose, I'm not sure whether that works for him or not.
Quoted 2 lines
Also, even if icecc is globally inherited and enabled, recipes can still disable its use on a per-recipe basis in which case I don't think we should have the network enabled?
The network is disabled by default unless set task[network] = "1", and BB_TASK_NETWORK is for preventing the disable, so you can't disable network per-recipe basis, otherwise, it won't work for the case like NIS.
Maybe a better name should be FORBIDDEN_DISABLE_TASK_NETWORK.
// Robert
Quoted 4 lines
I can't comment on NIS support. >
Cheers,
Quentin
Quoted 19 lines · Robert Yang
The NIS or icecc can't work when task network is dissable, add
BB_TASK_NETWORKto enable network globally for such exceptions.Note, enable nscd on the build machine might be a solution, but that isn't reliable since it depends on whether the network function has been cached or not.
Signed-off-by Robert Yang <[email protected]>--- meta/classes/icecc.bbclass | 2 ++ meta/conf/bitbake.conf | 3 +++ meta/lib/oe/utils.py | 15 +++++++++++++++ 3 files changed, 20 insertions(+)meta/classes/icecc.bbclass +0
-0
meta/classes/icecc.bbclass ICECC_ENV_EXEC ?= "${STAGING_BINDIR_NATIVE}/icecc-create-env" · line 45
Quoted 5 lines
meta/classes/icecc.bbclass +2
-0@@ -41,6 +41,8 @@ ICECC_ENV_EXEC ?= "${STAGING_BINDIR_NATIVE}/icecc-create-env" HOSTTOOLS_NONFATAL += "icecc patchelf" +BB_TASK_NETWORK ? = "1" +
As Quentin mentions, I'd much rather this was done on the tasks using icecc using the flag.
Quoted 27 lines
meta/classes/icecc.bbclass +0
-0# This version can be incremented when changes are made to the environment that # invalidate the version on the compile nodes. Changing it will cause a new # environment to be created.meta/conf/bitbake.conf +3
-0@@ -946,3 +946,6 @@ MULTILIB_VARIANTS ??= "" # what it would be anyway if the signature generator (e.g. OEEquivHash) doesn't # support unihashes. BB_UNIHASH ?= "${BB_TASKHASH}" + +# Enable task network for remote user such as NIS. +BB_TASK_NETWORK ??= "${@['1', '0'][oe.utils.is_local_uid()]}"meta/lib/oe/utils.py +5
-0@@ -595,3 +595,18 @@ def directory_size(root, blocksize=4096): total += sum(roundup(getsize(os.path.join(root, name))) for name in files) total += roundup(getsize(root)) return total + +def is_local_uid(uid=''): + """ + Check whether uid is a local one or not. + Can't use pwd module since it gets all UIDs, not local ones only.
meta/lib/oe/utils.py def directory_size(root, blocksize=4096): · line 612
Quoted 10 lines
meta/lib/oe/utils.py +10
-0+ """ + if not uid: + uid = os.getuid() + local_uids = set() + with open('/etc/passwd', 'r') as f: + for line in f.readlines(): + if not ':' in line: + continue + local_uids.add(line.split(':')[2]) + return uid in local_uids
I think we should move this code into bitbake as I really dislike having it in bitbake.conf and that [0][1] syntax is horrible too...
I'm also not convinced about the name "BB_TASK_NETWORK" and think we need something better. If we're moving the code to bitbake we might not need a name right now though.
Cheers,
Richard
Quoted 71 lines · Richard Purdie
Quoted 19 lines · Robert Yang
The NIS or icecc can't work when task network is dissable, add
BB_TASK_NETWORKto enable network globally for such exceptions.Note, enable nscd on the build machine might be a solution, but that isn't reliable since it depends on whether the network function has been cached or not.
Signed-off-by Robert Yang <[email protected]>--- meta/classes/icecc.bbclass | 2 ++ meta/conf/bitbake.conf | 3 +++ meta/lib/oe/utils.py | 15 +++++++++++++++ 3 files changed, 20 insertions(+)meta/classes/icecc.bbclass +0
-0meta/classes/icecc.bbclass ICECC_ENV_EXEC ?= "${STAGING_BINDIR_NATIVE}/icecc-create-env" · line 45
Quoted 5 lines
meta/classes/icecc.bbclass +2
-0@@ -41,6 +41,8 @@ ICECC_ENV_EXEC ?= "${STAGING_BINDIR_NATIVE}/icecc-create-env" HOSTTOOLS_NONFATAL += "icecc patchelf" +BB_TASK_NETWORK ? = "1" +As Quentin mentions, I'd much rather this was done on the tasks using icecc using the flag.
Quoted 27 lines
meta/classes/icecc.bbclass +0
-0# This version can be incremented when changes are made to the environment that # invalidate the version on the compile nodes. Changing it will cause a new # environment to be created.meta/conf/bitbake.conf +3
-0@@ -946,3 +946,6 @@ MULTILIB_VARIANTS ??= "" # what it would be anyway if the signature generator (e.g. OEEquivHash) doesn't # support unihashes. BB_UNIHASH ?= "${BB_TASKHASH}" + +# Enable task network for remote user such as NIS. +BB_TASK_NETWORK ??= "${@['1', '0'][oe.utils.is_local_uid()]}"meta/lib/oe/utils.py +5
-0@@ -595,3 +595,18 @@ def directory_size(root, blocksize=4096): total += sum(roundup(getsize(os.path.join(root, name))) for name in files) total += roundup(getsize(root)) return total + +def is_local_uid(uid=''): + """ + Check whether uid is a local one or not. + Can't use pwd module since it gets all UIDs, not local ones only.meta/lib/oe/utils.py def directory_size(root, blocksize=4096): · line 612
Quoted 10 lines
meta/lib/oe/utils.py +10
-0+ """ + if not uid: + uid = os.getuid() + local_uids = set() + with open('/etc/passwd', 'r') as f: + for line in f.readlines(): + if not ':' in line: + continue + local_uids.add(line.split(':')[2]) + return uid in local_uidsI think we should move this code into bitbake as I really dislike having it in bitbake.conf and that [0][1] syntax is horrible too...
Quoted 3 lines
I'm also not convinced about the name "
BB_TASK_NETWORK" and think we need something better. If we're moving the code to bitbake we might not need a name right now though.
Yes, it's enough for NIS without a name, it seems that NIS is the only case atm, I will send a V2 about this.
// Robert
Quoted 7 lines
Cheers,
Richard
Quoted 45 lines
-----Original Message-----
From: [email protected] <openembedded- [email protected]> On Behalf Of Robert Yang
Sent: den 20 januari 2022 08:09
To: [email protected]
Cc: [email protected]
Subject: [OE-core] [PATCH 1/1] bitbake.conf: AddBB_TASK_NETWORKto enable task network globallyThe NIS or icecc can't work when task network is dissable, add
BB_TASK_NETWORKto enable network globally for such exceptions.Note, enable nscd on the build machine might be a solution, but that isn't reliable since it depends on whether the network function has been cached or not.
Signed-off-by Robert Yang <[email protected]>--- meta/classes/icecc.bbclass | 2 ++ meta/conf/bitbake.conf | 3 +++ meta/lib/oe/utils.py | 15 +++++++++++++++ 3 files changed, 20 insertions(+)meta/classes/icecc.bbclass +2
-0@@ -41,6 +41,8 @@ ICECC_ENV_EXEC ?= "${STAGING_BINDIR_NATIVE}/icecc-create-env" HOSTTOOLS_NONFATAL += "icecc patchelf" +BB_TASK_NETWORK ? = "1" + # This version can be incremented when changes are made to the environment that # invalidate the version on the compile nodes. Changing it will cause a new # environment to be created.meta/conf/bitbake.conf +0
-0@@ -946,3 +946,6 @@ MULTILIB_VARIANTS ??= "" # what it would be anyway if the signature generator (e.g. OEEquivHash)doesn't
# support unihashes. BB_UNIHASH ?= "${BB_TASKHASH}"
meta/conf/bitbake.conf MULTILIB_VARIANTS ??= "" · line 951
Quoted 3 lines
meta/conf/bitbake.conf +3
-0+ +# Enable task network for remote user such as NIS. +BB_TASK_NETWORK ??= "${@['1', '0'][oe.utils.is_local_uid()]}"
I think this is more readable:
BB_TASK_NETWORK ??= "${@0 if oe.utils.is_local_uid() else 1}"
An even more readable solution would be to rename the function to is_nonlocal_uid() and make it return the inverse of what it returns now.
You probably also want to change it to use := instead of ??= so that /etc/passwd is only parsed once. This might require to introduce another variable:
UID_IS_NONLOCAL := "${@0 if oe.utils.is_local_uid() else 1}" BB_TASK_NETWORK ??= "${UID_IS_NONLOCAL}"
Quoted 24 lines
meta/lib/oe/utils.py +15
-1@@ -595,3 +595,18 @@ def directory_size(root, blocksize=4096): total += sum(roundup(getsize(os.path.join(root, name))) for name in files) total += roundup(getsize(root)) return total + +def is_local_uid(uid=''): + """ + Check whether uid is a local one or not. + Can't use pwd module since it gets all UIDs, not local ones only. + """ + if not uid: + uid = os.getuid() + local_uids = set() + with open('/etc/passwd', 'r') as f: + for line in f.readlines(): + if not ':' in line: + continue + local_uids.add(line.split(':')[2]) + return uid in local_uids --
Quoted 1 line
2.31.1
//Peter
Quoted 57 lines · Peter Kjellerstedt
Quoted 45 lines
-----Original Message-----
From: [email protected] <openembedded- [email protected]> On Behalf Of Robert Yang
Sent: den 20 januari 2022 08:09
To: [email protected]
Cc: [email protected]
Subject: [OE-core] [PATCH 1/1] bitbake.conf: AddBB_TASK_NETWORKto enable task network globallyThe NIS or icecc can't work when task network is dissable, add
BB_TASK_NETWORKto enable network globally for such exceptions.Note, enable nscd on the build machine might be a solution, but that isn't reliable since it depends on whether the network function has been cached or not.
Signed-off-by Robert Yang <[email protected]>--- meta/classes/icecc.bbclass | 2 ++ meta/conf/bitbake.conf | 3 +++ meta/lib/oe/utils.py | 15 +++++++++++++++ 3 files changed, 20 insertions(+)meta/classes/icecc.bbclass +2
-0@@ -41,6 +41,8 @@ ICECC_ENV_EXEC ?= "${STAGING_BINDIR_NATIVE}/icecc-create-env" HOSTTOOLS_NONFATAL += "icecc patchelf" +BB_TASK_NETWORK ? = "1" + # This version can be incremented when changes are made to the environment that # invalidate the version on the compile nodes. Changing it will cause a new # environment to be created.meta/conf/bitbake.conf +0
-0@@ -946,3 +946,6 @@ MULTILIB_VARIANTS ??= "" # what it would be anyway if the signature generator (e.g. OEEquivHash)doesn't
# support unihashes. BB_UNIHASH ?= "${BB_TASKHASH}"meta/conf/bitbake.conf MULTILIB_VARIANTS ??= "" · line 951
Quoted 3 lines
meta/conf/bitbake.conf +3
-0+ +# Enable task network for remote user such as NIS. +BB_TASK_NETWORK ??= "${@['1', '0'][oe.utils.is_local_uid()]}"I think this is more readable:
BB_TASK_NETWORK??= "${@0 ifoe.utils.is_local_uid()else 1}"An even more readable solution would be to rename the function to
is_nonlocal_uid()and make it return the inverse of what it returns now.
Quoted 6 lines
You probably also want to change it to use := instead of ??= so that
/etc/passwdis only parsed once. This might require to introduce another variable:
UID_IS_NONLOCAL:= "${@0 ifoe.utils.is_local_uid()else 1}"BB_TASK_NETWORK??= "${UID_IS_NONLOCAL}"
We may not need this any more since NIS seems that is the only case which doesn't work, and can't be fixed (or not easy to fix) by set network per task.
// Robert
Quoted 29 lines
Quoted 25 lines
meta/lib/oe/utils.py +15
-1@@ -595,3 +595,18 @@ def directory_size(root, blocksize=4096): total += sum(roundup(getsize(os.path.join(root, name))) for name in files) total += roundup(getsize(root)) return total + +def is_local_uid(uid=''): + """ + Check whether uid is a local one or not. + Can't use pwd module since it gets all UIDs, not local ones only. + """ + if not uid: + uid = os.getuid() + local_uids = set() + with open('/etc/passwd', 'r') as f: + for line in f.readlines(): + if not ':' in line: + continue + local_uids.add(line.split(':')[2]) + return uid in local_uids --Quoted 1 line
2.31.1
//Peter