NPM/NPM Shrinkwrap Basic Authentication Credential Support:
cover NPM/NPM Shrinkwrap Basic Authentication Credential Support:
From: "[email protected]" <[email protected]>
Changes in this commit include:
- npm fetcher changes:
- Add “username=”, “password=”, and “namespace=” parameters to “npm://” SRC_URI
- These parameters will be used to create a “npmrc” file that is passed into the NpmEnvironment object - used for authentication when connecting to NPM registry
- Username and password parameters will be forwarded along to wget fetcher after resolving the tarball_url in the npm view output.
- npmsw fetcher changes:
- Add “username=”, “password=”, “basic_auth_registry=” parameters to “npmsw://” SRC_URI
- For each registry specified in the shrinkwrap file, the fetcher will attempt to match the basic_auth_registry argument to the registry URL in the shrinkwrap, and if they match, then it will forward the username and password parameters along to the wget fetcher
- fetch test changes:
- skimdb.npmjs.com was shut down, so switched alternate_registry test to arbitrarily use registry.yarnpkg.
Eric Meyers (3):
fetch2/npm: Add basic auth credential support for npm fetcher fetch2/npmsw: Add basic auth credential support for npmsw fetcher tests/fetch: Switch to working npm registry for registry_alternate
test
lib/bb/fetch2/npm.py | 64 +++++++++++++++++++++++++++++++++++++++++- lib/bb/fetch2/npmsw.py | 36 ++++++++++++++++++++++++ lib/bb/tests/fetch.py | 2 +- 3 files changed, 100 insertions(+), 2 deletions(-)
PATCH 1/3 fetch2/npm: Add basic auth credential support for npm fetcher
From: Eric Meyers <[email protected]>
--- lib/bb/fetch2/npm.py | 64 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 63 insertions(+), 1 deletion(-)
lib/bb/fetch2/npm.py +63 -1
@@ -21,6 +21,15 @@ Supported SRC_URI options are: - destsuffix Specifies the directory to use to unpack the package (default: npm). + +- namespace + The npm namespace name (without the preceding "@"). + +- username + Specifies the basic auth username to use when accessing the npm registry. + +- password + Specifies the basic auth password (or API key) to use when accessing the npm registry. """ import base64
@@ -82,6 +91,33 @@ def npm_unpack(tarball, destdir, d): runfetchcmd(cmd, d, workdir=destdir) runfetchcmd("chmod -R +X '%s'" % (destdir), d, quiet=True, workdir=destdir) +def create_npmrc_file(registry, namespace=None, username=None, password=None): + """Creates a npmrc file with authentication credentials populated based on provided username and password""" + + # Create a temporary file we can store the npm authentication within + npmrc_file = tempfile.NamedTemporaryFile(mode="w", buffering=1).name + + # Convert "username:password" string into base64 auth string + base64_auth_string = base64.b64encode(f"{username}:{password}".encode('utf-8')).decode('utf-8') + + # Remove protocol from registry URL + registry = registry.replace('https://', '').replace('http://', '') + + # Create the npmrc file with the authentication parameters + npm_auth_string = f"//{registry}/:_auth=\"{base64_auth_string}\"" + + with open(npmrc_file, 'w') as file: + file.write(npm_auth_string) + file.write("\n") + + # Add namespace registry string if namespace is provided + if namespace: + npm_registry_string = f"@{namespace}:registry=https://{registry}" + file.write(npm_registry_string) + file.write("\n") + + return npmrc_file + class NpmEnvironment(object): """ Using a npm config file seems more reliable than using cli arguments.
@@ -142,6 +178,9 @@ class Npm(FetchMethod): ud.package = None ud.version = None ud.registry = None + ud.username = None + ud.password = None + ud.namespace = None # Get the 'package' parameter if "package" in ud.parm:
@@ -154,6 +193,16 @@ class Npm(FetchMethod): if "version" in ud.parm: ud.version = ud.parm.get("version") + if "username" in ud.parm: + ud.username = ud.parm.get("username") + + if "password" in ud.parm: + ud.password = ud.parm.get("password") + + if "namespace" in ud.parm: + ud.namespace = ud.parm.get("namespace") + ud.package = f"@{ud.namespace}/{ud.package}" + if not ud.version: raise MissingParameterError("Parameter 'version' required", ud.url)
@@ -189,7 +238,16 @@ class Npm(FetchMethod): args.append(("registry", ud.registry)) pkgver = shlex.quote(ud.package + "@" + ud.version) cmd = ud.basecmd + " view %s" % pkgver - env = NpmEnvironment(d) + + # Create an npmrc file if a username and password was supplied to the npm SRC_URI + npmrc_file = None + if (ud.username is not None and ud.password is not None): + try: + npmrc_file = create_npmrc_file(ud.registry, ud.namespace, ud.username, ud.password) + except Exception as e: + bb.warn(f"Error creating npmrc file: {e}") + + env = NpmEnvironment(d, [], npmrc=npmrc_file) check_network_access(d, cmd, ud.registry) view_string = env.run(cmd, args=args)
@@ -224,6 +282,10 @@ class Npm(FetchMethod): uri = URI(tarball_url) uri.params["downloadfilename"] = ud.localfile + if (ud.username is not None and ud.password is not None): + uri.params["user"] = ud.username + uri.params["pswd"] = ud.password + integrity = view.get("dist", {}).get("integrity") shasum = view.get("dist", {}).get("shasum")
PATCH 2/3 fetch2/npmsw: Add basic auth credential support for npmsw fetcher
In reply to [email protected]From: Eric Meyers <[email protected]>
--- lib/bb/fetch2/npmsw.py | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+)
lib/bb/fetch2/npmsw.py +36 -0
@@ -15,12 +15,24 @@ Supported SRC_URI options are: - destsuffix Specifies the directory to use to unpack the dependencies (default: ${S}). + +- username + Specifies the basic auth username to use when accessing an npm registry specified in the shrinkwrap. + +- password + Specifies the basic auth password (or API key) to use when accessing the npm registry specified in the shrinkwrap. + +- basic_auth_registry + Specifies the registry within the npm shrinkwrap to use basic authentication with. + NOTE: This parameter is optional - if nothing is passed in but a username/password IS supplied, then + fetcher will assume to use basic auth credentials for all registries within the shrinkwrap file. """ import json import os import re import bb +import urllib.parse from bb.fetch2 import Fetch from bb.fetch2 import FetchMethod from bb.fetch2 import ParameterError
@@ -67,10 +79,26 @@ class NpmShrinkWrap(FetchMethod): def urldata_init(self, ud, d): """Init npmsw specific variables within url data""" + ud.shrinkwrap_file = None + ud.dev = None + ud.deps = None + ud.basic_auth_registry = None + ud.username = None + ud.password = None # Get the 'shrinkwrap' parameter ud.shrinkwrap_file = re.sub(r"^npmsw://", "", ud.url.split(";")[0]) + if "basic_auth_registry" in ud.parm: + ud.basic_auth_registry = ud.parm.get("basic_auth_registry") + + if "username" in ud.parm: + ud.username = ud.parm.get("username") + + if "password" in ud.parm: + ud.password = ud.parm.get("password") + + # Get the 'dev' parameter ud.dev = bb.utils.to_boolean(ud.parm.get("dev"), False)
@@ -107,6 +135,14 @@ class NpmShrinkWrap(FetchMethod): checksum_name, checksum_expected = npm_integrity(integrity) uri.params[checksum_name] = checksum_expected + # Check if the registry we are using matches the registry passed in + # that the user wants to use for basic authentication + if ud.basic_auth_registry is not None: + use_basic_auth = urllib.parse.urlparse(resolved).hostname in ud.basic_auth_registry + if (ud.username is not None and ud.password is not None and use_basic_auth): + uri.params["user"] = ud.username + uri.params["pswd"] = ud.password + url = str(uri) localpath = os.path.join(d.getVar("DL_DIR"), localfile)
PATCH 3/3 tests/fetch: Switch to working npm registry for registry_alternate test
In reply to [email protected]From: Eric Meyers <[email protected]>
--- lib/bb/tests/fetch.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
lib/bb/tests/fetch.py +1 -1
@@ -2772,7 +2772,7 @@ class NPMTest(FetcherTest): @skipIfNoNpm() @skipIfNoNetwork() def test_npm_registry_alternate(self): - urls = ['npm://skimdb.npmjs.com;package=@savoirfairelinux/node-server-example;version=1.0.0'] + urls = ['npm://registry.yarnpkg.com;package=@savoirfairelinux/node-server-example;version=1.0.0'] fetcher = bb.fetch.Fetch(urls, self.d) fetcher.download() fetcher.unpack(self.unpackdir)
Quoted 4 lines · On Mon, 27 Jan 2025 at 15:29, eric.meyers15310 via lists.openembedded.org…
Changes in this commit include:
- npm fetcher changes:
- Add “username=”, “password=”, and “namespace=” parameters to “npm://”SRC_URI
- These parameters will be used to create a “npmrc” file that is passed into the NpmEnvironment object - used for authentication when connecting to NPM registry
You need to create the file with a separate step, then point the fetcher or the underlying executable to use it. We do not allow credentials in recipes or other metadata that gets committed to a layer repo. It's not different from placing ssh keys or http passwords into respective files in $HOME.
Alex
Quoted 9 lines · Alexander Kanavin
Quoted 4 lines · On Mon, 27 Jan 2025 at 15:29, eric.meyers15310 via lists.openembedded.org…
Changes in this commit include:
- npm fetcher changes:
- Add “username=”, “password=”, and “namespace=” parameters to “npm://”SRC_URI
- These parameters will be used to create a “npmrc” file that is passed into the NpmEnvironment object - used for authentication when connecting to NPM registry
Quoted 7 lines
You need to create the file with a separate step, then point the fetcher or the underlying executable to use it. We do not allow credentials in recipes or other metadata that gets committed to a layer repo. It's not different from placing ssh keys or http passwords into respective files in $HOME.
Alex
Are you suggesting that instead of username= and password= parameters in the npm/npmsw SRC_URI, it should be an "npmrc" parameter that effectively the user can point to the location of their .npmrc file for authentication on an NPM registry? Then, within a user's associated recipe, they would need to create the npmrc file in a do_fetch:prepend() step that that corresponds to the path passed into the SRC_URI?
That seems like it'd be fine to retrieve the initial "npm view" output
(https://git.openembedded.org/bitbake/tree/lib/bb/fetch2/npm.py#n186),
which the fetcher uses to resolve the tarball url. But then when subsequently passing the tarball url into the wget fetcher
(https://git.openembedded.org/bitbake/tree/lib/bb/fetch2/npm.py#n238),
the wget SRC_URI will need to provide a username/password for basic authentication to access the artifact - that's kinda where I'm stuck and thought a "username=" and "password=" parameter were needed for the npm fetcher similar to the "user=" and "pswd=" parameters available in the wget fetcher.
Please correct me if my understanding is wrong, I've been agonizing over this for a few weeks now and couldn't think of a better/more concise solution here, but may be overthinking things at this point.
Quoted 7 lines · Eric Meyers
Are you suggesting that instead of username= and password= parameters in the npm/npmsw
SRC_URI, it should be an "npmrc" parameter that effectively the user can point to the location of their .npmrc file for authentication on an NPM registry? Then, within a user's associated recipe, they would need to create the npmrc file in a do_fetch:prepend() step that that corresponds to the path passed into theSRC_URI?
The only acceptable solution would be one that takes passwords written in plain text out of layers and local.conf altogether. And the way to do it is to write these things into ~/.npmrc and ~/.netrc in a secure manner (e.g. pass in to users via encrypted email with instructions that the file must not be readable by anyone but the user), and ensure that npm and wget are able to pick them up from those files (wget does this already, npm I'm not sure).
I know we have the password parameter in wget and other 'old' fetchers, that's a bad mis-feature in my opinion and should be removed. For git fetcher patches to add the equivalent have been repeatedly rejected:
https://git.openembedded.org/bitbake/tree/lib/bb/fetch2/git.py#n823
Alex
Quoted 10 lines · Alexander Kanavin via lists.openembedded.org
Quoted 7 lines · Eric Meyers
Are you suggesting that instead of username= and password= parameters in the npm/npmsw
SRC_URI, it should be an "npmrc" parameter that effectively the user can point to the location of their .npmrc file for authentication on an NPM registry? Then, within a user's associated recipe, they would need to create the npmrc file in a do_fetch:prepend() step that that corresponds to the path passed into theSRC_URI?
Quoted 8 lines
The only acceptable solution would be one that takes passwords written in plain text out of layers and local.conf altogether. And the way to do it is to write these things into ~/.npmrc and ~/.netrc in a secure manner (e.g. pass in to users via encrypted email with instructions that the file must not be readable by anyone but the user), and ensure that npm and wget are able to pick them up from those files (wget does this already, npm I'm not sure).
Could this also be solved by letting the end user decide, sort of like the way FETCHCMD_*
works?
A CREDENTIALCMD_npm variable in local.conf could be saddled with exporting the
namespace, password, and username values into whatever environment needs them in a
"Just in Time" fashion to lower (but not completely eliminate) exposure.
An identity agent, like the 1P or vault CLI would be an obvious choice here, but one could
just as easily tie it to a script that devolves down to looking in ~/.npmrc if they wanted to.
..Ch:W..
Signature · 2 lines
*"Perfection must be reached by degrees; she requires the slow hand of time." - Voltaire*
Quoted 30 lines · Chuck Wolber
Quoted 10 lines · Alexander Kanavin via lists.openembedded.org
Quoted 7 lines · Eric Meyers
Are you suggesting that instead of username= and password= parameters in the npm/npmsw
SRC_URI, it should be an "npmrc" parameter that effectively the user can point to the location of their .npmrc file for authentication on an NPM registry? Then, within a user's associated recipe, they would need to create the npmrc file in a do_fetch:prepend() step that that corresponds to the path passed into theSRC_URI?Quoted 7 lines
The only acceptable solution would be one that takes passwords written in plain text out of layers and local.conf altogether. And the way to do it is to write these things into ~/.npmrc and ~/.netrc in a secure manner (e.g. pass in to users via encrypted email with instructions that the file must not be readable by anyone but the user), and ensure that npm and wget are able to pick them up from those files (wget does this already, npm I'm not sure).
Could this also be solved by letting the end user decide, sort of like the way FETCHCMD_* works?
A
CREDENTIALCMD_npmvariable in local.conf could be saddled with exporting the namespace, password, and username values into whatever environment needs them in a "Just in Time" fashion to lower (but not completely eliminate) exposure.
Quoted 7 lines
An identity agent, like the 1P or vault CLI would be an obvious choice here, but one could just as easily tie it to a script that devolves down to looking in ~/.npmrc if they wanted to.
..Ch:W..
--
"Perfection must be reached by degrees; she requires the slow hand of time." - Voltaire
Last week I had off, so I'm just getting around to addressing these comments this week. Chuck's suggestion is interesting and I will definitely consider it. So this amounts to two potential paths forward for this issue (1 by Alex and 1 by Chuck):
1. Take a similar approach to how the wget fetcher uses default netrc credentials within the user's home directory and use default credentials populated by the user within an npmrc file in their home directory.
- Pros: Moves away from specifying username/password in recipe,
using precedent set in wget fetcher, potentially less changes in fetcher (no parameter additions, no creation of npmrc file, etc)
- Cons: Currently, the wget fetcher relies on default behavior of
"netrc" python library - there is no analogous "npmrc" python library and we would need to add logic to check this file manually (shouldn't be too difficult but still would need to rework current NPM environment behavior in the fetcher). Might pose challenges in CI/CD environments where these files need to now be created if using private registries (again should be easy enough).
2. Take a similar approach to how FETCHCMD_npm is used and specify another variable named "FETCHCREDENTIALCMD_npm" that allows the user to point to a specific npmrc file to use in their environment.
- Pros: Moves away from specifying username/password in recipe
- Cons: Adding another variable to bitbake, lowers (but doesn't
completely eliminate) credential exposure, might require more support within recipetool within poky
I am going to take a stab at reworking this patch to match approach #1 outlined above, but if any hiccups come up, I'll post issues here and re-evaluate using another approach (potentially #2 above). #1 feels like it will be easier to implement and more straightforward than #2, but perhaps I'm missing something.
Quoted 91 lines · Eric Meyers via lists.openembedded.org
Quoted 53 lines · Chuck Wolber
Quoted 16 lines · Alexander Kanavin via lists.openembedded.org
Quoted 4 lines · Eric Meyers
Are you suggesting that instead of username= and password= parameters
in the npm/npmswSRC_URI, it should be an "npmrc" parameter thatQuoted 8 lines
effectively the user can point to the location of their .npmrc file
for authentication on an NPM registry? Then, within a user's associated recipe, they would need to create the npmrc file in a
do_fetch:prepend() step that that corresponds to the path passed into
theSRC_URI?Quoted 13 lines
The only acceptable solution would be one that takes passwords written
in plain text out of layers and local.conf altogether. And the way to
do it is to write these things into ~/.npmrc and ~/.netrc in a secure
manner (e.g. pass in to users via encrypted email with instructions
that the file must not be readable by anyone but the user), and ensure
that npm and wget are able to pick them up from those files (wget does
this already, npm I'm not sure).Could this also be solved by letting the end user decide, sort of like the way FETCHCMD_*
works?A
CREDENTIALCMD_npmvariable in local.conf could be saddled with exporting the
namespace, password, and username values into whatever environment needs them in a
"Just in Time" fashion to lower (but not completely eliminate) exposure.An identity agent, like the 1P or vault CLI would be an obvious choice here, but one could
just as easily tie it to a script that devolves down to looking in ~/.npmrc if they wanted to.Quoted 5 lines
..Ch:W..
--
"Perfection must be reached by degrees; she requires the slow hand of time." - VoltaireLast week I had off, so I'm just getting around to addressing these comments this week. Chuck's suggestion is interesting and I will definitely consider it. So this amounts to two potential paths forward
for this issue (1 by Alex and 1 by Chuck):1. Take a similar approach to how the wget fetcher uses default netrc credentials within the user's home directory and use default credentials populated by the user within an npmrc file in their home directory.
- Pros: Moves away from specifying username/password in recipe, using precedent set in wget fetcher, potentially less changes in fetcher (no parameter additions, no creation of npmrc file, etc)
- Cons: Currently, the wget fetcher relies on default behavior of "netrc" python library - there is no analogous "npmrc" python library and we would need to add logic to check this file manually (shouldn't be too difficult but still would need to rework current NPM environment behavior in the fetcher). Might pose challenges in CI/CD environments where these files need to now be created if using private
registries (again should be easy enough).2. Take a similar approach to how
FETCHCMD_npmis used and specify another variable named "FETCHCREDENTIALCMD_npm" that allows the user to point to a specific npmrc file to use in their environment.
- Pros: Moves away from specifying username/password in recipe
- Cons: Adding another variable to bitbake, lowers (but doesn't completely eliminate) credential exposure, might require more support within recipetool within poky
Quoted 6 lines
I am going to take a stab at reworking this patch to match approach #1
outlined above, but if any hiccups come up, I'll post issues here and re-evaluate using another approach (potentially #2 above). #1 feels like it will be easier to implement and more straightforward than #2, but perhaps I'm missing something.
FWIW I think this is the right thing to do and #1 matches what we've done elsewhere.
Having watched what happens in the real world with passwords in urls, we really don't want to support that.
Cheers,
Richard
Hi Eric,
have you seen my RFC [1] to remove the npm dependency from the npm fetcher and use plain wget? I have a Work in Progress to parse the package-lock.json inside a task. This task generate plain https SRC_URIs and fetch everything with the wget fetcher. Furthermore I have eliminate the npm cache to speed up the build. I plan to post a first RFC with cargo, go mod and npm support in the next days.
Regards
Stefan
[1] https://lists.openembedded.org/g/bitbake-devel/topic/110212697
Am 03.02.2025 um 16:33 schrieb Eric Meyers via lists.openembedded.org:
Quoted 76 lines · Chuck Wolber
Quoted 26 lines · Alexander Kanavin via lists.openembedded.org
Quoted 8 lines · Eric Meyers
Quoted 7 lines
Are you suggesting that instead of username= and password= parameters in the npm/npmsw
SRC_URI, it should be an "npmrc" parameter that effectively the user can point to the location of their .npmrc file for authentication on an NPM registry? Then, within a user's associated recipe, they would need to create the npmrc file in a do_fetch:prepend() step that that corresponds to the path passed into theSRC_URI?Quoted 7 lines
The only acceptable solution would be one that takes passwords written in plain text out of layers and local.conf altogether. And the way to do it is to write these things into ~/.npmrc and ~/.netrc in a secure manner (e.g. pass in to users via encrypted email with instructions that the file must not be readable by anyone but the user), and ensure that npm and wget are able to pick them up from those files (wget does this already, npm I'm not sure).
Could this also be solved by letting the end user decide, sort of like the way FETCHCMD_* works?
A
CREDENTIALCMD_npmvariable in local.conf could be saddled with exporting the namespace, password, and username values into whatever environment needs them in a "Just in Time" fashion to lower (but not completely eliminate) exposure.Quoted 7 lines
An identity agent, like the 1P or vault CLI would be an obvious choice here, but one could just as easily tie it to a script that devolves down to looking in ~/.npmrc if they wanted to.
..Ch:W..
--
"Perfection must be reached by degrees; she requires the slow hand of time." - VoltaireLast week I had off, so I'm just getting around to addressing these comments this week. Chuck's suggestion is interesting and I will definitely consider it. So this amounts to two potential paths forward for this issue (1 by Alex and 1 by Chuck):
1. Take a similar approach to how the wget fetcher uses default netrc credentials within the user's home directory and use default credentials populated by the user within an npmrc file in their home directory.
- Pros: Moves away from specifying username/password in recipe,
using precedent set in wget fetcher, potentially less changes in fetcher (no parameter additions, no creation of npmrc file, etc)
- Cons: Currently, the wget fetcher relies on default behavior of
"netrc" python library - there is no analogous "npmrc" python library and we would need to add logic to check this file manually (shouldn't be too difficult but still would need to rework current NPM environment behavior in the fetcher). Might pose challenges in CI/CD environments where these files need to now be created if using private registries (again should be easy enough).2. Take a similar approach to how
FETCHCMD_npmis used and specify another variable named "FETCHCREDENTIALCMD_npm" that allows the user to point to a specific npmrc file to use in their environment.
- Pros: Moves away from specifying username/password in recipe
- Cons: Adding another variable to bitbake, lowers (but doesn't
completely eliminate) credential exposure, might require more support within recipetool within pokyI am going to take a stab at reworking this patch to match approach #1 outlined above, but if any hiccups come up, I'll post issues here and re-evaluate using another approach (potentially #2 above). #1 feels like it will be easier to implement and more straightforward than #2, but perhaps I'm missing something.
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#17122): https://lists.openembedded.org/g/bitbake-devel/message/17122
Mute This Topic: https://lists.openembedded.org/mt/110839321/6374899
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 10 lines · Stefan Herbrechtsmeier
Hi Eric,
have you seen my RFC [1] to remove the npm dependency from the npm fetcher and use plain wget? I have a Work in Progress to parse the package-lock.json inside a task. This task generate plain https
SRC_URIsand fetch everything with the wget fetcher. Furthermore I have eliminate the npm cache to speed up the build. I plan to post a first RFC with cargo, go mod and npm support in the next days.
Quoted 5 lines
Regards
Stefan[1] https://lists.openembedded.org/g/bitbake-devel/topic/110212697
Hi Stefan,
No I was not aware of that change - thanks for bringing it to my attention. It looks interesting and I think would make my (eventual) patch resubmission obsolete if I’m understanding it correctly.
A few questions:
- Sounds like this change will remove the need for an npmrc file
when using private npm registries - correct? (since npm altogether is being removed)
- Do these changes impact devtool/recipetool within the upstream
poky repo at all? I assume so if the parameters to npm SRC_URI have changed. Did you have any plans to make those changes by any chance? If not, then I will try and pick up this change in my environment to test alongside.
- Our workflow here eventually is to run “devtool add
“npm://<private_npm_registry>;params”” and have the recipe and shrinkwrap generated automatically.
- Do you plan on backporting these changes over to scarthgap?
- What commit are you based off of within bitbake master for your patchset?
Thanks,
-Eric