tools: ynl: try to avoid the very slow YAML loader
Subsystems (3)
Jakub Kicinski
maintainerPATCH 1/1 tools: ynl: try to avoid the very slow YAML loader
Turns out Python YAML defaults to a pure Python loader for YAML files which is a lot slower than the C loader (using libyaml). Try to use the C one whenever possible.
The avg time to run:
$ tools/net/ynl/pyynl/cli.py --family tc --no-schema drops from 300+ ms to 115 ms with this change (40 samples).
We could drop the load time further to 85 ms if we "compiled" the specs to JSON. Slightly tricky parts are that we don't currently install the specs at all on make install, so it's unclear where to put the conversion. Also JSON has questionable support for comments and we need an SPDX line.
---
CC: [email protected]
--- tools/net/ynl/pyynl/lib/nlspec.py | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
tools/net/ynl/pyynl/lib/nlspec.py +7 -2
@@ -439,6 +439,11 @@ import yaml as pyyaml # To be loaded dynamically as needed jsonschema = None + try: + _yaml_loader = pyyaml.CSafeLoader + except AttributeError: + _yaml_loader = pyyaml.SafeLoader + def __init__(self, spec_path, schema_path=None, exclude_ops=None): with open(spec_path, "r", encoding='utf-8') as stream: prefix = '# SPDX-License-Identifier: '
@@ -448,7 +453,7 @@ import yaml as pyyaml self.license = first[len(prefix):] stream.seek(0) - spec = pyyaml.safe_load(stream) + spec = pyyaml.load(stream, Loader=self._yaml_loader) self.fixed_header = None self._resolution_list = []
@@ -464,7 +469,7 @@ import yaml as pyyaml schema_path = os.path.dirname(os.path.dirname(spec_path)) + f'/{self.proto}.yaml' if schema_path: with open(schema_path, "r", encoding='utf-8') as stream: - schema = pyyaml.safe_load(stream) + schema = pyyaml.load(stream, Loader=self._yaml_loader) if SpecFamily.jsonschema is None: SpecFamily.jsonschema = importlib.import_module("jsonschema")
Signature · 1 line
2.54.0
Quoted 37 lines · Jakub Kicinski
Turns out Python YAML defaults to a pure Python loader for YAML files which is a lot slower than the C loader (using libyaml). Try to use the C one whenever possible.
The avg time to run:
$ tools/net/ynl/pyynl/cli.py --family tc --no-schema drops from 300+ ms to 115 ms with this change (40 samples).We could drop the load time further to 85 ms if we "compiled" the specs to JSON. Slightly tricky parts are that we don't currently install the specs at all on make install, so it's unclear where to put the conversion. Also JSON has questionable support for comments and we need an SPDX line.
Quoted 1 line
Signed-off-by Jakub Kicinski <[email protected]>---
CC: [email protected]--- tools/net/ynl/pyynl/lib/nlspec.py | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)tools/net/ynl/pyynl/lib/nlspec.py +5
-0@@ -439,6 +439,11 @@ import yaml as pyyaml # To be loaded dynamically as needed jsonschema = None + try: + _yaml_loader = pyyaml.CSafeLoader + except AttributeError: + _yaml_loader = pyyaml.SafeLoader + def __init__(self, spec_path, schema_path=None, exclude_ops=None): with open(spec_path, "r", encoding='utf-8') as stream: prefix = '# SPDX-License-Identifier: '
tools/net/ynl/pyynl/lib/nlspec.py import yaml as pyyaml · line 457
Quoted 6 lines
tools/net/ynl/pyynl/lib/nlspec.py +1
-1@@ -448,7 +453,7 @@ import yaml as pyyaml self.license = first[len(prefix):] stream.seek(0) - spec = pyyaml.safe_load(stream) + spec = pyyaml.load(stream, Loader=self._yaml_loader)
Hmm. I was a bit confused at first by the switch from safe_load to load.. but we're passing Loader as _yaml_loader which will either be CSafeLoader *or* the default SafeLoader, so we'll get the appropriate loader equivalent to what safe_load would have done, so there's no change. Ok
Quoted 11 lines
tools/net/ynl/pyynl/lib/nlspec.py +1
-1self.fixed_header = None self._resolution_list = []@@ -464,7 +469,7 @@ import yaml as pyyaml schema_path = os.path.dirname(os.path.dirname(spec_path)) + f'/{self.proto}.yaml' if schema_path: with open(schema_path, "r", encoding='utf-8') as stream: - schema = pyyaml.safe_load(stream) + schema = pyyaml.load(stream, Loader=self._yaml_loader) if SpecFamily.jsonschema is None: SpecFamily.jsonschema = importlib.import_module("jsonschema")
Jakub Kicinski
maintainerQuoted 5 lines · On Wed, 3 Jun 2026 15:08:46 -0700 Jacob Keller wrote:
Hmm. I was a bit confused at first by the switch from
safe_loadto load.. but we're passing Loader as_yaml_loaderwhich will either be CSafeLoader *or* the default SafeLoader, so we'll get the appropriate loader equivalent to whatsafe_loadwould have done, so there's no change. Ok
Maybe I should have mentioned this in the commit msg. I was also super confused by these APIs but IDK how much is this me not knowing Python and how much it's special. AFAIU basically:
somewhere in pyyaml...
def safe_load(file):
return load(file, Loader=SafeLoader)
so safe_load() is just a "shorthand" for using SafeLoader, which unlike load()s default loader doesn't allow constructing real/binary Python objects ?
Why it doesn't default to the C one is beyond my understanding.
Quoted 11 lines · Jakub Kicinski
Quoted 5 lines · On Wed, 3 Jun 2026 15:08:46 -0700 Jacob Keller wrote:
Hmm. I was a bit confused at first by the switch from
safe_loadto load.. but we're passing Loader as_yaml_loaderwhich will either be CSafeLoader *or* the default SafeLoader, so we'll get the appropriate loader equivalent to whatsafe_loadwould have done, so there's no change. OkMaybe I should have mentioned this in the commit msg. I was also super confused by these APIs but IDK how much is this me not knowing Python and how much it's special. AFAIU basically:
This is a pyyaml issue not a generic python one I think.
Quoted 8 lines
somewhere in pyyaml...
def
safe_load(file):
returnload(file, Loader=SafeLoader)so
safe_load()is just a "shorthand" for using SafeLoader, which unlike load()s default loader doesn't allow constructing real/binary Python objects ?
load() by default uses the generic Loader that supports the full spec and because of that is "unsafe":
https://github.com/yaml/pyyaml/wiki/PyYAML-yaml.load(input)-Deprecation
Basically, loading the full YAML spec on untrusted input is unsafe since it allows arbitrary execution.
Quoted 2 lines
Why it doesn't default to the C one is beyond my understanding.
Right. I'm not super familiar here as to how it ends up not defaulting to CSafeLoader, but probably its because its not always available.
Nicolai Buchwitz
Reviewed-byQuoted 29 lines · Jacob Keller
Quoted 11 lines · Jakub Kicinski
Quoted 5 lines · On Wed, 3 Jun 2026 15:08:46 -0700 Jacob Keller wrote:
Hmm. I was a bit confused at first by the switch from
safe_loadto load.. but we're passing Loader as_yaml_loaderwhich will either be CSafeLoader *or* the default SafeLoader, so we'll get the appropriate loader equivalent to whatsafe_loadwould have done, so there's no change. OkMaybe I should have mentioned this in the commit msg. I was also super confused by these APIs but IDK how much is this me not knowing Python and how much it's special. AFAIU basically:
This is a pyyaml issue not a generic python one I think.
Quoted 8 lines
somewhere in pyyaml...
def
safe_load(file):
returnload(file, Loader=SafeLoader)so
safe_load()is just a "shorthand" for using SafeLoader, which unlike load()s default loader doesn't allow constructing real/binary Python objects ?
load()by default uses the generic Loader that supports the full spec and because of that is "unsafe":https://github.com/yaml/pyyaml/wiki/PyYAML-yaml.load(input)-Deprecation
Quoted 7 lines
Basically, loading the full YAML spec on untrusted input is unsafe since it allows arbitrary execution.
Quoted 2 lines
Why it doesn't default to the C one is beyond my understanding.
Right. I'm not super familiar here as to how it ends up not defaulting to CSafeLoader, but probably its because its not always available.
That's correct. CSafeLoader is only available if libyaml-dev was present when pyyaml was built. Auto-enabling the libyaml bindings was requested a few years ago, but it stalled due to internals [1]. It would also require reworking safe_load() and others, which are atm hardcoded. Hence the usual try/except import pattern.
Anyway, the changes match the usual patterns.
[1] https://github.com/yaml/pyyaml/issues/437
Regards
Nicolai
Quoted 14 lines · Jakub Kicinski
Turns out Python YAML defaults to a pure Python loader for YAML files which is a lot slower than the C loader (using libyaml). Try to use the C one whenever possible.
The avg time to run:
$ tools/net/ynl/pyynl/cli.py --family tc --no-schema drops from 300+ ms to 115 ms with this change (40 samples).We could drop the load time further to 85 ms if we "compiled" the specs to JSON. Slightly tricky parts are that we don't currently install the specs at all on make install, so it's unclear where to put the conversion. Also JSON has questionable support for comments and we need an SPDX line.
Quoted 3 lines
Signed-off-by Jakub Kicinski <[email protected]>---
CC: [email protected]
This patch was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Hello:
This patch was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Quoted 9 lines · On Wed, 3 Jun 2026 14:08:10 -0700 you wrote:
Turns out Python YAML defaults to a pure Python loader for YAML files which is a lot slower than the C loader (using libyaml). Try to use the C one whenever possible.
The avg time to run:
$ tools/net/ynl/pyynl/cli.py --family tc --no-schema drops from 300+ ms to 115 ms with this change (40 samples).[...]
Here is the summary with links:
- [net-next] tools: ynl: try to avoid the very slow YAML loader
https://git.kernel.org/netdev/net-next/c/07c4bca9d92e
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html