python3-cython: make generated _cyutility.c be reproducible
PATCH 1/1 python3-cython: make generated _cyutility.c be reproducible
While python3 module use cython to build library, the generated source file _cyutility.c is not stable at each build and made the generated library not be reproducible
This commit replace un-predictable tmpfolder string with fixed symlink in generated source file to assure the generated library should be reproducible.
--- ...link-to-the-unpredictable-tmpdir-for.patch | 65 +++++++++++++++++++ .../python/python3-cython_3.2.4.bb | 1 + 2 files changed, 66 insertions(+) create mode 100644 meta/recipes-devtools/python/python3-cython/0001-Create-fixed-symlink-to-the-unpredictable-tmpdir-for.patch
meta/recipes-devtools/python/python3-cython/0001-Create-fixed-symlink-to-the-unpredictable-tmpdir-for.patch new file +65 -0
@@ -0,0 +1,65 @@ +From 42bb8d553defb250cd3ccdd539ff5a9c14a8421a Mon Sep 17 00:00:00 2001 +From: Zhixiong Chi <[email protected]> +Date: Thu, 23 Apr 2026 00:19:59 -0700 +Subject: [PATCH] Create fixed symlink to the unpredictable tmpdir for + sharedmodule + +When the option "--generated-shared" is used to generated __cyutility as the +link https://github.com/scikit-learn/scikit-learn/pull/31151/files, the path +for filename_table in the generated pyx/c file contains tmp dir which is not +predictable, and it caused the generated output file is not stable at each +build and made the generated library is not reproducible [1] between builds. + +example as python3_pandas: + +vim build/_cyutility.c +...... +/* #### Code section: filename_table ### */ + +static const char* const __pyx_f[] = { + "../../../../../../../../../../../../tmp/tmpXXXXXX/_cyutility.pyx", + "<stringsource>", +}; + +Replace with the fixed tmp symlink, it is no harm to tweak comments in source +file, after applied this commit, vim build/_cyutility.c +...... +/* #### Code section: filename_table ### */ + +static const char* const __pyx_f[] = { + "../../../../../../../../../../../../tmp/cython_shared/_cyutility.pyx", + "<stringsource>", +}; + +[1] https://reproducible-builds.org/ + +Upstream-Status: Submitted [https://github.com/cython/cython/pull/7634] +Signed-off-by: Zhixiong Chi <[email protected]> +--- + Cython/Build/SharedModule.py | 9 +++++++-- + 1 file changed, 7 insertions(+), 2 deletions(-) + +diff --git a/Cython/Build/SharedModule.py b/Cython/Build/SharedModule.py +index b466e57ed..f99fdabb6 100644 +--- a/Cython/Build/SharedModule.py ++++ b/Cython/Build/SharedModule.py +@@ -76,9 +76,14 @@ def generate_shared_module(options): + context = Main.Context.from_options(options) + scope = Symtab.ModuleScope('MemoryView', parent_module = None, context = context, is_package=False) + ++ fixed_tmp_link = "/tmp/cython_shared" ++ + with tempfile.TemporaryDirectory() as tmpdirname: +- pyx_file = os.path.join(tmpdirname, f'{module_name}.pyx') +- c_file = os.path.join(tmpdirname, f'{module_name}.c') ++ if os.path.exists(fixed_tmp_link) or os.path.islink(fixed_tmp_link): ++ os.unlink(fixed_tmp_link) ++ os.symlink(tmpdirname, fixed_tmp_link) ++ pyx_file = os.path.join(fixed_tmp_link, f'{module_name}.pyx') ++ c_file = os.path.join(fixed_tmp_link, f'{module_name}.c') + with open(pyx_file, 'w'): + pass + source_desc = FileSourceDescriptor(pyx_file) +-- +2.49.0 +
meta/recipes-devtools/python/python3-cython_3.2.4.bb +1 -0
@@ -13,6 +13,7 @@ inherit pypi setuptools3 cython SRC_URI += " \ file://0001-Replace-not-predictable-build-path-prefix-with-hardc.patch \ + file://0001-Create-fixed-symlink-to-the-unpredictable-tmpdir-for.patch \ " # No need to depend on self
Signature · 1 line
2.49.0
Quoted 74 lines · Zhixiong Chi via lists.openembedded.org
While python3 module use cython to build library, the generated source file
_cyutility.cis not stable at each build and made the generated library not be reproducibleThis commit replace un-predictable tmpfolder string with fixed symlink in generated source file to assure the generated library should be reproducible.
Signed-off-by Zhixiong Chi <[email protected]>---
...link-to-the-unpredictable-tmpdir-for.patch | 65 +++++++++++++++++++ .../python/python3-cython_3.2.4.bb | 1 +2 files changed, 66 insertions(+)create mode 100644
meta/recipes-devtools/python/python3-cython/0001-Create-fixed-symlink-to-the-unpredictable-tmpdir-for.patchmeta/recipes-devtools/python/python3-cython/0001-Create-fixed-symlink-to-the-unpredictable-tmpdir-for.patch new file +53
-0@@ -0,0 +1,65 @@ +From 42bb8d553defb250cd3ccdd539ff5a9c14a8421a Mon Sep 17 00:00:00 2001 +From: Zhixiong Chi <[email protected]> +Date: Thu, 23 Apr 2026 00:19:59 -0700 +Subject: [PATCH] Create fixed symlink to the unpredictable tmpdir for + sharedmodule + +When the option "--generated-shared" is used to generated __cyutility as the +link https://github.com/scikit-learn/scikit-learn/pull/31151/files, the path +for filename_table in the generated pyx/c file contains tmp dir which is not +predictable, and it caused the generated output file is not stable at each +build and made the generated library is not reproducible [1] between builds. + +example as python3_pandas: + +vim build/_cyutility.c +...... +/* #### Code section: filename_table ### */ + +static const char* const __pyx_f[] = { + "../../../../../../../../../../../../tmp/tmpXXXXXX/_cyutility.pyx", + "<stringsource>", +}; + +Replace with the fixed tmp symlink, it is no harm to tweak comments in source +file, after applied this commit, vim build/_cyutility.c +...... +/* #### Code section: filename_table ### */ + +static const char* const __pyx_f[] = { + "../../../../../../../../../../../../tmp/cython_shared/_cyutility.pyx", + "<stringsource>", +}; + +[1] https://reproducible-builds.org/ + +Upstream-Status: Submitted [https://github.com/cython/cython/pull/7634] +Signed-off-by: Zhixiong Chi <[email protected]> +--- + Cython/Build/SharedModule.py | 9 +++++++-- + 1 file changed, 7 insertions(+), 2 deletions(-) + +diff --git a/Cython/Build/SharedModule.py b/Cython/Build/SharedModule.py +index b466e57ed..f99fdabb6 100644 +--- a/Cython/Build/SharedModule.py ++++ b/Cython/Build/SharedModule.py +@@ -76,9 +76,14 @@ def generate_shared_module(options): + context = Main.Context.from_options(options) + scope = Symtab.ModuleScope('MemoryView', parent_module = None, context = context, is_package=False) + ++ fixed_tmp_link = "/tmp/cython_shared" ++ + with tempfile.TemporaryDirectory() as tmpdirname: +- pyx_file = os.path.join(tmpdirname, f'{module_name}.pyx')
meta/recipes-devtools/python/python3-cython/0001-Create-fixed-symlink-to-the-unpredictable-tmpdir-for.patch line 63
Quoted 10 lines
meta/recipes-devtools/python/python3-cython/0001-Create-fixed-symlink-to-the-unpredictable-tmpdir-for.patch new file +10
-0+- c_file = os.path.join(tmpdirname, f'{module_name}.c') ++ if os.path.exists(fixed_tmp_link) or os.path.islink(fixed_tmp_link): ++ os.unlink(fixed_tmp_link) ++ os.symlink(tmpdirname, fixed_tmp_link) ++ pyx_file = os.path.join(fixed_tmp_link, f'{module_name}.pyx') ++ c_file = os.path.join(fixed_tmp_link, f'{module_name}.c') + with open(pyx_file, 'w'): + pass + source_desc = FileSourceDescriptor(pyx_file) +--
During review we realised this hardcodes a /tmp/ path. What happens if there are mutliple builds running in parallel?
Ross also mentioned we may be able to tweak the cython class to improve things.
Cheers,
Richard
Sorry for the late response. I realized this method may block multiple builds in parallel and will send V2 with another way to fix this issue later. Thanks.
________________________________
From: Richard Purdie <[email protected]>
Sent: Tuesday, April 28, 2026 21:30
To: Chi, Zhixiong <[email protected]>; [email protected] <[email protected]>
Subject: Re: [OE-core][PATCH] python3-cython: make generated _cyutility.c be reproducible
CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know the content is safe.
Quoted 74 lines · Zhixiong Chi via lists.openembedded.org
While python3 module use cython to build library, the generated source file
_cyutility.cis not stable at each build and made the generated library not be reproducibleThis commit replace un-predictable tmpfolder string with fixed symlink in generated source file to assure the generated library should be reproducible.
Signed-off-by Zhixiong Chi <[email protected]>--- ...link-to-the-unpredictable-tmpdir-for.patch | 65 +++++++++++++++++++ .../python/python3-cython_3.2.4.bb | 1 + 2 files changed, 66 insertions(+) create mode 100644 meta/recipes-devtools/python/python3-cython/0001-Create-fixed-symlink-to-the-unpredictable-tmpdir-for.patchmeta/recipes-devtools/python/python3-cython/0001-Create-fixed-symlink-to-the-unpredictable-tmpdir-for.patch new file +53
-0@@ -0,0 +1,65 @@ +From 42bb8d553defb250cd3ccdd539ff5a9c14a8421a Mon Sep 17 00:00:00 2001 +From: Zhixiong Chi <[email protected]> +Date: Thu, 23 Apr 2026 00:19:59 -0700 +Subject: [PATCH] Create fixed symlink to the unpredictable tmpdir for + sharedmodule + +When the option "--generated-shared" is used to generated __cyutility as the +link https://github.com/scikit-learn/scikit-learn/pull/31151/files, the path +for filename_table in the generated pyx/c file contains tmp dir which is not +predictable, and it caused the generated output file is not stable at each +build and made the generated library is not reproducible [1] between builds. + +example as python3_pandas: + +vim build/_cyutility.c +...... +/* #### Code section: filename_table ### */ + +static const char* const __pyx_f[] = { + "../../../../../../../../../../../../tmp/tmpXXXXXX/_cyutility.pyx", + "<stringsource>", +}; + +Replace with the fixed tmp symlink, it is no harm to tweak comments in source +file, after applied this commit, vim build/_cyutility.c +...... +/* #### Code section: filename_table ### */ + +static const char* const __pyx_f[] = { + "../../../../../../../../../../../../tmp/cython_shared/_cyutility.pyx", + "<stringsource>", +}; + +[1] https://reproducible-builds.org/ + +Upstream-Status: Submitted [https://github.com/cython/cython/pull/7634] +Signed-off-by: Zhixiong Chi <[email protected]> +--- + Cython/Build/SharedModule.py | 9 +++++++-- + 1 file changed, 7 insertions(+), 2 deletions(-) + +diff --git a/Cython/Build/SharedModule.py b/Cython/Build/SharedModule.py +index b466e57ed..f99fdabb6 100644 +--- a/Cython/Build/SharedModule.py ++++ b/Cython/Build/SharedModule.py +@@ -76,9 +76,14 @@ def generate_shared_module(options): + context = Main.Context.from_options(options) + scope = Symtab.ModuleScope('MemoryView', parent_module = None, context = context, is_package=False) + ++ fixed_tmp_link = "/tmp/cython_shared" ++ + with tempfile.TemporaryDirectory() as tmpdirname: +- pyx_file = os.path.join(tmpdirname, f'{module_name}.pyx')
meta/recipes-devtools/python/python3-cython/0001-Create-fixed-symlink-to-the-unpredictable-tmpdir-for.patch line 63
Quoted 10 lines
meta/recipes-devtools/python/python3-cython/0001-Create-fixed-symlink-to-the-unpredictable-tmpdir-for.patch new file +10
-0+- c_file = os.path.join(tmpdirname, f'{module_name}.c') ++ if os.path.exists(fixed_tmp_link) or os.path.islink(fixed_tmp_link): ++ os.unlink(fixed_tmp_link) ++ os.symlink(tmpdirname, fixed_tmp_link) ++ pyx_file = os.path.join(fixed_tmp_link, f'{module_name}.pyx') ++ c_file = os.path.join(fixed_tmp_link, f'{module_name}.c') + with open(pyx_file, 'w'): + pass + source_desc = FileSourceDescriptor(pyx_file) +--
During review we realised this hardcodes a /tmp/ path. What happens if there are mutliple builds running in parallel?
Ross also mentioned we may be able to tweak the cython class to improve things.
Cheers,
Richard