Skip to contentSkip to conversation

openssl: set package level PERL5LIB

PATCH Superseded Yang Xu 3 participants 9 messages Apr 20, 2023 – Apr 25, 2023 1 message not in the archive

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Yang Xu

author
#0 lore raw

PATCH 1/1 openssl: set package level PERL5LIB

Build openssl with shared state cache may meet problem under below conditions:
1. sstate cache generated by another user on same machine
2. inappropriate PERL5LIB setting
3. openssl build from scratch but perl using cache

When building openssl, without appropriate PERL5LIB setting, perl tries to search Text::Template in no permission path and abort.

This patch does three things:
1. openssl upgrade external Text-Template from 1.46 to 1.56, correct perl module path.
2. add necessary module path while openssl build
3. set package level PERL5LIB to make perl found module in S

---
 meta/recipes-connectivity/openssl/openssl_3.1.0.bb | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
meta/recipes-connectivity/openssl/openssl_3.1.0.bb +5 -1
index b319c66044..ed1c579df9 100644
@@ -54,6 +54,10 @@ CFLAGS:append:class-nativesdk = " -DOPENSSLDIR=/not/builtin -DENGINESDIR=/not/bu
 # The default is to trust upstream choices.
 DEPRECATED_CRYPTO_FLAGS ?= ""
 
+# Set package level PERL5LIB
+PERL5LIB="${S}/external/perl/Text-Template-1.56/lib/:${S}/crypto/perlasm"
+export PERL5LIB
+
 do_configure () {
 	# When we upgrade glibc but not uninative we see obtuse failures in openssl. Make
 	# the issue really clear that perl isn't functional due to symbol mismatch issues.
@@ -138,7 +142,7 @@ do_configure () {
 	fi
 	# WARNING: do not set compiler/linker flags (-I/-D etc.) in EXTRA_OECONF, as they will fully replace the
 	# environment variables set by bitbake. Adjust the environment variables instead.
-	HASHBANGPERL="/usr/bin/env perl" PERL=perl PERL5LIB="${S}/external/perl/Text-Template-1.46/lib/" \
+	HASHBANGPERL="/usr/bin/env perl" PERL=perl \
 	perl ${S}/Configure ${EXTRA_OECONF} ${PACKAGECONFIG_CONFARGS} ${DEPRECATED_CRYPTO_FLAGS} --prefix=$useprefix --openssldir=${libdir}/ssl-3 --libdir=${libdir} $target
 	perl ${B}/configdata.pm --dump
 }
Signature · 1 line
2.25.1
#1 lore raw
Quoted 6 lines · Yang Xu

Build openssl with shared state cache may meet problem under below conditions:
1. sstate cache generated by another user on same machine
2. inappropriate PERL5LIB setting
3. openssl build from scratch but perl using cache

Quoted 8 lines

When building openssl, without appropriate PERL5LIB setting, perl tries to search Text::Template in no permission path and abort.

This patch does three things:
1. openssl upgrade external Text-Template from 1.46 to 1.56, correct perl module path.
2. add necessary module path while openssl build
3. set package level PERL5LIB to make perl found module in S

I'm not sure I understand the description of the problem, or the fix. What makes PERL5LIB setting inappropriate? Why is perl that has been built by a different user (and placed into sstate) not suitable? Doesn't this mean that perl from sstate can be broken in certain scenarios, and that needs to be fixed instead?

Alex

#2 lore raw

Dear Alexander,

Sorry, let me explain the problem we met.

The simplified environment and scenarios is:
1) User A, his home directory is /home/A, generate one sstate cache under /home/A.
2) This sstate cache moved to /home/Public, act as shared sstate cache
3) User B, his home directory is /home/B, using sstate cache in /home/Public and build yocto.
4) User B has no permission to access /home/A.

The problem we met is while openssl built from scratch but perl using cache,
it will be aborted at do_configure stage,
error message is "Can not locate Text/Template.pm at /home/A/[build path]/[perl module path]/Text/Template.pm : Permission Denied".

There are some information about perl:
1. perl will hardcode buildtime path in @INC in binary (module search directories)
2. when "use MODULE", perl will search @INC from begin to end until it found MODULE.
3. when perl search module through directories in @INC
3.1 if [directory]/MODULE.pm is not exist, search in next directory
in @INC.
3.2 if [directory]/MODULE.pm has no permission to access, perl will
ABORT.

While build openssl from from scratch with perl using cache, @INC directory order is:
1.<PERL5LIB>
2.<default module path in ${WORKDIR}/recipe-sysroot-native> 3.<hardcode module path in binary, which point to /home/A/XXX/YYY> 4.<module found by FindBin, module Text-Template can be found by it>

Q:What makes PERL5LIB setting inappropriate? Why is perl that has been built by a different user (and placed into sstate) not suitable? A:
There is no path external/perl/Text-Template-1.46/lib/ exists, openssl has upgraded to external/perl/Text-Template-1.56/lib/. With current recipe, highest priority folder <PERL5LIB> pointed to an non-exist folder and meaningless.
Lowest priority folder "<module found by FindBin>" includes modules in openssl source code.

While perl search "Text/Template.pm", it can not found it in <PERL5LIB> and <default module path>.
And then perl search module in <hardcode module path in binary>, *if /home/A is not existed, perl will search in <module found by FindBin> and found Text/Template.pm finally. (OK case)
*if /home/A is existed but has no permission access, perl will ABORT immediately. (NG case, case we met)

So my fixup is
1) set appropriate PERL5LIB to make perl can found Text/Tempalte.pm at highest priority folder <PERL5LIB>.
2) add other perl module directories needed by openssl build in PERL5LIB
3) set PERL5LIB as package environment variable

Q: Doesn't this mean that perl from sstate can be broken in certain scenarios, and that needs to be fixed instead?
A:
I think it is a module/mechanisms co-work problem because I think sstate cache and perl is behavior normally.
Caller of perl has response to prepare appropriate module setting.

Sorry, I hope I explain the situation clearly.

Thank you

Quoted 29 lines · Alexander Kanavin

External email : Please do not click links or open attachments until you have verified the sender or the content.

Quoted 11 lines · Yang Xu

Build openssl with shared state cache may meet problem under below conditions:
1. sstate cache generated by another user on same machine
2. inappropriate PERL5LIB setting
3. openssl build from scratch but perl using cache

When building openssl, without appropriate PERL5LIB setting, perl tries
to search Text::Template in no permission path and abort.

Quoted 5 lines

This patch does three things:
1. openssl upgrade external Text-Template from 1.46 to 1.56, correct perl module path.
2. add necessary module path while openssl build
3. set package level PERL5LIB to make perl found module in S

I'm not sure I understand the description of the problem, or the fix. What makes PERL5LIB setting inappropriate? Why is perl that has been built by a different user (and placed into sstate) not suitable? Doesn't this mean that perl from sstate can be broken in certain scenarios, and that needs to be fixed instead?

Alex

#3 lore raw

Thank you, I understand this now. I'm fine with the fix, but can you add this additional information to the commit message? So that it's available in git history if anyone needs to understand it.

Alex

Quoted 103 lines · Yang Xu (徐扬)

Dear Alexander,

Sorry, let me explain the problem we met.

The simplified environment and scenarios is:
1) User A, his home directory is /home/A, generate one sstate cache under /home/A.
2) This sstate cache moved to /home/Public, act as shared sstate cache
3) User B, his home directory is /home/B, using sstate cache in /home/Public and build yocto.
4) User B has no permission to access /home/A.

The problem we met is while openssl built from scratch but perl using cache,
it will be aborted at do_configure stage,
error message is "Can not locate Text/Template.pm at /home/A/[build path]/[perl module path]/Text/Template.pm : Permission Denied".

There are some information about perl:
1. perl will hardcode buildtime path in @INC in binary (module search directories)
2. when "use MODULE", perl will search @INC from begin to end until it found MODULE.
3. when perl search module through directories in @INC
3.1 if [directory]/MODULE.pm is not exist, search in next directory
in @INC.
3.2 if [directory]/MODULE.pm has no permission to access, perl will
ABORT.

While build openssl from from scratch with perl using cache, @INC directory order is:
1.<PERL5LIB>
2.<default module path in ${WORKDIR}/recipe-sysroot-native> 3.<hardcode module path in binary, which point to /home/A/XXX/YYY> 4.<module found by FindBin, module Text-Template can be found by it>

Q:What makes PERL5LIB setting inappropriate? Why is perl that has been built by a different user (and placed into sstate) not suitable? A:
There is no path external/perl/Text-Template-1.46/lib/ exists, openssl has upgraded to external/perl/Text-Template-1.56/lib/. With current recipe, highest priority folder <PERL5LIB> pointed to an non-exist folder and meaningless.
Lowest priority folder "<module found by FindBin>" includes modules in openssl source code.

While perl search "Text/Template.pm", it can not found it in <PERL5LIB> and <default module path>.
And then perl search module in <hardcode module path in binary>, *if /home/A is not existed, perl will search in <module found by FindBin> and found Text/Template.pm finally. (OK case)
*if /home/A is existed but has no permission access, perl will ABORT immediately. (NG case, case we met)

So my fixup is
1) set appropriate PERL5LIB to make perl can found Text/Tempalte.pm at highest priority folder <PERL5LIB>.
2) add other perl module directories needed by openssl build in PERL5LIB
3) set PERL5LIB as package environment variable

Q: Doesn't this mean that perl from sstate can be broken in certain scenarios, and that needs to be fixed instead?
A:
I think it is a module/mechanisms co-work problem because I think sstate cache and perl is behavior normally.
Caller of perl has response to prepare appropriate module setting.

Sorry, I hope I explain the situation clearly.

Thank you

Quoted 29 lines · Alexander Kanavin

External email : Please do not click links or open attachments until you have verified the sender or the content.

Quoted 11 lines · Yang Xu

Build openssl with shared state cache may meet problem under below conditions:
1. sstate cache generated by another user on same machine
2. inappropriate PERL5LIB setting
3. openssl build from scratch but perl using cache

When building openssl, without appropriate PERL5LIB setting, perl tries
to search Text::Template in no permission path and abort.

Quoted 5 lines

This patch does three things:
1. openssl upgrade external Text-Template from 1.46 to 1.56, correct perl module path.
2. add necessary module path while openssl build
3. set package level PERL5LIB to make perl found module in S

I'm not sure I understand the description of the problem, or the fix. What makes PERL5LIB setting inappropriate? Why is perl that has been built by a different user (and placed into sstate) not suitable? Doesn't this mean that perl from sstate can be broken in certain scenarios, and that needs to be fixed instead?

Alex

#4 lore raw
Quoted 3 lines · Alexander Kanavin

Thank you, I understand this now. I'm fine with the fix, but can you add this additional information to the commit message? So that it's available in git history if anyone needs to understand it.

I think there is something additional we need to do. If the path compiled into perl breaks when it is installed from sstate, we should make sure this path is *always* incorrect. That way things will work or fail deterministicly.

Cheers,

Richard

author
#5 lore raw

Thank you, I will submit a new patch with more detailed information.

Because two below behavior, I think it is a definite result under certain conditions.
1) @INC is set at build time with host hardcode path,
2) ABORT when searching module in @​INC paths exists but can not read.

Thank you

#6 lore raw
In reply to Richard Purdie
Quoted 8 lines · Richard Purdie
Quoted 3 lines

Thank you, I understand this now. I'm fine with the fix, but can you add this additional information to the commit message? So that it's available in git history if anyone needs to understand it.

I think there is something additional we need to do. If the path compiled into perl breaks when it is installed from sstate, we should make sure this path is *always* incorrect. That way things will work or fail deterministicly.

That's right. I'll take a look at what happens inside perl, finding the correct place to patch can be tricky.

Alex

A message that is not in the archive (sent off-list or not yet received): [email protected]

#8 lore raw
Quoted 11 lines · Alexander Kanavin via lists.openembedded.org
Quoted 8 lines
Quoted 3 lines

Thank you, I understand this now. I'm fine with the fix, but can you add this additional information to the commit message? So that it's available in git history if anyone needs to understand it.

I think there is something additional we need to do. If the path compiled into perl breaks when it is installed from sstate, we should make sure this path is *always* incorrect. That way things will work or fail deterministicly.

That's right. I'll take a look at what happens inside perl, finding the correct place to patch can be tricky.

I have a patch for this now, which I'll take into testing:

meta/recipes-devtools/perl/perl_5.36.0.bb +3 -0
@@ -109,6 +109,10 @@ do_configure:class-native() {
     -Ui_xlocale \
     -Alddlflags=' ${LDFLAGS}' \
     ${PACKAGECONFIG_CONFARGS}
+
+    # This prevents leakage of build paths into perl-native binaries, which
+    # causes non-deterministic troubles when those paths no longer

exist or aren't accessible.
+ sed -i -e "s,${STAGING_LIBDIR},/completelyboguspath,g" config.h
}

do_configure:append() {

Alex

#9 lore raw
In reply to Richard Purdie

Dear Sirs,

Sorry, please NOT accept this patch because I found it will break its own sstate cache.

I will found other solution for this openssl/perl co-work issue.

Quoted 19 lines · Richard Purdie

External email : Please do not click links or open attachments until you have verified the sender or the content.

Quoted 4 lines · Alexander Kanavin

Thank you, I understand this now. I'm fine with the fix, but can you
add this additional information to the commit message? So that it's available in git history if anyone needs to understand it.

I think there is something additional we need to do. If the path compiled into perl breaks when it is installed from sstate, we should make sure this path is *always* incorrect. That way things will work or
fail deterministicly.

Cheers,

Richard