Skip to contentSkip to conversation

archiver: avoid empty incfile in ar_recipe

PATCH In review grygorii tertychnyi 2 participants 9 messages Nov 1, 2019 – Nov 12, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 archiver: avoid empty incfile in ar_recipe

do_ar_recipe fails on perf recipe on line:

include ${@bb.utils.contains('PACKAGECONFIG', 'scripting', 'perf-perl.inc', '', d)}

1. "${...}" part expands into empty string
2. bb.utils.which() takes empty string and returns first directory name from bbpath
3. shutil.copy() fails on copying directory:

Exception: IsADirectoryError: [Errno 21] Is a directory: ......

Hence, check "incfile" variable on each step.

---
 meta/classes/archiver.bbclass | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
meta/classes/archiver.bbclass +3 -2
index 093e2d95af..7c46cff91f 100644
@@ -441,9 +441,10 @@ python do_ar_recipe () {
             incfile = include_re.match(line).group(1)
         if incfile:
             incfile = d.expand(incfile)
+        if incfile:
             incfile = bb.utils.which(bbpath, incfile)
-            if incfile:
-                shutil.copy(incfile, outdir)
+        if incfile:
+            shutil.copy(incfile, outdir)
 
     create_tarball(d, outdir, 'recipe', d.getVar('ARCHIVER_OUTDIR'))
     bb.utils.remove(outdir, recurse=True)
Signature · 1 line
2.23.0
#1 lore raw

Hi,

Quoted 6 lines · grygorii tertychnyi via Openembedded-core

do_ar_recipe fails on perf recipe on line:

include ${@bb.utils.contains('PACKAGECONFIG', 'scripting', 'perf-perl.inc', '', d)}

1. "${...}" part expands into empty string
2. bb.utils.which() takes empty string and returns first directory name from bbpath

This doesn't sound sane. If the include directive has no argument, incfile should end up None. That's what the code "assumes" at this point. I would fix it either at the regex expression level or strip the match. I reckon former would make more sense. It sounds like the match expression should use * (none or more).

Quoted 29 lines

3. shutil.copy() fails on copying directory:

Exception: IsADirectoryError: [Errno 21] Is a directory: ......

Hence, check "incfile" variable on each step.

---
  meta/classes/archiver.bbclass | 5 +++--
  1 file changed, 3 insertions(+), 2 deletions(-)
meta/classes/archiver.bbclass +3 -2
index 093e2d95af..7c46cff91f 100644
@@ -441,9 +441,10 @@ python do_ar_recipe () {
              incfile = include_re.match(line).group(1)
          if incfile:
              incfile = d.expand(incfile)
+        if incfile:
              incfile = bb.utils.which(bbpath, incfile)
-            if incfile:
-                shutil.copy(incfile, outdir)
+        if incfile:
+            shutil.copy(incfile, outdir)
 
      create_tarball(d, outdir, 'recipe', d.getVar('ARCHIVER_OUTDIR'))
      bb.utils.remove(outdir, recurse=True)
Signature · 1 line
Andrei Gherzan
#2 lore raw
In reply to grygorii tertychnyi

Hi,

Quoted 6 lines · grygorii tertychnyi via Openembedded-core

do_ar_recipe fails on perf recipe on line:

include ${@bb.utils.contains('PACKAGECONFIG', 'scripting', 'perf-perl.inc', '', d)}

1. "${...}" part expands into empty string
2. bb.utils.which() takes empty string and returns first directory name from bbpath

This doesn't sound sane. If the include directive has no argument, incfile should end up None. That's what the code "assumes" at this point. I would fix it either at the regex expression level or stripping the matched string. I reckon the former makes more sense (.*).

--
Andrei Gherzan

#3 lore raw

Andrei,

From: Andrei Gherzan <[email protected]>
Sent: Friday, November 1, 2019 13:28
To: Grygorii Tertychnyi (gtertych); [email protected]
Cc: xe-linux-external(mailer list)
Subject: Re: [OE-core] [PATCH] archiver: avoid empty incfile in ar_recipe

Quoted 9 lines
Quoted 6 lines

do_ar_recipe fails on perf recipe on line:

include ${@bb.utils.contains('PACKAGECONFIG', 'scripting', 'perf-perl.inc', '', d)}

1. "${...}" part expands into empty string
2. bb.utils.which() takes empty string and returns first directory name from bbpath

This doesn't sound sane. If the include directive has no argument, incfile should end up None. That's what the code "assumes" at this

I agree.

Quoted 3 lines

point. I would fix it either at the regex expression level or stripping the matched string. I reckon the former makes more sense (.*).

Not sure I understand. Archiver class does not interpret "include" directive. It just parses text files. The regular expression looks correct:

These lines:

440 elif include_re.match(line):
441 incfile = include_re.match(line).group(1)

put "${...}" _string_ into "incfile" variable. So, "incfile" is not None at this stage. Then,

443 incfile = d.expand(incfile)

Now "incfile" is empty and nobody checks it.

444 incfile = bb.utils.which(bbpath, incfile)

Now "incfile" is set to first directory name in BBPATH (wrong behavour?)

445 if incfile:
446 shutil.copy(incfile, outdir)

Exception here: "incfile" is directory, not a file.

Signature · 1 line
Grygorii
#4 lore raw

ping

________________________________________
From: Grygorii Tertychnyi (gtertych) <[email protected]>
Sent: Monday, November 4, 2019 17:09
To: Andrei Gherzan; [email protected]
Cc: xe-linux-external(mailer list)
Subject: Re: [OE-core] [PATCH] archiver: avoid empty incfile in ar_recipe

Andrei,

From: Andrei Gherzan <[email protected]>
Sent: Friday, November 1, 2019 13:28
To: Grygorii Tertychnyi (gtertych); [email protected]
Cc: xe-linux-external(mailer list)
Subject: Re: [OE-core] [PATCH] archiver: avoid empty incfile in ar_recipe

Quoted 9 lines
Quoted 6 lines

do_ar_recipe fails on perf recipe on line:

include ${@bb.utils.contains('PACKAGECONFIG', 'scripting', 'perf-perl.inc', '', d)}

1. "${...}" part expands into empty string
2. bb.utils.which() takes empty string and returns first directory name from bbpath

This doesn't sound sane. If the include directive has no argument, incfile should end up None. That's what the code "assumes" at this

I agree.

Quoted 3 lines

point. I would fix it either at the regex expression level or stripping the matched string. I reckon the former makes more sense (.*).

Not sure I understand. Archiver class does not interpret "include" directive. It just parses text files. The regular expression looks correct:

These lines:

440 elif include_re.match(line):
441 incfile = include_re.match(line).group(1)

put "${...}" _string_ into "incfile" variable. So, "incfile" is not None at this stage. Then,

443 incfile = d.expand(incfile)

Now "incfile" is empty and nobody checks it.

444 incfile = bb.utils.which(bbpath, incfile)

Now "incfile" is set to first directory name in BBPATH (wrong behavour?)

445 if incfile:
446 shutil.copy(incfile, outdir)

Exception here: "incfile" is directory, not a file.

--
Grygorii

#5 lore raw
In reply to Grygorii Tertychnyi (gtertych)

Hi Grygorii,

Quoted 27 lines · Grygorii Tertychnyi (gtertych)

Andrei,

From: Andrei Gherzan <[email protected]>
Sent: Friday, November 1, 2019 13:28
To: Grygorii Tertychnyi (gtertych); [email protected]
Cc: xe-linux-external(mailer list)
Subject: Re: [OE-core] [PATCH] archiver: avoid empty incfile in ar_recipe

Quoted 9 lines
Quoted 6 lines

do_ar_recipe fails on perf recipe on line:

include ${@bb.utils.contains('PACKAGECONFIG', 'scripting', 'perf-perl.inc', '', d)}

1. "${...}" part expands into empty string
2. bb.utils.which() takes empty string and returns first directory name from bbpath

This doesn't sound sane. If the include directive has no argument, incfile should end up None. That's what the code "assumes" at this

I agree.

Quoted 3 lines

point. I would fix it either at the regex expression level or stripping the matched string. I reckon the former makes more sense (.*).

Not sure I understand. Archiver class does not interpret "include" directive. It just parses text files. The regular expression looks correct:

Quoted 6 lines

These lines:

440 elif include_re.match(line):
441 incfile = include_re.match(line).group(1)

put "${...}" _string_ into "incfile" variable. So, "incfile" is not None at "this stage.

"${...}" it's already expanded to a white-space. So in that case it matches "include ".

Quoted 5 lines

Then,

443 incfile = d.expand(incfile)

Now "incfile" is empty and nobody checks it.

incfile is not empty. It's actually a string containing one white-space.

Quoted 10 lines

444 incfile = bb.utils.which(bbpath, incfile)

Now "incfile" is set to first directory name in BBPATH (wrong behavour?)

445 if incfile:
446 shutil.copy(incfile, outdir)

Exception here: "incfile" is directory, not a file.

The include regex is the following:

include_re = re.compile( r"include\s+(.+)" )

The issue is that when this is matched on a string suffixed with only spaces, it will match the last space as group(1). This is because ".+" forces to match the last white-space. Changing that to ".*" will make the group(1) be an empty string and later the if will evaluate False.

Signature · 1 line
Andrei Gherzan
#6 lore raw

Hi Andrei,

From: Andrei Gherzan <[email protected]>
Sent: Monday, November 11, 2019 13:18
Subject: Re: [OE-core] [PATCH] archiver: avoid empty incfile in ar_recipe

Quoted 5 lines
Quoted 5 lines

Not sure I understand. Archiver class does not interpret "include" directive. It just parses text files. The regular expression looks correct:

These lines:

Quoted 7 lines
Quoted 4 lines

440 elif include_re.match(line):
441 incfile = include_re.match(line).group(1)

put "${...}" _string_ into "incfile" variable. So, "incfile" is not None at "this stage.

"${...}" it's already expanded to a white-space. So in that case it matches "include ".

No. "incfile" is not expanded here, it actually contains "${...}", e.g. ${@bb.utils.contains('PACKAGECONFIG', 'scripting', 'perf-perl.inc', '', d)}

Quoted 7 lines
Quoted 5 lines

Then,

443 incfile = d.expand(incfile)

Now "incfile" is empty and nobody checks it.

incfile is not empty. It's actually a string containing one white-space.

It is actually empty, e.g. "".

Quoted 11 lines
Quoted 10 lines

444 incfile = bb.utils.which(bbpath, incfile)

Now "incfile" is set to first directory name in BBPATH (wrong behavour?)

445 if incfile:
446 shutil.copy(incfile, outdir)

Exception here: "incfile" is directory, not a file.

Quoted 8 lines

The include regex is the following:

include_re = re.compile( r"include\s+(.+)" )

The issue is that when this is matched on a string suffixed with only spaces, it will match the last space as group(1). This is because ".+" forces to match the last white-space. Changing that to ".*" will make the group(1) be an empty string and later the if will evaluate False.

I don't see how it helps here.

Signature · 1 line
Grygorii
#7 lore raw

Hi,

Quoted 14 lines · Grygorii Tertychnyi (gtertych)

Hi Andrei,

From: Andrei Gherzan <[email protected]>
Sent: Monday, November 11, 2019 13:18
Subject: Re: [OE-core] [PATCH] archiver: avoid empty incfile in ar_recipe

Quoted 8 lines
Quoted 8 lines

Not sure I understand. Archiver class does not interpret "include" directive. It just parses text files. The regular expression looks correct:

These lines:

440 elif include_re.match(line):
441 incfile = include_re.match(line).group(1)

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

put "${...}" _string_ into "incfile" variable. So, "incfile" is not None at "this stage.

"${...}" it's already expanded to a white-space. So in that case it matches "include ".

No. "incfile" is not expanded here, it actually contains "${...}", e.g. ${@bb.utils.contains('PACKAGECONFIG', 'scripting', 'perf-perl.inc', '', d)}

Give me a simple way to reproduce this.

Signature · 1 line
Andrei Gherzan
#8 lore raw

Hi Andrei,

Quoted 19 lines · Andrei Gherzan

Hi,

Quoted 16 lines · Grygorii Tertychnyi (gtertych)

Hi Andrei,

From: Andrei Gherzan <[email protected]>
Sent: Monday, November 11, 2019 13:18
Subject: Re: [OE-core] [PATCH] archiver: avoid empty incfile in ar_recipe

Quoted 9 lines
Quoted 9 lines

Not sure I understand. Archiver class does not interpret "include" directive. It just parses text files. The regular expression looks correct:

These lines:

440 elif include_re.match(line):
441 incfile = include_re.match(line).group(1)

put "${...}" _string_ into "incfile" variable. So, "incfile" is not None at "this stage.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

"${...}" it's already expanded to a white-space. So in that case it matches "include ".

No. "incfile" is not expanded here, it actually contains "${...}", e.g. ${@bb.utils.contains('PACKAGECONFIG', 'scripting', 'perf-perl.inc', '', d)}

Give me a simple way to reproduce this.

1. Put two lines to local.conf
% cat <<CONF >> conf/local.conf
PACKAGECONFIG_remove_pn-perf = "scripting"
INHERIT += "archiver"
CONF

2. Run do_ar_recipe for perf
% bitbake -c ar_recipe perf

3. Observe the output error:
Exception: IsADirectoryError: [Errno 21] Is a directory

Signature · 1 line
Grygorii