Skip to contentSkip to conversation

command.py: cleanup bb.cache.parse_recipe

PATCH In review Chen Qi 4 participants 7 messages Nov 24, 2022 – Dec 1, 2022

lore
Share & tools
lore

Chen Qi

author
#0 lore raw

PATCH 1/1 command.py: cleanup bb.cache.parse_recipe

The parse_recipe has been removed from bb.cache, replace it with the databuilder._parse_recipe.

---
 lib/bb/command.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/bb/command.py +1 -1
index fa9fd054c..c9571f63d 100644
@@ -564,7 +564,7 @@ class CommandsSync:
         if config_data:
             # We have to use a different function here if we're passing in a datastore
             # NOTE: we took a copy above, so we don't do it here again
-            envdata = bb.cache.parse_recipe(config_data, fn, appendfiles, mc)['']
+            envdata = command.cooker.databuilder._parse_recipe(config_data, fn, appendfiles, mc)['']
         else:
             # Use the standard path
             envdata = command.cooker.databuilder.parseRecipe(fn, appendfiles)
Signature · 1 line
2.37.1
#1 lore raw
Quoted 13 lines · Chen Qi

The parse_recipe has been removed from bb.cache, replace it with the databuilder._parse_recipe.

---
 lib/bb/command.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/bb/command.py +0 -0
index fa9fd054c..c9571f63d 100644

lib/bb/command.py class CommandsSync: · line 570

Quoted 8 lines
lib/bb/command.py +1 -1
@@ -564,7 +564,7 @@ class CommandsSync:
         if config_data:
             # We have to use a different function here if we're passing in a datastore
             # NOTE: we took a copy above, so we don't do it here again
-            envdata = bb.cache.parse_recipe(config_data, fn, appendfiles, mc)['']
+            envdata = command.cooker.databuilder._parse_recipe(config_data, fn, appendfiles, mc)['']
         else:
             # Use the standard path
             envdata = command.cooker.databuilder.parseRecipe(fn, appendfiles)

How did you run into that? I'm a bit worried we're missing some test coverage here?

Cheers,

Richard

author
#2 lore raw

Hi Richard,

I met the error on our layerindex.

Error message is as below.
NOTE: Starting bitbake server...
ERROR: Unable to read /home/oelayer/layerindex/git___lxgit_wrs_com_layers_meta-arm/meta-arm/recipes-bsp/trusted -firmware-m/trusted-firmware-m_1.6.0.bb: Traceback (most recent call last):
File "/home/oelayer/layerindex/git___lxgit_wrs_com_bitbake/lib/bb/command.py", line 90, in runCommand
result = command_method(self, commandline)
File "/home/oelayer/layerindex/git___lxgit_wrs_com_bitbake/lib/bb/command.py", line 567, in parseRecipeFile
envdata = bb.cache.parse_recipe(config_data, fn, appendfiles, mc)['']
AttributeError: module 'bb.cache' has no attribute 'parse_recipe'

Regards,
Qi

-----Original Message-----
From: Richard Purdie <[email protected]>
Sent: Thursday, November 24, 2022 8:02 PM
To: Chen, Qi <[email protected]>; [email protected]
Subject: Re: [bitbake-devel] [PATCH] command.py: cleanup bb.cache.parse_recipe

Quoted 23 lines · Chen Qi

The parse_recipe has been removed from bb.cache, replace it with the databuilder._parse_recipe.

---
 lib/bb/command.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/bb/command.py index renamed from lib/bb/command.py +0 -0

fa9fd054c..c9571f63d 100644

lib/bb/command.py +3 -1
@@ -564,7 +564,7 @@ class CommandsSync:
         if config_data:
             # We have to use a different function here if we're passing in a datastore
             # NOTE: we took a copy above, so we don't do it here again
-            envdata = bb.cache.parse_recipe(config_data, fn, appendfiles, mc)['']
+            envdata = 
+ command.cooker.databuilder._parse_recipe(config_data, fn, 
+ appendfiles, mc)['']
         else:
             # Use the standard path
             envdata = command.cooker.databuilder.parseRecipe(fn, 
Quoted 1 line

appendfiles)

How did you run into that? I'm a bit worried we're missing some test coverage here?

Cheers,

Richard

#3 lore raw

Hi Qi,

Quoted 11 lines · Qi

I met the error on our layerindex.

Error message is as below.
NOTE: Starting bitbake server...
ERROR: Unable to read /home/oelayer/layerindex/git___lxgit_wrs_com_layers_meta-arm/meta-arm/recipes-bsp/trusted -firmware-m/trusted-firmware-m_1.6.0.bb: Traceback (most recent call last):
File "/home/oelayer/layerindex/git___lxgit_wrs_com_bitbake/lib/bb/command.py", line 90, in runCommand
result = command_method(self, commandline)
File "/home/oelayer/layerindex/git___lxgit_wrs_com_bitbake/lib/bb/command.py", line 567, in parseRecipeFile
envdata = bb.cache.parse_recipe(config_data, fn, appendfiles, mc)['']
AttributeError: module 'bb.cache' has no attribute 'parse_recipe'

Thanks.

I had a look at what could call parseRecipeFile(). I can see some commented out code in layerindex but it doesn't pass a config_data. The only other calls are from tinfoil's parse_recipe_file() function and none of those I can see pass config_data either.

Can you see where the config_data is coming from?

It looks like we've broken this at least once before:

https://git.yoctoproject.org/poky/commit/bitbake/lib/bb/command.py?id=70ee928f8f4152f198f4db61517c958d54f1c0a7

so we really need to figure out what is using it and add a test to ensure coverage. Is this some local customisation you have to layerindex?

Cheers,

Richard

#4 lore raw
In reply to Chen, Qi
Quoted 4 lines · Chen Qi

Hi Richard,

I met the error on our layerindex.

"our" is
https://layers.openembedded.org

or some other instance in windriver?

I'm asking because parsing on https://layers.openembedded.org seems to be broken for a while. I've checked randomly selected 3 layers and all show the same issue as "Updates" tab in:
https://layers.openembedded.org/layerindex/branch/master/layer/openembedded-core/

ERROR: fatal: Unable to create '/opt/workdir/git___git_openembedded_org_bitbake/.git/index.lock': File exists.

Another git process seems to be running in this repository, e.g. an editor opened by 'git commit'. Please make sure all processes are terminated then try again. If it still fails, a git process may have crashed in this repository earlier:
remove the file manually to continue.
ERROR: Traceback (most recent call last):
File "update_layer.py", line 384, in main
(tinfoil, tempdir) = recipeparse.init_parser(settings, branch,
bitbakepath, nocheckout=options.nocheckout, logger=logger)
File "/opt/layerindex/layerindex/recipeparse.py", line 36, in init_parser
utils.checkout_repo(bitbakepath, bitbake_ref, logger=logger)
File "/opt/layerindex/layerindex/utils.py", line 282, in checkout_repo

    runcmd(['git', 'reset', '--hard'], repodir, logger=logger)

File "/opt/layerindex/layerindex/utils.py", line 369, in runcmd
raise e
subprocess.CalledProcessError: Command '['git', 'reset', '--hard']' returned non-zero exit status 128.

Quoted 72 lines

Error message is as below.
NOTE: Starting bitbake server...
ERROR: Unable to read /home/oelayer/layerindex/git___lxgit_wrs_com_layers_meta-arm/meta-arm/recipes-bsp/trusted -firmware-m/trusted-firmware-m_1.6.0.bb: Traceback (most recent call
last):
File
"/home/oelayer/layerindex/git___lxgit_wrs_com_bitbake/lib/bb/command.py", line 90, in runCommand
result = command_method(self, commandline)
File
"/home/oelayer/layerindex/git___lxgit_wrs_com_bitbake/lib/bb/command.py", line 567, in parseRecipeFile
envdata = bb.cache.parse_recipe(config_data, fn, appendfiles, mc)['']
AttributeError: module 'bb.cache' has no attribute 'parse_recipe'

Regards,
Qi

-----Original Message-----
From: Richard Purdie <[email protected]>
Sent: Thursday, November 24, 2022 8:02 PM
To: Chen, Qi <[email protected]>; [email protected]
Subject: Re: [bitbake-devel] [PATCH] command.py: cleanup bb.cache.parse_recipe

Quoted 13 lines · Chen Qi

The parse_recipe has been removed from bb.cache, replace it with the databuilder._parse_recipe.

---
 lib/bb/command.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/bb/command.py index renamed from lib/bb/command.py +0 -0

fa9fd054c..c9571f63d 100644

lib/bb/command.py class CommandsSync: · line 565

Quoted 2 lines
lib/bb/command.py +0 -0
@@ -564,7 +564,7 @@ class CommandsSync:
         if config_data:
             # We have to use a different function here if we're passing

in a datastore

lib/bb/command.py class CommandsSync: · line 567

Quoted 2 lines
lib/bb/command.py +0 -1
             # NOTE: we took a copy above, so we don't do it here again
-            envdata = bb.cache.parse_recipe(config_data, fn,

appendfiles, mc)['']

Quoted 7 lines
lib/bb/command.py +3 -0
+            envdata =
+ command.cooker.databuilder._parse_recipe(config_data, fn,
+ appendfiles, mc)['']
         else:
             # Use the standard path
             envdata = command.cooker.databuilder.parseRecipe(fn,

appendfiles)

How did you run into that? I'm a bit worried we're missing some test coverage here?

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#14107):
https://lists.openembedded.org/g/bitbake-devel/message/14107
Mute This Topic: https://lists.openembedded.org/mt/95234638/3617156
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#5 lore raw

Hi Richard and Martin,

I've setup a pure oe-core layerindex (No Wind River specific settings) locally, it works well.

For Richard's question, where is the config_data from, it is from layerindex-web/layerindex/update_layer.py:

envdata = tinfoil.parse_recipe_file(fn, appends=False, config_data=data)

The error will be gone if remove "config_date=data", so we can confirm that it is from the line.

For Martin, maybe a git process had crashed so that index.lock was not cleaned up, you can try to remove the lock file (Or ask layerindex maintainer to do that):

$ rm -f /opt/workdir/git___git_openembedded_org_bitbake/.git/index.lock

// Robert

Quoted 117 lines · Martin Jansa

On Thu, Nov 24, 2022 at 1:17 PM Chen Qi <[email protected] <mailto:[email protected]>> wrote:

Hi Richard,

I met the error on our layerindex.

"our" is
https://layers.openembedded.org <https://urldefense.com/v3/__https://layers.openembedded.org__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYcmIy7OIU$>

or some other instance in windriver?

I'm asking because parsing on https://layers.openembedded.org <https://urldefense.com/v3/__https://layers.openembedded.org__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYcmIy7OIU$> seems to be broken for a while. I've checked randomly selected 3 layers and all show the same issue as "Updates" tab in:
https://layers.openembedded.org/layerindex/branch/master/layer/openembedded-core/ <https://urldefense.com/v3/__https://layers.openembedded.org/layerindex/branch/master/layer/openembedded-core/__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYcM7oxI8A$>

ERROR: fatal: Unable to create '/opt/workdir/git___git_openembedded_org_bitbake/.git/index.lock': File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
ERROR: Traceback (most recent call last):
  File "update_layer.py", line 384, in main
    (tinfoil, tempdir) = recipeparse.init_parser(settings, branch, bitbakepath,
nocheckout=options.nocheckout, logger=logger)
  File "/opt/layerindex/layerindex/recipeparse.py", line 36, in init_parser utils.checkout_repo(bitbakepath, bitbake_ref, logger=logger)
  File "/opt/layerindex/layerindex/utils.py", line 282, in checkout_repo runcmd(['git', 'reset', '--hard'], repodir, logger=logger)
  File "/opt/layerindex/layerindex/utils.py", line 369, in runcmd
    raise e
subprocess.CalledProcessError: Command '['git', 'reset', '--hard']' returned non-zero exit status 128.

Error message is as below.
NOTE: Starting bitbake server...
ERROR: Unable to read
/home/oelayer/layerindex/git___lxgit_wrs_com_layers_meta-arm/meta-arm/recipes-bsp/trusted
-firmware-m/trusted-firmware-m_1.6.0.bb
<https://urldefense.com/v3/__http://trusted-firmware-m_1.6.0.bb__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYcR8tYvdQ$>: Traceback (most recent call last):
  File
"/home/oelayer/layerindex/git___lxgit_wrs_com_bitbake/lib/bb/command.py",
line 90, in runCommand
    result = command_method(self, commandline)
  File
"/home/oelayer/layerindex/git___lxgit_wrs_com_bitbake/lib/bb/command.py",
line 567, in parseRecipeFile
    envdata = bb.cache.parse_recipe(config_data, fn, appendfiles, mc)['']
AttributeError: module 'bb.cache' has no attribute 'parse_recipe'

Regards,
Qi

-----Original Message-----
From: Richard Purdie <[email protected]
<mailto:[email protected]>>
Sent: Thursday, November 24, 2022 8:02 PM
To: Chen, Qi <[email protected] <mailto:[email protected]>>;
[email protected]
<mailto:[email protected]>
Subject: Re: [bitbake-devel] [PATCH] command.py
<https://urldefense.com/v3/__http://command.py__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYcJ_BcC5o$>: cleanup bb.cache.parse_recipe

On Thu, 2022-11-24 at 01:10 -0800, Chen Qi wrote:
> The parse_recipe has been removed from bb.cache, replace it with the
> databuilder._parse_recipe.
>
> Signed-off-by: Chen Qi <[email protected] <mailto:[email protected]>>
> ---
>  lib/bb/command.py
<https://urldefense.com/v3/__http://command.py__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYcJ_BcC5o$> | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/lib/bb/command.py
<https://urldefense.com/v3/__http://command.py__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYcJ_BcC5o$> b/lib/bb/command.py <https://urldefense.com/v3/__http://command.py__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYcJ_BcC5o$> index
> fa9fd054c..c9571f63d 100644
> --- a/lib/bb/command.py
<https://urldefense.com/v3/__http://command.py__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYcJ_BcC5o$>
> +++ b/lib/bb/command.py
<https://urldefense.com/v3/__http://command.py__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYcJ_BcC5o$>
> @@ -564,7 +564,7 @@ class CommandsSync:
>          if config_data:
>              # We have to use a different function here if we're passing
in a datastore
>              # NOTE: we took a copy above, so we don't do it here again
> -            envdata = bb.cache.parse_recipe(config_data, fn,
appendfiles, mc)['']
> +            envdata =
> + command.cooker.databuilder._parse_recipe(config_data, fn,
> + appendfiles, mc)['']
>          else:
>              # Use the standard path
>              envdata = command.cooker.databuilder.parseRecipe(fn,
> appendfiles)

How did you run into that? I'm a bit worried we're missing some test coverage here?

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#14107):
https://lists.openembedded.org/g/bitbake-devel/message/14107
<https://urldefense.com/v3/__https://lists.openembedded.org/g/bitbake-devel/message/14107__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYcFi8osuw$>
Mute This Topic: https://lists.openembedded.org/mt/95234638/3617156
<https://urldefense.com/v3/__https://lists.openembedded.org/mt/95234638/3617156__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYcb2VAskM$>
Group Owner: [email protected]
<mailto:bitbake-devel%[email protected]>
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub
<https://urldefense.com/v3/__https://lists.openembedded.org/g/bitbake-devel/unsub__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYc339AjuA$> [[email protected] <mailto:[email protected]>]
-=-=-=-=-=-=-=-=-=-=-=-

#6 lore raw
Quoted 5 lines · Robert Yang

Hi Richard and Martin,

I've setup a pure oe-core layerindex (No Wind River specific settings) locally, it works well.

Quoted 8 lines

For Richard's question, where is the config_data from, it is from layerindex-web/layerindex/update_layer.py:

envdata = tinfoil.parse_recipe_file(fn, appends=False, config_data=data)

The error will be gone if remove "config_date=data", so we can confirm that it is from the line.

Thanks, that helps. I've sent a patch to the tinfoil selftests in OE- Core so we will test this piece of API in future and try and prevent regressions.

Cheers,

Richard