Skip to contentSkip to conversation

bitbake is corrupting my files during unpacking

DISCUSSION MOHAMMAD RASIM 3 participants 7 messages Jul 21, 2018 – Jul 27, 2018

lore
Share & tools
lore
#0 lore raw

Hi,

I'm new to openembedded and yocto. I'm using the openpli meta layers to build openembedded https://github.com/OpenPLi/openpli-oe-core

Now I'm trying to build a recipe for the kernel but I encountered a strange issue, bitbake is corrupting my source files when unpacking from zip file (http link of zip file) but not when using git to get the source the files are fine and not corrupted, the downloaded zip file itself is not corrupted and If I unzip it from the sources directory(outside bitbake environment) then the resulting files are not corrupted

The zip file I'm downloading https://github.com/PLi-metas/linux-amlogic/archive/amlogic-3.14-nougat.zip

examples of corruption I saw :

1- /drivers/w1/slaves/Kconfig (this file is converted to a broken symlink not to a path but to the content of the file itself, see:

file build/tmp/work-shared/k2pro_s905/kernel-source/drivers/w1/slaves/Kconfig
build/tmp/work-shared/k2pro_s905/kernel-source/drivers/w1/slaves/Kconfig: broken symbolic link to #\012# 1-wire slaves configuration\012#\012\012menu "1-wire Slaves"\012\012config W1_SLAVE_THERM\012\011tristate "Thermal family implementation"\012\011help\012\011  Say Y here if you want to connect 1-wire thermal sensors to your\012\011 wire.\012\012config W1_SLAVE_SMEM\012\011tristate "Simple 64bit memory family implementation"\012\011help\012\011  Say Y here if you want to connect 1-wire\012\011  simple 64bit memory rom(ds2401/ds2411/ds1990*) to your wire.\012\012config W1_SLAVE_DS2408\012\011tristate "8-Channel Addressable Switch (IO Expander) 0x29 family support (DS2408)"\012\011help\012\011 Say Y here if you want to use a 1-wire\012\011  DS2408 8-Channel Addressable Switch device support\012\012config W1_SLAVE_DS2408_READBACK\012\011bool "Read-back values written to DS2408's output register"\012\011depends on W1_SLAVE_DS2408\012\011default y\012\011help\012\011  Enabling this will cause the driver to read back the values written\012\011  to the chip's output register in order to detect errors.\012\012\011  This is slower but useful when debugging chips and/or busses.\012\012config W1_SLAVE_DS2413\012\011tristate "Dual Channel Addressable Switch 0x3a family support (DS2413)"\012\011help\012\011  Say Y here if you want to use a 1-wire\012\011  DS2413 Dual Channel Addressable Switch device support\012\012config W1_SLAVE_DS2423\012\011tristate "Counter 1-wire device (DS2423)"\012\011select CRC16\012\011help\012\011  If you enable this you can read the counter values available\012\011  in the DS2423 chipset from the w1_slave file under the\012\011  sys file system.\012\012\011  Say Y here if you want to use a 1-wire\012\011  counter family device (DS2423).\012\012config W1_SLAVE_DS2431\012\011tristate "1kb EEPROM family support (DS2431)"\012\011help\012\011  Say Y here if you want to use a 1-wire\012\011  1kb EEPROM family device (DS2431)\012\012config W1_SLAVE_DS2433\012\011tristate "4kb EEPROM family support (DS2433)"\012\011help\012\011  Say Y here if you want to use a 1-wire\012\011  4kb EEPROM family device (DS2433).\012\012config W1_SLAVE_DS2433_CRC\012\011bool "Protect DS2433 data with a CRC16"\012\011depends on W1_SLAVE_DS2433\012\011select CRC16\012\011help\012\011  Say Y here to protect DS2433 data with a CRC16.\012\011  Each block has 30 bytes of data and a two byte CRC16.\012\011  Full block writes are only allowed if the CRC is valid.\012\012config W1_SLAVE_DS2760\012\011tristate "Dallas 2760 battery monitor chip (HP iPAQ & others)"\012\011depends on W1\012\011help\012\011  If you enable this you will have the DS2760 battery monitor\012\011  chip support.\012\012\011  The battery monitor chip is used in many batteries/devices\012\011  as the one who is responsible for charging/discharging/monitoring\012\011  Li+ batteries.\012\012\011  If you are unsure, say N.\012\012config W1_SLAVE_DS2780\012\011tristate "Dallas 2780 battery monitor chip"\012\011depends on W1\012\011help\012\011  If you enable this you will have the DS2780 battery monitor\012\011  chip support.\012\012\011  The battery monitor chip is used in many batteries/devices\012\011  as the one who is responsible for charging/discharging/monitoring\012\011  Li+ batteries.\012\012\011  If you are unsure, say N.\012\012config W1_SLAVE_DS2781\012\011tristate "Dallas 2781 battery monitor chip"\012\011depends on W1\012\011help\012\011  If you enable this you will have the DS2781 battery monitor\012\011  chip support.\012\012\011  The battery monitor chip is used in many batteries/devices\012\011  as the one who is responsible for charging/discharging/monitoring\012\011  Li+ batteries.\012\012\011  If you are unsure, say N.\012\012config W1_SLAVE_DS28E04\012\011tristate "4096-Bit Addressable 1-Wire EEPROM with PIO (DS28E04-100)"\012\011depends on W1\012\011select CRC16\012\011help\012\011  If you enable this you will have the DS28E04-100\012\011  chip support.\012\012\011  Say Y here if you want to use a 1-wire\012\011  4kb EEPROM with PIO family device (DS28E04).\012\012\011  If you are unsure, say N.\012\012config W1_SLAVE_BQ27000\012\011tristate "BQ27000 slave support"\012\011depends on W1\012\011help\012\011  Say Y here if you want to use a hdq\012\011  bq27000 slave support.\012\012endmenu\012

2- /linux-amlogic-amlogic-3.14-nougat/security/keys/request_key_auth.c (this file is not extracted)

#1 lore raw

well apparently this is caused by the unzip binary that is compiled by the openembedded unzip recipe.
If I unzip the same zip file with the unzip binary that is shipped with my system(manjaro) then the files are not corrupted but when I use the unzip binary compiled by the openembedded recipe I get error and file corruptions.
These are some of the errors I get when unzipping with the openembedded unzip:
lchmod (file attributes) error: Function not implemented
linux-amlogic-amlogic-3.14-nougat/security/keys/request_key_auth.c -> /* Request key authorisation token key definition.^J *^J * Copyright (C) 2005 Red Hat, Inc. All Rights Reserved.^J * Written by David Howells ([email protected])^J *^J * This program is free software; you can redistribute it and/or^J * modify it under the terms of the GNU General Public License^J * as published by the Free Software Foundation; either version^J * 2 of the License, or (at your option) any later version.^J *^J * See Documentation/security/keys-request-key.txt^J */^J^J#include <linux/module.h>^J#include <linux/sched.h>^J#include <linux/err.h>^J#include <linux/seq_file.h>^J#include <linux/slab.h>^J#include <asm/uaccess.h>^J#include "internal.h"^J#include <keys/user-type.h>^J^Jstatic int request_key_auth_instantiate(struct key *,^J^I^I^I^I^Istruct key_preparsed_payload *);^Jstatic void request_key_auth_describe(const struct key *, struct seq_file *);^Jstatic void request_key_auth_revoke(struct key *);^Jstatic void request_key_auth_destroy(struct key *);^Jstatic long request_key_auth_read(const struct key *, char __user *, size_t);^J^J/*^J * The request-key authorisation key type definition.^J */^Jstruct key_type key_type_request_key_auth = {^J^I.name^I^I= ".request_key_auth",^J^I.def_datalen^I= sizeof(struct request_key_auth),^J^I.instantiate^I= request_key_auth_instantiate,^J^I.describe^I= request_key_auth_describe,^J^I.revoke^I^I= request_key_auth_revoke,^J^I.destroy^I= request_key_auth_destroy,^J^I.read^I^I= request_key_auth_read,^J};^J^J/*^J * Instantiate a request-key authorisation key.^J */^Jstatic int request_key_auth_instantiate(struct key *key,^J^I^I^I^I^Istruct key_preparsed_payload *prep)^J{^J^Ikey->payload.data = (struct request_key_auth *)prep->data;^J^Ireturn 0;^J}^J^J/*^J * Describe an authorisation token.^J */^Jstatic void request_key_auth_describe(const struct key *key,^J^I^I^I^I struct seq_file *m)^J{^J^Istruct request_key_auth *rka = key->payload.data;^J^J^Iseq_puts(m, "key:");^J^Iseq_puts(m, key->description);^J^Iif (key_is_instantiated(key))^J^I^Iseq_printf(m, " pid:%d ci:%zu", rka->pid, rka->callout_len);^J}^J^J/*^J * Read the callout_info data (retrieves the callout information).^J * - the key's semaphore is read-locked^J */^Jstatic long request_key_auth_read(const struct key *key,^J^I^I^I^I  char __user *buffer, size_t buflen)^J{^J^Istruct request_key_auth *rka = key->payload.data;^J^Isize_t datalen;^J^Ilong ret;^J^J^Idatalen = rka->callout_len;^J^Iret = datalen;^J^J^I/* we can return the data as is */^J^Iif (buffer && buflen > 0) {^J^I^Iif (buflen > datalen)^J^I^I^Ibuflen = datalen;^J^J^I^Iif (copy_to_user(buffer, rka->callout_info, buflen) != 0)^J^I^I^Iret = -EFAULT;^J^I}^J^J^Ireturn ret;^J}^J^J/*^J * Handle revocation of an authorisation token key.^J *^J * Called with the key sem write-locked.^J */^Jstatic void request_key_auth_revoke(struct key *key)^J{^J^Istruct request_key_auth *rka = key->payload.data;^J^J^Ikenter("{%d}", key->serial);^J^J^Iif (rka->cred) {^J^I^Iput_cred(rka->cred);^J^I^Irka->cred = NULL;^J^I}^J}^J^J/*^J * Destroy an instantiation authorisation token key.^J */^Jstatic void request_key_auth_destroy(struct key *key)^J{^J^Istruct request_key_auth *rka = key->payload.data;^J^J^Ikenter("{%d}", key->serial);^J^J^Iif (rka->cred) {^J^I^Iput_cred(rka->cred);^J^I^Irka->cred = NULL;^J^I}^J^J^Ikey_put(rka->target_key);^J^Ikey_put(rka->dest_keyring);^J^Ikfree(rka->callout_info);^J^Ikfree(rka);^J}^J^J/*^J * Create an authorisation token for /sbin/request-key or whoever to gain^J * access to the caller's security data.^J */^Jstruct key *request_key_auth_new(struct key *target, const void *callout_info,^J^I^I^I^I size_t callout_len, struct key *dest_keyring)^J{^J^Istruct request_key_auth *rka, *irka;^J^Iconst struct cred *cred = current->cred;^J^Istruct key *authkey = NULL;^J^Ichar desc[20];^J^Iint ret;^J^J^Ikenter("%d,", target->serial);^J^J^I/* allocate a auth record */^J^Irka = kmalloc(sizeof(*rka), GFP_KERNEL);^J^Iif (!rka) {^J^I^Ikleave(" = -ENOMEM");^J^I^Ireturn ERR_PTR(-ENOMEM);^J^I}^J^Irka->callout_info = kmalloc(callout_len, GFP_KERNEL);^J^Iif (!rka->callout_info) {^J^I^Ikleave(" = -ENOMEM");^J^I^Ikfree(rka);^J^I^Ireturn ERR_PTR(-ENOMEM);^J^I}^J^J^I/* see if the calling process is already servicing the key request of^J^I * another process */^J^Iif (cred->request_key_auth) {^J^I^I/* it is - use that instantiation context here too */^J^I^Idown_read(&cred->request_key_auth->sem);^J^J^I^I/* if the auth key has been revoked, then the key we're^J^I^I * servicing is already instantiated */^J^I^Iif (test_bit(KEY_FLAG_REVOKED, &cred->request_key_auth->flags))^J^I^I^Igoto auth_key_revoked;^J^J^I^Iirka = cred->request_key_auth->payload.data;^J^I^Irka->cred = get_cred(irka->cred);^J^I^Irka->pid = irka->pid;^J^J^I^Iup_read(&cred->request_key_auth->sem);^J^I}^J^Ielse {^J^I^I/* it isn't - use this process as the context */^J^I^Irka->cred = get_cred(cred);^J^I^Irka->pid = current->pid;^J^I}^J^J^Irka->target_key = key_get(target);^J^Irka->dest_keyring = key_get(dest_keyring);^J^Imemcpy(rka->callout_info, callout_info, callout_len);^J^Irka->callout_len = callout_len;^J^J^I/* allocate the auth key */^J^Isprintf(desc, "%x", target->serial);^J^J^Iauthkey = key_alloc(&key_type_request_key_auth, desc,^J^I^I^I cred->fsuid, cred->fsgid, cred,^J^I^I^I    KEY_POS_VIEW | KEY_POS_READ | KEY_POS_SEARCH |^J^I^I^I    KEY_USR_VIEW, KEY_ALLOC_NOT_IN_QUOTA);^J^Iif (IS_ERR(authkey)) {^J^I^Iret = PTR_ERR(authkey);^J^I^Igoto error_alloc;^J^I}^J^J^I/* construct the auth key */^J^Iret = key_instantiate_and_link(authkey, rka, 0, NULL, NULL);^J^Iif (ret < 0)^J^I^Igoto error_inst;^J^J^Ikleave(" = {%d,%d}", authkey->serial, atomic_read(&authkey->usage));^J^Ireturn authkey;^J^Jauth_key_revoked:^J^Iup_read(&cred->request_key_auth->sem);^J^Ikfree(rka->callout_info);^J^Ikfree(rka);^J^Ikleave("= -EKEYREVOKED");^J^Ireturn ERR_PTR(-EKEYREVOKED);^J^Jerror_inst:^J^Ikey_revoke(authkey);^J^Ikey_put(authkey);^Jerror_alloc:^J^Ikey_put(rka->target_key);^J^Ikey_put(rka->dest_keyring);^J^Ikfree(rka->callout_info);^J^Ikfree(rka);^J^Ikleave("= %d", ret);^J^Ireturn ERR_PTR(ret);^J}^J^J/*^J * Search the current process's keyrings for the authorisation key for^J * instantiation of a key.^J */^Jstruct key *key_get_instantiation_authkey(key_serial_t target_id)^J{^J^Ichar description[16];^J^Istruct keyring_search_context ctx = {^J^I^I.index_key.type^I^I= &key_type_request_key_auth,^J^I^I.index_key.description^I= description,^J^I^I.cred^I^I^I= current_cred(),^J^I^I.match^I^I^I= user_match,^J^I^I.match_data^I^I= description,^J^I^I.flags^I^I^I= KEYRING_SEARCH_LOOKUP_DIRECT,^J^I};^J^Istruct key *authkey;^J^Ikey_ref_t authkey_ref;^J^J^Isprintf(description, "%x", target_id);^J^J^Iauthkey_ref = search_process_keyrings(&ctx);^J^J^Iif (IS_ERR(authkey_ref)) {^J^I^Iauthkey = ERR_CAST(authkey_ref);^J^I^Iif (authkey == ERR_PTR(-EAGAIN))^J^I^I^Iauthkey = ERR_PTR(-ENOKEY);^J^I^Igoto error;^J^I}^J^J^Iauthkey = key_ref_to_ptr(authkey_ref);^J^Iif (test_bit(KEY_FLAG_REVOKED, &authkey->flags)) {^J^I^Ikey_put(authkey);^J^I^Iauthkey = ERR_PTR(-EKEYREVOKED);^J^I}^J^Jerror:^J^Ireturn authkey;^J}^J symlink error: File name too long
lchmod (file attributes) error: Function not implemented
linux-amlogic-amlogic-3.14-nougat/tools/gator/daemon/k/perf_event.h -> perf_event.3.12.h
lchmod (file attributes) error: Function not implemented

This error happens to multiple files where the file is symlinked to the content of the file and not to a path ! Where should I report this bug ? openembedded ? unzip upstream ?

#2 lore raw
Quoted 10 lines · MOHAMMAD RASIM

well apparently this is caused by the unzip binary that is compiled by the openembedded unzip recipe.
If I unzip the same zip file with the unzip binary that is shipped with my system(manjaro) then the files are not corrupted but when I use the unzip binary compiled by the openembedded recipe I get error and file corruptions. These are some of the errors I get when unzipping with the openembedded unzip:
...
symlink error: File name too long

Quoted 3 lines

This error happens to multiple files where the file is symlinked to the content of the file and not to a path !
Where should I report this bug ? openembedded ? unzip upstream ?

If you google the "symlink error" you should find multiple hits, which all indicate a known bug in upstream unzip 6.0.

So, it should be reported upstream. However, given that the last unzip release is now over 9 years old and there doesn't appear to be a public upstream development branch, the chances of getting a timely response don't look too good.

We should probably add the fix to the unzip recipe in oe-core:

https://bugzilla.redhat.com/show_bug.cgi?id=972427

#3 lore raw

Yes, please do. Mohammad, as you can replicate this on demand, would you be able to fix the unzip recipe with the patch in that bug and send a patch?

Ross

Quoted 32 lines · Andre McCurdy
Quoted 10 lines · MOHAMMAD RASIM

well apparently this is caused by the unzip binary that is compiled by the openembedded unzip recipe.
If I unzip the same zip file with the unzip binary that is shipped with my system(manjaro) then the files are not corrupted but when I use the unzip binary compiled by the openembedded recipe I get error and file corruptions. These are some of the errors I get when unzipping with the openembedded unzip:
...
symlink error: File name too long

Quoted 3 lines

This error happens to multiple files where the file is symlinked to the content of the file and not to a path !
Where should I report this bug ? openembedded ? unzip upstream ?

If you google the "symlink error" you should find multiple hits, which all indicate a known bug in upstream unzip 6.0.

So, it should be reported upstream. However, given that the last unzip release is now over 9 years old and there doesn't appear to be a public upstream development branch, the chances of getting a timely response don't look too good.

We should probably add the fix to the unzip recipe in oe-core:

https://bugzilla.redhat.com/show_bug.cgi?id=972427
--
_______________________________________________
yocto mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/yocto

#4 lore raw

Yes I made a pull request to the master branch https://github.com/openembedded/openembedded-core/pull/34

note that this issue can be replicated by unzipping any linux-kernel (I think), I tested that it exist in the zip file of torvalds tree https://github.com/torvalds/linux/archive/master.zip

so it's easy to replicate and test

Also, to anyone how face the same issue and find this message, I solved the issue without patching unzip by using tar.gz file instead of unzip file (thanks to github for providing that) so instead of https://github.com/torvalds/linux/archive/master.zip use https://github.com/torvalds/linux/archive/master.tar.gz

Thanks

Quoted 38 lines · Ross

Yes, please do. Mohammad, as you can replicate this on demand, would you be able to fix the unzip recipe with the patch in that bug and send a patch?

Ross

Quoted 31 lines · Andre McCurdy
Quoted 10 lines · MOHAMMAD RASIM

well apparently this is caused by the unzip binary that is compiled by the openembedded unzip recipe.
If I unzip the same zip file with the unzip binary that is shipped with my system(manjaro) then the files are not corrupted but when I use the unzip binary compiled by the openembedded recipe I get error and file corruptions. These are some of the errors I get when unzipping with the openembedded unzip:
...
symlink error: File name too long

Quoted 3 lines

This error happens to multiple files where the file is symlinked to the content of the file and not to a path !
Where should I report this bug ? openembedded ? unzip upstream ?

If you google the "symlink error" you should find multiple hits, which all indicate a known bug in upstream unzip 6.0.

So, it should be reported upstream. However, given that the last unzip release is now over 9 years old and there doesn't appear to be a public upstream development branch, the chances of getting a timely response don't look too good.

We should probably add the fix to the unzip recipe in oe-core:

   https://bugzilla.redhat.com/show_bug.cgi?id=972427

--
_______________________________________________
yocto mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/yocto

#5 lore raw

We don't take patches via pull requests (that repo is a mirror for convenience and should have PRs disabled), but patches on the openembedded-core mailing list.

https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines has details as to the required patch formatting. Summary:

Commit message should be of the form 'recipe: summary'. The patch you're adding also needs a summary explaining what it does, an Upstream-Status value (Pending I guess, as upstream is dead we can't submit it) and your Signed-off-by.

Ross

Quoted 69 lines · MOHAMMAD RASIM

Yes I made a pull request to the master branch
https://github.com/openembedded/openembedded-core/pull/34

note that this issue can be replicated by unzipping any linux-kernel (I think), I tested that it exist in the zip file of torvalds tree
https://github.com/torvalds/linux/archive/master.zip

so it's easy to replicate and test

Also, to anyone how face the same issue and find this message, I solved the issue without patching unzip by using tar.gz file instead of unzip file (thanks to github for providing that) so instead of
https://github.com/torvalds/linux/archive/master.zip use
https://github.com/torvalds/linux/archive/master.tar.gz

Thanks

Quoted 46 lines · Ross

Yes, please do. Mohammad, as you can replicate this on demand, would you be able to fix the unzip recipe with the patch in that bug and send a patch?

Ross

Quoted 38 lines · Andre McCurdy
Quoted 15 lines · MOHAMMAD RASIM

well apparently this is caused by the unzip binary that is compiled by the
openembedded unzip recipe.
If I unzip the same zip file with the unzip binary that is shipped with my
system(manjaro) then the files are not corrupted but when I use the unzip
binary compiled by the openembedded recipe I get error and file corruptions.
These are some of the errors I get when unzipping with the openembedded unzip:
...
symlink error: File name too long

Quoted 3 lines

This error happens to multiple files where the file is symlinked to the content of the file and not to a path !
Where should I report this bug ? openembedded ? unzip upstream ?

If you google the "symlink error" you should find multiple hits, which all indicate a known bug in upstream unzip 6.0.

So, it should be reported upstream. However, given that the last unzip release is now over 9 years old and there doesn't appear to be a public upstream development branch, the chances of getting a timely response don't look too good.

We should probably add the fix to the unzip recipe in oe-core:

   https://bugzilla.redhat.com/show_bug.cgi?id=972427

--
_______________________________________________
yocto mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/yocto

#6 lore raw

Can you send the patch yourself since you already know how to do it ?

Quoted 79 lines · Ross

We don't take patches via pull requests (that repo is a mirror for convenience and should have PRs disabled), but patches on the openembedded-core mailing list.

https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines has details as to the required patch formatting. Summary:

Commit message should be of the form 'recipe: summary'. The patch you're adding also needs a summary explaining what it does, an Upstream-Status value (Pending I guess, as upstream is dead we can't submit it) and your Signed-off-by.

Ross

Quoted 64 lines · MOHAMMAD RASIM

Yes I made a pull request to the master branch
https://github.com/openembedded/openembedded-core/pull/34

note that this issue can be replicated by unzipping any linux-kernel (I think), I tested that it exist in the zip file of torvalds tree
https://github.com/torvalds/linux/archive/master.zip

so it's easy to replicate and test

Also, to anyone how face the same issue and find this message, I solved the issue without patching unzip by using tar.gz file instead of unzip file (thanks to github for providing that) so instead of
https://github.com/torvalds/linux/archive/master.zip use
https://github.com/torvalds/linux/archive/master.tar.gz

Thanks

Quoted 42 lines · Ross

Yes, please do. Mohammad, as you can replicate this on demand, would you be able to fix the unzip recipe with the patch in that bug and send a patch?

Ross

Quoted 35 lines · Andre McCurdy
Quoted 14 lines · MOHAMMAD RASIM

well apparently this is caused by the unzip binary that is compiled by the
openembedded unzip recipe.
If I unzip the same zip file with the unzip binary that is shipped with my
system(manjaro) then the files are not corrupted but when I use the unzip
binary compiled by the openembedded recipe I get error and file corruptions.
These are some of the errors I get when unzipping with the openembedded unzip:
...
symlink error: File name too long

Quoted 3 lines

This error happens to multiple files where the file is symlinked to the content of the file and not to a path !
Where should I report this bug ? openembedded ? unzip upstream ?

If you google the "symlink error" you should find multiple hits, which all indicate a known bug in upstream unzip 6.0.

So, it should be reported upstream. However, given that the last unzip release is now over 9 years old and there doesn't appear to be a public upstream development branch, the chances of getting a timely response don't look too good.

We should probably add the fix to the unzip recipe in oe-core:

    https://bugzilla.redhat.com/show_bug.cgi?id=972427

--
_______________________________________________
yocto mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/yocto