Source Archiver Class

This is a progression list of what the source archives should include:

1 - Original Upstream Archive & Patches
- 2 archives (tarballs)

2 - Original Source code & Patches
- could include additional code

- post unpack
- 1 archive

3 - Original Source code & Patches & temp (scripts & logs)
- Could possibly include the .bb and .inc files
- 2 archives (from #2 & temp tarball)

4 - Patched source code
- original source code could be removed
- post do_patch

5 - Patched source code & temp
- 2 archives (from #4 & temp tarball)

6 - Configured Source
- post do_configure

7 - SRPM format of Original Source & Patches
- rpm will apply the patches
- internally contains #2 above

8 - SRPM various of #3 above

...
100 - Buildable SRPM
- can actually build, this is way future!

(Patches = patch files & series list)

Each of these build on the previous in some way, the key being that we generate a tarball for the source from the existing state of the WORKDIR, a challenge maybe to create the source snapshot after the build has already occurred.

The SPDX License info could also be included in any of these

After talking with Richard, we think we have a novel approach to make this work. It would entail using the postfuncs feature similar to the way that Shared State does it's work. The existing copyleft_compliance class functionality can be folded into this as a filter. Additional flags could be passed from individual classes to a core set of methods in an archiver class defining the type of data (source, patches, temp, env, recipe info, ...) and format (tar, sprm, ...)

I noted that recipe type code might be better suited as generic code, as I believe there are other places that could benefit from that code.

The sourcepkg class seems to be a basic archiver and differ that includes the metadata/environment (as dumpdata), this could be replaced by the new approach. While the src_distribute class copies the downloaded archive and then creates a link, into LICENSE directories along with the patches. The src_distribute by default seems to move files and create links (incorrectly btw!). This work can be done by the archiver class.

The Nugget: Create a new core "archiver" class that implements a general functions that can archive the original tarball or workdir at various states along task list with additional metadata (recipe info, temp dir, environment). This class would be inherited by a set of classes that use the postfunc (similar to sstate) that setup what level of archive is needed (based on the list above).

Thoughts, Comments?

Thanks

Hi Saul,

I have some issues when I writing design document. The following description is my understanding. I take package "zlib" for example .

Quoted 2 lines

Quoted 4 lines

This is a progression list of what the source archives should include:

1 - Original Upstream Archive & Patches
- 2 archives (tarballs)

$ls zlib-orgsource
zlib-1.2.5.tar.bz2(after do_fetch)
zlib-patches.tar.bz2( the patches come from
meta/recipes-core/zlib/zlib-1.2.5)

Quoted 2 lines

2 - Original Source code & Patches
- could include additional code

Please give me more detailed information

Quoted 2 lines

- post unpack
- 1 archive

$ls zlib-orgsource
zlib-1.2.5.tar.bz2(after do_unpack, patches are not included in this package.)

No patch in this directory.

Quoted 4 lines

3 - Original Source code & Patches & temp (scripts & logs)
- Could possibly include the .bb and .inc files
- 2 archives (from #2 & temp tarball)

$ls zlib-orgsource
zlib-1.2.5.tar.bz2(after do_unpack, patches are not included in this package.)
zlib-patches.tar.bz2( the patches come from meta/recipes-core/zlib/zlib-1.2.5)

zlib-scripts-logs.tar.bz2(include zlib_1.2.5.bb,zlib.inc and tmp/log_*)

So there are 3 tarballs in this directory.

Quoted 3 lines

4 - Patched source code
- original source code could be removed
- post do_patch

$ls zlib-source
zlib-1.2.5.tar.bz2(after do_patch)

Quoted 2 lines

5 - Patched source code & temp
- 2 archives (from #4 & temp tarball)

$ls zlib-source
zlib-1.2.5.tar.bz2(after do_patch)
zlib-logs.tar.bz2(include tmp/log_*)

Quoted 2 lines

6 - Configured Source
- post do_configure

$ls zlib-source
zlib-1.2.5.tar.bz2(after do_configure)

Quoted 3 lines

7 - SRPM format of Original Source & Patches
- rpm will apply the patches
- internally contains #2 above

$ls zlib-srpm
zlib-1.2.5.src.rpm

zlib-1.2.5.src.rpm includes zlib-1.2.5.tar.bz2(after do_unpack, patches are not included in this package.) and zlib-patches.tar.bz2

Quoted 1 line

8 - SRPM various of #3 above

$ls zlib-srpm
zlib-1.2.5.src.rpm

zlib-1.2.5.src.rpm includes zlib-1.2.5.tar.bz2(after do_unpack, patches are not included in this package.) ,zlib-patches.tar.bz2 and zlib-scripts-logs.tar.bz2

Do you have any suggestion about the above description?

Thanks
Yan

Quoted 45 lines

...
100 - Buildable SRPM
- can actually build, this is way future!

(Patches = patch files & series list)

Each of these build on the previous in some way, the key being that we generate a tarball for the source from the existing state of the WORKDIR, a challenge maybe to create the source snapshot after the build has already occurred.

The SPDX License info could also be included in any of these

After talking with Richard, we think we have a novel approach to make this work. It would entail using the postfuncs feature similar to the way that Shared State does it's work. The existing copyleft_compliance class functionality can be folded into this as a filter. Additional flags could be passed from individual classes to a core set of methods in an archiver class defining the type of data (source, patches, temp, env, recipe info, ...) and format (tar, sprm, ...)

I noted that recipe type code might be better suited as generic code, as I believe there are other places that could benefit from that code.

The sourcepkg class seems to be a basic archiver and differ that includes the metadata/environment (as dumpdata), this could be replaced by the new approach. While the src_distribute class copies the downloaded archive and then creates a link, into LICENSE directories along with the patches. The src_distribute by default seems to move files and create links (incorrectly btw!). This work can be done by the archiver class.

The Nugget: Create a new core "archiver" class that implements a general functions that can archive the original tarball or workdir at various states along task list with additional metadata (recipe info, temp dir, environment). This class would be inherited by a set of classes that use the postfunc (similar to sstate) that setup what level of archive is needed (based on the list above).

Thoughts, Comments?

Thanks

Quoted 6 lines · Xiaofeng Yan

Hi Saul,

I have some issues when I writing design document. The following description is my understanding. I take package "zlib" for example .

Quoted 2 lines

Quoted 8 lines

Quoted 4 lines

This is a progression list of what the source archives should include:

1 - Original Upstream Archive & Patches
- 2 archives (tarballs)

$ls zlib-orgsource
zlib-1.2.5.tar.bz2(after do_fetch)
zlib-patches.tar.bz2( the patches come from
meta/recipes-core/zlib/zlib-1.2.5)

Almost correct, I believe that the work Bruce and Chris have done recently will allow you to get the patches.

I am not sure we want to just tar up the meta/recipes-core/zlib/lib-1.2.5 directory since it might contain other file or patches that we don't actaully use. We need to get the patches that are actually applied.

Quoted 3 lines

Quoted 2 lines

2 - Original Source code & Patches
- could include additional code

Please give me more detailed information

One tarball with the patches extracted in the patches directory with the series file (possibly a script to apply the patches).

Quoted 7 lines

Quoted 2 lines

- post unpack
- 1 archive

$ls zlib-orgsource
zlib-1.2.5.tar.bz2(after do_unpack, patches are not included in this package.)

No patch in this directory.

Need to add the patches via patch mechanism to the source tarball, but without actually applying the patches.

So, this is just one tarball.
zlib-1.2.5-prepatch.tar.bz2

Quoted 9 lines

Quoted 4 lines

3 - Original Source code & Patches & temp (scripts & logs)
- Could possibly include the .bb and .inc files
- 2 archives (from #2 & temp tarball)

$ls zlib-orgsource
zlib-1.2.5.tar.bz2(after do_unpack, patches are not included in this package.)
zlib-patches.tar.bz2( the patches come from meta/recipes-core/zlib/zlib-1.2.5)

Again see above, we need to get only the patches that will actually be applied.

Quoted 3 lines

zlib-scripts-logs.tar.bz2(include zlib_1.2.5.bb,zlib.inc and tmp/log_*)

So there are 3 tarballs in this directory.

Yes

Quoted 5 lines

Quoted 3 lines

4 - Patched source code
- original source code could be removed
- post do_patch

$ls zlib-source
zlib-1.2.5.tar.bz2(after do_patch)

Correct
Maybe better zlib-1.2.5-patched.tar.bz2

Quoted 5 lines

Quoted 2 lines

5 - Patched source code & temp
- 2 archives (from #4 & temp tarball)

$ls zlib-source
zlib-1.2.5.tar.bz2(after do_patch)
zlib-logs.tar.bz2(include tmp/log_*)

Correct
Same as #4 change the tarball name

Quoted 4 lines

Quoted 2 lines

6 - Configured Source
- post do_configure

$ls zlib-source
zlib-1.2.5.tar.bz2(after do_configure)

zlib-1.2.5-configured.tar.bz2

Quoted 9 lines

Quoted 3 lines

7 - SRPM format of Original Source & Patches
- rpm will apply the patches
- internally contains #2 above

$ls zlib-srpm
zlib-1.2.5.src.rpm

zlib-1.2.5.src.rpm includes zlib-1.2.5.tar.bz2(after do_unpack, patches are not included in this package.) and zlib-patches.tar.bz2

Quoted 1 line

8 - SRPM various of #3 above

Quoted 9 lines

$ls zlib-srpm
zlib-1.2.5.src.rpm

zlib-1.2.5.src.rpm includes zlib-1.2.5.tar.bz2(after do_unpack, patches are not included in this package.) ,zlib-patches.tar.bz2 and zlib-scripts-logs.tar.bz2

Do you have any suggestion about the above description?

Just the naming and where the patches come from we need to provide the mechanism (series file) for how to apply the patches also.

Sau!

Quoted 51 lines

Thanks
Yan

Quoted 45 lines

...
100 - Buildable SRPM
- can actually build, this is way future!

(Patches = patch files & series list)

Each of these build on the previous in some way, the key being that we generate a tarball for the source from the existing state of the WORKDIR, a challenge maybe to create the source snapshot after the build has already occurred.

The SPDX License info could also be included in any of these

After talking with Richard, we think we have a novel approach to make this work. It would entail using the postfuncs feature similar to the way that Shared State does it's work. The existing copyleft_compliance class functionality can be folded into this as a filter. Additional flags could be passed from individual classes to a core set of methods in an archiver class defining the type of data (source, patches, temp, env, recipe info, ...) and format (tar, sprm, ...)

I noted that recipe type code might be better suited as generic code, as I believe there are other places that could benefit from that code.

The sourcepkg class seems to be a basic archiver and differ that includes the metadata/environment (as dumpdata), this could be replaced by the new approach. While the src_distribute class copies the downloaded archive and then creates a link, into LICENSE directories along with the patches. The src_distribute by default seems to move files and create links (incorrectly btw!). This work can be done by the archiver class.

The Nugget: Create a new core "archiver" class that implements a general functions that can archive the original tarball or workdir at various states along task list with additional metadata (recipe info, temp dir, environment). This class would be inherited by a set of classes that use the postfunc (similar to sstate) that setup what level of archive is needed (based on the list above).

Thoughts, Comments?

Thanks

Xiaofeng,

I replied last night to your IRC, but was not sure if you saw it, I am adding it to this email.

(11:05:36 PM) sgw: xiaofeng, you around?
(11:08:22 PM) sgw: zlib is a bad example, since there are no patches!
(11:08:38 PM) sgw: Let's use "at" since it is a simple recipe and has patches
(11:09:27 PM) sgw: if you start with bitbake -c cleanall at, so we have a fresh start.
(11:12:26 PM) sgw: bitbake -c unpack at, will give you the upacked source in at-3.1.12-r7/at-3.1.12/..., this will give you a base to build from
(11:15:12 PM) sgw: "bitbake -c patch at" will then patch the above sources and create the patches directory with the series file. So given that info, you be able to add a do_patch[prefunc] to first tar the unpatched directory and then a do_patch[postfunc] to tar the patches directory into the tar file from the first tar. (11:16:32 PM) sgw: Does this make sense?
(11:17:49 PM) sgw: Something else we will want to consider is creating one tarball of all the gnu source and patches in one large tarball.

Quoted 157 lines · Xiaofeng Yan

Quoted 6 lines

Hi Saul,

I have some issues when I writing design document. The following description is my understanding. I take package "zlib" for example .

Quoted 2 lines

Quoted 8 lines

Quoted 4 lines

This is a progression list of what the source archives should include:

1 - Original Upstream Archive & Patches
- 2 archives (tarballs)

$ls zlib-orgsource
zlib-1.2.5.tar.bz2(after do_fetch)
zlib-patches.tar.bz2( the patches come from
meta/recipes-core/zlib/zlib-1.2.5)

Almost correct, I believe that the work Bruce and Chris have done recently will allow you to get the patches.

I am not sure we want to just tar up the meta/recipes-core/zlib/lib-1.2.5 directory since it might contain other file or patches that we don't actaully use. We need to get the patches that are actually applied.

Quoted 3 lines

Quoted 2 lines

2 - Original Source code & Patches
- could include additional code

Please give me more detailed information

One tarball with the patches extracted in the patches directory with the series file (possibly a script to apply the patches).

Quoted 7 lines

Quoted 2 lines

- post unpack
- 1 archive

$ls zlib-orgsource
zlib-1.2.5.tar.bz2(after do_unpack, patches are not included in this package.)

No patch in this directory.

Need to add the patches via patch mechanism to the source tarball, but without actually applying the patches.

So, this is just one tarball.
zlib-1.2.5-prepatch.tar.bz2

Quoted 9 lines

Quoted 4 lines

3 - Original Source code & Patches & temp (scripts & logs)
- Could possibly include the .bb and .inc files
- 2 archives (from #2 & temp tarball)

$ls zlib-orgsource
zlib-1.2.5.tar.bz2(after do_unpack, patches are not included in this package.)
zlib-patches.tar.bz2( the patches come from meta/recipes-core/zlib/zlib-1.2.5)

Again see above, we need to get only the patches that will actually be applied.

Quoted 3 lines

zlib-scripts-logs.tar.bz2(include zlib_1.2.5.bb,zlib.inc and tmp/log_*)

So there are 3 tarballs in this directory.

Yes

Quoted 5 lines

Quoted 3 lines

4 - Patched source code
- original source code could be removed
- post do_patch

$ls zlib-source
zlib-1.2.5.tar.bz2(after do_patch)

Correct
Maybe better zlib-1.2.5-patched.tar.bz2

Quoted 5 lines

Quoted 2 lines

5 - Patched source code & temp
- 2 archives (from #4 & temp tarball)

$ls zlib-source
zlib-1.2.5.tar.bz2(after do_patch)
zlib-logs.tar.bz2(include tmp/log_*)

Correct
Same as #4 change the tarball name

Quoted 4 lines

Quoted 2 lines

6 - Configured Source
- post do_configure

$ls zlib-source
zlib-1.2.5.tar.bz2(after do_configure)

zlib-1.2.5-configured.tar.bz2

Quoted 9 lines

Quoted 3 lines

7 - SRPM format of Original Source & Patches
- rpm will apply the patches
- internally contains #2 above

$ls zlib-srpm
zlib-1.2.5.src.rpm

zlib-1.2.5.src.rpm includes zlib-1.2.5.tar.bz2(after do_unpack, patches are not included in this package.) and zlib-patches.tar.bz2

Quoted 1 line

8 - SRPM various of #3 above

Quoted 9 lines

$ls zlib-srpm
zlib-1.2.5.src.rpm

zlib-1.2.5.src.rpm includes zlib-1.2.5.tar.bz2(after do_unpack, patches are not included in this package.) ,zlib-patches.tar.bz2 and zlib-scripts-logs.tar.bz2

Do you have any suggestion about the above description?

Just the naming and where the patches come from we need to provide the mechanism (series file) for how to apply the patches also.

Sau!

Quoted 51 lines

Thanks
Yan

Quoted 45 lines

...
100 - Buildable SRPM
- can actually build, this is way future!

(Patches = patch files & series list)

Each of these build on the previous in some way, the key being that we generate a tarball for the source from the existing state of the WORKDIR, a challenge maybe to create the source snapshot after the build has already occurred.

The SPDX License info could also be included in any of these

After talking with Richard, we think we have a novel approach to make this work. It would entail using the postfuncs feature similar to the way that Shared State does it's work. The existing copyleft_compliance class functionality can be folded into this as a filter. Additional flags could be passed from individual classes to a core set of methods in an archiver class defining the type of data (source, patches, temp, env, recipe info, ...) and format (tar, sprm, ...)

I noted that recipe type code might be better suited as generic code, as I believe there are other places that could benefit from that code.

The sourcepkg class seems to be a basic archiver and differ that includes the metadata/environment (as dumpdata), this could be replaced by the new approach. While the src_distribute class copies the downloaded archive and then creates a link, into LICENSE directories along with the patches. The src_distribute by default seems to move files and create links (incorrectly btw!). This work can be done by the archiver class.

The Nugget: Create a new core "archiver" class that implements a general functions that can archive the original tarball or workdir at various states along task list with additional metadata (recipe info, temp dir, environment). This class would be inherited by a set of classes that use the postfunc (similar to sstate) that setup what level of archive is needed (based on the list above).

Thoughts, Comments?

Thanks

Quoted 15 lines · Saul Wold

Xiaofeng,

I replied last night to your IRC, but was not sure if you saw it, I am adding it to this email.

(11:05:36 PM) sgw: xiaofeng, you around?
(11:08:22 PM) sgw: zlib is a bad example, since there are no patches! (11:08:38 PM) sgw: Let's use "at" since it is a simple recipe and has patches
(11:09:27 PM) sgw: if you start with bitbake -c cleanall at, so we have a fresh start.
(11:12:26 PM) sgw: bitbake -c unpack at, will give you the upacked source in at-3.1.12-r7/at-3.1.12/..., this will give you a base to build from

Quoted 10 lines

(11:15:12 PM) sgw: "bitbake -c patch at" will then patch the above sources and create the patches directory with the series file. So given that info, you be able to add a do_patch[prefunc] to first tar the unpatched directory and then a do_patch[postfunc] to tar the patches directory into the tar file from the first tar. (11:16:32 PM) sgw: Does this make sense?
(11:17:49 PM) sgw: Something else we will want to consider is creating one tarball of all the gnu source and patches in one large tarball.

Hi Saul,

I saw the reply in IRC yesterday.
Thanks very much. I will also realize a function to include all of gnu sources and patches in one tarball in archiver.bbclass.

Quoted 166 lines · Saul Wold

Quoted 164 lines · Xiaofeng Yan

Quoted 6 lines

Hi Saul,

I have some issues when I writing design document. The following description is my understanding. I take package "zlib" for example .

Quoted 2 lines

Quoted 8 lines

Quoted 4 lines

This is a progression list of what the source archives should include:

1 - Original Upstream Archive & Patches
- 2 archives (tarballs)

$ls zlib-orgsource
zlib-1.2.5.tar.bz2(after do_fetch)
zlib-patches.tar.bz2( the patches come from
meta/recipes-core/zlib/zlib-1.2.5)

Almost correct, I believe that the work Bruce and Chris have done recently will allow you to get the patches.

I am not sure we want to just tar up the meta/recipes-core/zlib/lib-1.2.5 directory since it might contain other file or patches that we don't actaully use. We need to get the patches that are actually applied.

Quoted 3 lines

Quoted 2 lines

2 - Original Source code & Patches
- could include additional code

Please give me more detailed information

One tarball with the patches extracted in the patches directory with the series file (possibly a script to apply the patches).

Quoted 7 lines

Quoted 2 lines

- post unpack
- 1 archive

$ls zlib-orgsource
zlib-1.2.5.tar.bz2(after do_unpack, patches are not included in this package.)

No patch in this directory.

Need to add the patches via patch mechanism to the source tarball, but without actually applying the patches.

So, this is just one tarball.
zlib-1.2.5-prepatch.tar.bz2

Quoted 9 lines

Quoted 4 lines

3 - Original Source code & Patches & temp (scripts & logs)
- Could possibly include the .bb and .inc files
- 2 archives (from #2 & temp tarball)

$ls zlib-orgsource
zlib-1.2.5.tar.bz2(after do_unpack, patches are not included in this package.)
zlib-patches.tar.bz2( the patches come from meta/recipes-core/zlib/zlib-1.2.5)

Again see above, we need to get only the patches that will actually be applied.

Quoted 3 lines

zlib-scripts-logs.tar.bz2(include zlib_1.2.5.bb,zlib.inc and tmp/log_*)

So there are 3 tarballs in this directory.

Yes

Quoted 5 lines

Quoted 3 lines

4 - Patched source code
- original source code could be removed
- post do_patch

$ls zlib-source
zlib-1.2.5.tar.bz2(after do_patch)

Correct
Maybe better zlib-1.2.5-patched.tar.bz2

Quoted 5 lines

Quoted 2 lines

5 - Patched source code & temp
- 2 archives (from #4 & temp tarball)

$ls zlib-source
zlib-1.2.5.tar.bz2(after do_patch)
zlib-logs.tar.bz2(include tmp/log_*)

Correct
Same as #4 change the tarball name

Quoted 4 lines

Quoted 2 lines

6 - Configured Source
- post do_configure

$ls zlib-source
zlib-1.2.5.tar.bz2(after do_configure)

zlib-1.2.5-configured.tar.bz2

Quoted 9 lines

Quoted 3 lines

7 - SRPM format of Original Source & Patches
- rpm will apply the patches
- internally contains #2 above

$ls zlib-srpm
zlib-1.2.5.src.rpm

zlib-1.2.5.src.rpm includes zlib-1.2.5.tar.bz2(after do_unpack, patches are not included in this package.) and zlib-patches.tar.bz2

Quoted 1 line

8 - SRPM various of #3 above

Quoted 9 lines

$ls zlib-srpm
zlib-1.2.5.src.rpm

zlib-1.2.5.src.rpm includes zlib-1.2.5.tar.bz2(after do_unpack, patches are not included in this package.) ,zlib-patches.tar.bz2 and zlib-scripts-logs.tar.bz2

Do you have any suggestion about the above description?

Just the naming and where the patches come from we need to provide the mechanism (series file) for how to apply the patches also.

Sau!

Quoted 58 lines

Thanks
Yan

Quoted 52 lines

...
100 - Buildable SRPM
- can actually build, this is way future!

(Patches = patch files & series list)

Each of these build on the previous in some way, the key being that we generate a tarball for the source from the existing state of the WORKDIR, a challenge maybe to create the source snapshot after the build
has already occurred.

The SPDX License info could also be included in any of these

After talking with Richard, we think we have a novel approach to make this work. It would entail using the postfuncs feature similar to the way that Shared State does it's work. The existing copyleft_compliance class functionality can be folded into this as a filter. Additional flags could be passed from individual classes to a core set of methods in an archiver class defining the type of data (source, patches, temp, env, recipe info, ...) and format (tar, sprm, ...)

I noted that recipe type code might be better suited as generic code, as
I believe there are other places that could benefit from that code.

The sourcepkg class seems to be a basic archiver and differ that includes the metadata/environment (as dumpdata), this could be replaced
by the new approach. While the src_distribute class copies the downloaded archive and then creates a link, into LICENSE directories along with the patches. The src_distribute by default seems to move files and create links (incorrectly btw!). This work can be done by the
archiver class.

The Nugget: Create a new core "archiver" class that implements a general
functions that can archive the original tarball or workdir at various states along task list with additional metadata (recipe info, temp dir,
environment). This class would be inherited by a set of classes that use
the postfunc (similar to sstate) that setup what level of archive is needed (based on the list above).

Thoughts, Comments?

Thanks

Xiaofeng & Community:

We had a chat with Chris Larson and Mark Hatle here at ELC.

We focused this issue down to a fewer number of options that meet the needs of the licenses. Please review this and let us know if there are any issues or concerns.

Best Practices:
- Archive during build, we do not support nor recommend post archiving
- Original tarball and patches w/ordering file (maybe comment the series), grab non-patch files also.

- Unless requesting configured sources, which will just be the configured source tarball
- Grab temp as postfunc of do_package - latest logs via links & pid

For sstate Builds (LATER):
- need to verify that fetch/patch/configure will get re-run for
archiving
- Add temp dir to sstate capture (without links)

3 Implementations classes
- Filter in oe/lib/license.py
- source_archive.bb
- archives tarballs to ${BP}/...
- reuse copyleft_compliance for patch handling
- srpm
- take output from above and create SRPM - (LATER)

2 Configuration classes
- uses prefunc/postfunc at correct place
- Original Tarball / Patches
- Post Configuration

SOURCE_ARCHIVE_PACKAGE_TYPE = {tar, srpm}
SOURCE_ARCHIVE_LOG = {True, False}

Please let us know if you have any questions.

Sau!

Quoted 10 lines · Saul Wold

Xiaofeng & Community:

We had a chat with Chris Larson and Mark Hatle here at ELC.

We focused this issue down to a fewer number of options that meet the needs of the licenses. Please review this and let us know if there are any issues or concerns.

Hi Saul,

I comment my understanding as follow:

Quoted 4 lines

Best Practices:
- Archive during build, we do not support nor recommend post archiving
- Original tarball and patches w/ordering file (maybe comment the series), grab non-patch files also.

for example:
file://a.patch
file://b.config.in
we also archive b.config.in.

Quoted 4 lines

- Unless requesting configured sources, which will just be the configured source tarball
- Grab temp as postfunc of do_package - latest logs via links & pid

That means logs package include all of logs in temp. for example. You has described a following function we will realize at the previous email.
3 - Original Source code & Patches & temp (scripts & logs)

source codes and patches should be in the stage do_patch[postfunc] = " do_get_source"
logs should be in the stage do_package[postfunc] = " do_get_logs" So logs file shouldn't be archived in do_patch[postfunc] = " do_get_logs" The logs in the stage of do_packge are more than do_patch.

Quoted 4 lines

For sstate Builds (LATER):
- need to verify that fetch/patch/configure will get re-run for archiving
- Add temp dir to sstate capture (without links)

Quoted 8 lines

3 Implementations classes
- Filter in oe/lib/license.py
- source_archive.bb
- archives tarballs to ${BP}/...
- reuse copyleft_compliance for patch handling
- srpm
- take output from above and create SRPM - (LATER)

We will do the archive work according the license . I will implement two functions, one is for left, the other for right.

Quoted 5 lines

2 Configuration classes
- uses prefunc/postfunc at correct place
- Original Tarball / Patches
- Post Configuration

We can define 8 classes to complete 8 kinds of archiving methods. every class will inherit archiver.bbclass.

and then realize do_stage[prefunc/postfunc] in this class. stage include unpack, patch, configure, build package and so on. for example. we want only to get original tarball, then we can define a bbclass named "source.bbclass"
source.bbclass:
inherit archiver.bbclass
do_unpack[postfunc] = " do_get_source"

if we want to get both original and patches tarball, then we can define bbclass names "source_patches.bbclass" source_patches.bbclass
inherit archiver.bbclass
do_unpack[postfunc] = " do_get_source_patches"

Quoted 7 lines

SOURCE_ARCHIVE_PACKAGE_TYPE = {tar, srpm}
SOURCE_ARCHIVE_LOG = {True, False}

Please let us know if you have any questions.

Sau!

If my understanding don't meet your ideas, Please correct me.

Thanks for your help very much.

Thanks
Yan

Quoted 2 lines · Xiaofeng Yan

Quoted 2 lines · Xiaofeng Yan

Quoted 1 line · Saul Wold

Quoted 12 lines

Quoted 9 lines

Xiaofeng & Community:

We had a chat with Chris Larson and Mark Hatle here at ELC.

We focused this issue down to a fewer number of options that meet the needs of the licenses. Please review this and let us know if there are any issues or concerns.

Hi Saul,

I comment my understanding as follow:

Quoted 8 lines

Quoted 4 lines

Best Practices:
- Archive during build, we do not support nor recommend post archiving
- Original tarball and patches w/ordering file (maybe comment the series), grab non-patch files also.

for example:
file://a.patch
file://b.config.in
we also archive b.config.in.

Yes, correct.

Quoted 9 lines

Quoted 4 lines

- Unless requesting configured sources, which will just be the configured source tarball
- Grab temp as postfunc of do_package - latest logs via links & pid

That means logs package include all of logs in temp. for example. You has described a following function we will realize at the previous email.
3 - Original Source code & Patches & temp (scripts & logs)

Quoted 6 lines

source codes and patches should be in the stage do_patch[postfunc] = " do_get_source"
logs should be in the stage do_package[postfunc] = " do_get_logs" So logs file shouldn't be archived in do_patch[postfunc] = " do_get_logs" The logs in the stage of do_packge are more than do_patch.

Correct, you want to get the logs and scripts from temp when they are most complete (after do_package not after do_patch).

As mentioned below, you would use the variable SOURCE_ARCHIVE_LOG in the do_package[postfunc] to determine if you need to archive the logs or not.

Quoted 13 lines

Quoted 13 lines

For sstate Builds (LATER):
- need to verify that fetch/patch/configure will get re-run for archiving
- Add temp dir to sstate capture (without links)

3 Implementations classes
- Filter in oe/lib/license.py
- source_archive.bb
- archives tarballs to ${BP}/...
- reuse copyleft_compliance for patch handling
- srpm
- take output from above and create SRPM - (LATER)

Quoted 2 lines

We will do the archive work according the license . I will implement two functions, one is for left, the other for right.

I am not sure what you mean by one is for left and the other is for right?

Can you explain your thoughts here.

Quoted 9 lines

Quoted 7 lines

2 Configuration classes
- uses prefunc/postfunc at correct place
- Original Tarball / Patches
- Post Configuration

We can define 8 classes to complete 8 kinds of archiving methods. every class will inherit archiver.bbclass.

I am not sure we need 8 classes anymore, that was the point of this email to simplify the needs.

2 Classes:

source_patches.bbclass
- Original Tarball & Patches & Other files
- do_patch[postfunc] = <archive ${S}/patches with Original tarball>
- Included temp dir if enabled via SOURCE_ARCHIVE_LOG

configured_source.bbclass
- Get the ${S} dir (or $S & any build dir) after do_configure
- do_configure[postfunc] = <archive configured source>
- Include temp dir if enabled via SOURCE_ARCHIVE_LOG

Quoted 14 lines

and then realize do_stage[prefunc/postfunc] in this class. stage include unpack, patch, configure, build package and so on. for example. we want only to get original tarball, then we can define a bbclass named "source.bbclass"
source.bbclass:
inherit archiver.bbclass
do_unpack[postfunc] = " do_get_source"

if we want to get both original and patches tarball, then we can define bbclass names "source_patches.bbclass" source_patches.bbclass
inherit archiver.bbclass
do_unpack[postfunc] = " do_get_source_patches"

Quoted 9 lines

Quoted 7 lines

SOURCE_ARCHIVE_PACKAGE_TYPE = {tar, srpm}
SOURCE_ARCHIVE_LOG = {True, False}

Please let us know if you have any questions.

Sau!

If my understanding don't meet your ideas, Please correct me.

I should be available via IRC in your afternoon, I will be away during your morning.

Sau!

Quoted 7 lines

Thanks for your help very much.

Thanks
Yan

Quoted 2 lines · Xiaofeng Yan

Quoted 6 lines · Saul Wold

Quoted 4 lines · Xiaofeng Yan

Quoted 2 lines · Saul Wold

Hi Saul,

I have implemented the most functions but srpm functions. I will complete the left work(merge functions from sourcepkg, src_distribute  to archiver.bbclass, implement srpm functions. )

Quoted 64 lines

Quoted 18 lines

Quoted 9 lines

Xiaofeng & Community:

We had a chat with Chris Larson and Mark Hatle here at ELC.

We focused this issue down to a fewer number of options that meet the needs of the licenses. Please review this and let us know if there are any issues or concerns.

Hi Saul,

I comment my understanding as follow:

Quoted 5 lines

Best Practices:
- Archive during build, we do not support nor recommend post archiving
- Original tarball and patches w/ordering file (maybe comment the series), grab non-patch files also.

Quoted 4 lines

for example:
file://a.patch
file://b.config.in
we also archive b.config.in.

Yes, correct.

Quoted 9 lines

Quoted 4 lines

- Unless requesting configured sources, which will just be the configured source tarball
- Grab temp as postfunc of do_package - latest logs via links & pid

That means logs package include all of logs in temp. for example. You has described a following function we will realize at the previous email.
3 - Original Source code & Patches & temp (scripts & logs)

Quoted 6 lines

source codes and patches should be in the stage do_patch[postfunc] = " do_get_source"
logs should be in the stage do_package[postfunc] = " do_get_logs" So logs file shouldn't be archived in do_patch[postfunc] = " do_get_logs" The logs in the stage of do_packge are more than do_patch.

Correct, you want to get the logs and scripts from temp when they are most complete (after do_package not after do_patch).

As mentioned below, you would use the variable SOURCE_ARCHIVE_LOG in the do_package[postfunc] to determine if you need to archive the logs or not.

Quoted 15 lines

Quoted 5 lines

For sstate Builds (LATER):
- need to verify that fetch/patch/configure will get re-run for archiving
- Add temp dir to sstate capture (without links)

Quoted 8 lines

3 Implementations classes
- Filter in oe/lib/license.py
- source_archive.bb
- archives tarballs to ${BP}/...
- reuse copyleft_compliance for patch handling
- srpm
- take output from above and create SRPM - (LATER)

We will do the archive work according the license . I will implement two functions, one is for left, the other for right.

Quoted 3 lines

I am not sure what you mean by one is for left and the other is for right?

Can you explain your thoughts here.

I mean I need implement a function to select to archive the packages from copyright(no free)or copyleft(free).