Skip to contentSkip to conversation

duplicate code

npm Jean-marie Lemetayer 3 participants 8 messages Oct 3, 2019 – Oct 8, 2019

lore
Share & tools
lore
#0 lore raw

Hi all,

I am planning to do some work on npm and recipetool.

I have seen that there is some duplicate code between bitbake/lib/bb/fetch2/npm.py and scripts/lib/recipetool/create_npm.py but there is some inconsistencies.

Here is an example, the _parse_view() function which I patched recently has differ:
https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/lib/bb/fetch2/npm.py#n157
https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/scripts/lib/recipetool/create_npm.py#n244

I think merging this "npm utility functions" is a good idea. But where ?

I think about creating a new file: bitbake/lib/bb/npm_utils.py

Or maybe use the existing bitbake/lib/bb/utils.py ?

What do you think ?

Jean-Marie LEMETAYER
Enthusiast embedded systems engineer
Savoir-faire Linux

#1 lore raw

Hi Jean-Marie,

what work do you plan to do on npm and recipetool?

I have start to rework the complete npm integration to synchronize it with other languages like python. I replace the bitbake npm fetch with the wget fetcher and download the npm tgz files direct. The idea is to handle every package as a separate recipe and move the package dependencies into the recipes. Therefore I parse the package.json and the npm server json to create a recipe.

Regards
Stefan

Am 03.10.19 um 17:37 schrieb Jean-marie Lemetayer:

Quoted 23 lines

Hi all,

I am planning to do some work on npm and recipetool.

I have seen that there is some duplicate code between bitbake/lib/bb/fetch2/npm.py and scripts/lib/recipetool/create_npm.py but there is some inconsistencies.

Here is an example, the _parse_view() function which I patched recently has differ:
https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/lib/bb/fetch2/npm.py#n157
https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/scripts/lib/recipetool/create_npm.py#n244

I think merging this "npm utility functions" is a good idea. But where ?

I think about creating a new file: bitbake/lib/bb/npm_utils.py

Or maybe use the existing bitbake/lib/bb/utils.py ?

What do you think ?

Jean-Marie LEMETAYER
Enthusiast embedded systems engineer
Savoir-faire Linux

#2 lore raw

Hi Stefan,

I have recently worked on a yocto project using npm and I have seen some issues. I have solved a few but only for bitbake:
https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=eecba41822e86b69ebdb9cbc8fbfd512ad9a47d7
https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=a34d0d539e5fdf341541fb628652d22289e80512
https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=e90cd2ed61b93ee7e290e7e592f1f0242ab5c281
https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=a35abe31dc23916fd06afdb3a5e22a81ef3df579

As I have more time now, I wanted to continue my work by fixing devtool / recipetool.

I have also checked the bugzilla for issues that I could fix / that should be tested again:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=10515
https://bugzilla.yoctoproject.org/show_bug.cgi?id=10760
https://bugzilla.yoctoproject.org/show_bug.cgi?id=11028
https://bugzilla.yoctoproject.org/show_bug.cgi?id=11029
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12534
https://bugzilla.yoctoproject.org/show_bug.cgi?id=13415

I ended up with this todo list:
- merge the duplicate code between bitbake and recipetool
- fix the npm package name handling in recipetool
- fix the npm package version handling in recipetool
- fix the lockdown.json file generation in recipetool
- create an example nodejs application to test all these cases
- update the wiki using this example application:
https://wiki.yoctoproject.org/wiki/TipsAndTricks/NPM

Finally, in my recent project, we were using angular (https://angular.io) for the front-end. I have planned to add the work done to support angular applications in yocto (angular.bbclass) and update recipetool to handle them.

Your work sounds very interesting. The good point is that npm-shrinkwrap.json and lockdown.json files (which have generation issues btw) will no longer be required. But projects using npm can have a lot of dependencies (e.g. the angular example app have 1053 dependencies). Is recipetool will be handling the whole recipes creation in one time ?

Is it possible to see your work ? A public fork would be nice. I would gladly help you / test your work / add my fixes if needed.

Best regards,
Jean-Marie Lemetayer

----- Mail original -----
De: "Stefan Herbrechtsmeier" <[email protected]> À: "Yocto-mailing-list" <[email protected]> Envoyé: Vendredi 4 Octobre 2019 08:53:10
Objet: Re: [yocto] [npm] duplicate code

Hi Jean-Marie,

what work do you plan to do on npm and recipetool?

I have start to rework the complete npm integration to synchronize it with other languages like python. I replace the bitbake npm fetch with the wget fetcher and download the npm tgz files direct. The idea is to handle every package as a separate recipe and move the package dependencies into the recipes. Therefore I parse the package.json and the npm server json to create a recipe.

Regards
Stefan

Am 03.10.19 um 17:37 schrieb Jean-marie Lemetayer:

Quoted 23 lines

Hi all,

I am planning to do some work on npm and recipetool.

I have seen that there is some duplicate code between bitbake/lib/bb/fetch2/npm.py and scripts/lib/recipetool/create_npm.py but there is some inconsistencies.

Here is an example, the _parse_view() function which I patched recently has differ:
https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/lib/bb/fetch2/npm.py#n157
https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/scripts/lib/recipetool/create_npm.py#n244

I think merging this "npm utility functions" is a good idea. But where ?

I think about creating a new file: bitbake/lib/bb/npm_utils.py

Or maybe use the existing bitbake/lib/bb/utils.py ?

What do you think ?

Jean-Marie LEMETAYER
Enthusiast embedded systems engineer
Savoir-faire Linux

Signature · 4 lines
_______________________________________________
yocto mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/yocto
#3 lore raw

Hi Jean-Marie,

Am 04.10.19 um 14:37 schrieb Jean-marie Lemetayer:

Quoted 25 lines
Quoted 1 line

I have recently worked on a yocto project using npm and I have seen some issues. I have solved a few but only for bitbake:

https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=eecba41822e86b69ebdb9cbc8fbfd512ad9a47d7
https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=a34d0d539e5fdf341541fb628652d22289e80512
https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=e90cd2ed61b93ee7e290e7e592f1f0242ab5c281
https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=a35abe31dc23916fd06afdb3a5e22a81ef3df579

As I have more time now, I wanted to continue my work by fixing devtool / recipetool.

I have also checked the bugzilla for issues that I could fix / that should be tested again:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=10515
https://bugzilla.yoctoproject.org/show_bug.cgi?id=10760
https://bugzilla.yoctoproject.org/show_bug.cgi?id=11028
https://bugzilla.yoctoproject.org/show_bug.cgi?id=11029
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12534
https://bugzilla.yoctoproject.org/show_bug.cgi?id=13415

I ended up with this todo list:
- merge the duplicate code between bitbake and recipetool
- fix the npm package name handling in recipetool
- fix the npm package version handling in recipetool
- fix the lockdown.json file generation in recipetool
- create an example nodejs application to test all these cases
- update the wiki using this example application:
https://wiki.yoctoproject.org/wiki/TipsAndTricks/NPM

Quoted 1 line

Finally, in my recent project, we were using angular (https://angular.io) for the front-end. I have planned to add the work done to support angular applications in yocto (angular.bbclass) and update recipetool to handle them.

Do you want to build an angular application? In this case you have to produce a machine (independent) package via a native npm build because you need to run the angular compiler on the build machine.

Quoted 2 lines

Your work sounds very interesting. The good point is that npm-shrinkwrap.json and lockdown.json files (which have generation issues btw) will no longer be required. But projects using npm can have a lot of dependencies (e.g. the angular example app have 1053 dependencies).

A lot of dependencies are the problem at the moment. But many dependencies are not update frequently and my current solution assume that I can always use the latest version per major release version

Quoted 1 line

Is recipetool will be handling the whole recipes creation in one time ?

That's the problem at the moment. I don't manage to build multiple recipes from one recipetool run. Therefore I have a two stage approach. I use recipetool to build exact one recipe. A second script generates a list of all dependencies and call recipetool per package if the recipe doesn't already exist. This works but the run time is very high.

Quoted 1 line

Is it possible to see your work ? A public fork would be nice. I would gladly help you / test your work / add my fixes if needed.

I only have a unfinished Proof of Concept. If you like I could clean it up a bit and post it on github as a separate layer.

Regards
Stefan

#4 lore raw

Hi Stefan,

I thought about your idea of using Yocto to manage NPM package dependencies and I ran into an issue. NPM projects can have multiple dependencies on a single package, and sometimes with multiple versions. NPM will manage this by creating sub 'node_modules' tree.

Here is an example with a newly created angular application. The app depends on 3 different version of 'ansi-regex'. NPM will install the packages this way:
node_modules/ansi-regex @ 2.1.1
node_modules/cliui/node_modules/ansi-regex @ 3.0.0
node_modules/inquirer/node_modules/ansi-regex @ 4.1.0
node_modules/string-width/node_modules/ansi-regex @ 3.0.0
node_modules/@angular/compiler-cli/node_modules/ansi-regex @ 4.1.0

How can you handle that with Yocto ? I am not sure but I think it is not possible.

For that reason I think I will base my developments on the master branch.

----- Mail original -----

Quoted 37 lines

De: "Stefan Herbrechtsmeier" <[email protected]>
À: "Jean-marie Lemetayer" <[email protected]>
Cc: "Yocto-mailing-list" <[email protected]>, "Savoir-faire Linux Rennes" <[email protected]> Envoyé: Vendredi 4 Octobre 2019 16:55:47
Objet: Re: [yocto] [npm] duplicate code

Hi Jean-Marie,

Am 04.10.19 um 14:37 schrieb Jean-marie Lemetayer:

Quoted 28 lines
Quoted 2 lines

I have recently worked on a yocto project using npm and I have seen some issues. I have solved a few but only for bitbake:

https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=eecba41822e86b69ebdb9cbc8fbfd512ad9a47d7
https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=a34d0d539e5fdf341541fb628652d22289e80512
https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=e90cd2ed61b93ee7e290e7e592f1f0242ab5c281
https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=a35abe31dc23916fd06afdb3a5e22a81ef3df579

As I have more time now, I wanted to continue my work by fixing devtool / recipetool.

I have also checked the bugzilla for issues that I could fix / that should be tested again:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=10515
https://bugzilla.yoctoproject.org/show_bug.cgi?id=10760
https://bugzilla.yoctoproject.org/show_bug.cgi?id=11028
https://bugzilla.yoctoproject.org/show_bug.cgi?id=11029
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12534
https://bugzilla.yoctoproject.org/show_bug.cgi?id=13415

I ended up with this todo list:
  - merge the duplicate code between bitbake and recipetool
  - fix the npm package name handling in recipetool
  - fix the npm package version handling in recipetool
  - fix the lockdown.json file generation in recipetool
  - create an example nodejs application to test all these cases
  - update the wiki using this example application:
    https://wiki.yoctoproject.org/wiki/TipsAndTricks/NPM

Quoted 7 lines
Quoted 3 lines

Finally, in my recent project, we were using angular (https://angular.io) for the front-end. I have planned to add the work done to support angular applications in yocto (angular.bbclass) and update recipetool to handle them.

Do you want to build an angular application? In this case you have to produce a machine (independent) package via a native npm build because you need to run the angular compiler on the build machine.

Yes, I have an 'angular-cli_x.y.z.bb' and an 'angular.bbclass' which depends on 'angular-cli-native'. I have planned to upstream this work.

Quoted 19 lines
Quoted 6 lines

Your work sounds very interesting. The good point is that npm-shrinkwrap.json and lockdown.json files (which have generation issues btw) will no longer be required. But projects using npm can have a lot of dependencies (e.g. the angular example app have 1053 dependencies).

A lot of dependencies are the problem at the moment. But many dependencies are not update frequently and my current solution assume that I can always use the latest version per major release version

Quoted 1 line

Is recipetool will be handling the whole recipes creation in one time ?

That's the problem at the moment. I don't manage to build multiple recipes from one recipetool run. Therefore I have a two stage approach. I use recipetool to build exact one recipe. A second script generates a list of all dependencies and call recipetool per package if the recipe doesn't already exist. This works but the run time is very high.

Quoted 5 lines
Quoted 2 lines

Is it possible to see your work ? A public fork would be nice. I would gladly help you / test your work / add my fixes if needed.

I only have a unfinished Proof of Concept. If you like I could clean it up a bit and post it on github as a separate layer.

Like I said at the start of this email, I will not base my dev on yours. But I am still interested by seeing your work.

Best regards,
Jean-Marie

#5 lore raw

Hi Jean-Marie,

Am 07.10.19 um 14:16 schrieb Jean-marie Lemetayer:

Quoted 8 lines
Quoted 1 line

I thought about your idea of using Yocto to manage NPM package dependencies and I ran into an issue. NPM projects can have multiple dependencies on a single package, and sometimes with multiple versions. NPM will manage this by creating sub 'node_modules' tree.

Here is an example with a newly created angular application. The app depends on 3 different version of 'ansi-regex'. NPM will install the packages this way:
node_modules/ansi-regex @ 2.1.1
node_modules/cliui/node_modules/ansi-regex @ 3.0.0
node_modules/inquirer/node_modules/ansi-regex @ 4.1.0
node_modules/string-width/node_modules/ansi-regex @ 3.0.0
node_modules/@angular/compiler-cli/node_modules/ansi-regex @ 4.1.0

I use symbolic links instead of folders, append the major version to the folder name and move the folders into the main node_modules folder. At the moment I use one version per major version or first version unequal to zero:

node_modules/ansi-regex-2
node_modules/ansi-regex-3
node_modules/ansi-regex-4
node_modules/cliui-X/node_modules/ansi-regex -> ../../ansi-regex-3 node_modules/inquirer-X/node_modules/ansi-regex -> ../../ansi-regex-3 node_modules/string-width-X/node_modules/ansi-regex -> ../../ansi-regex-3 node_modules/@angular/compiler-cli-X/node_modules/ansi-regex -> ../../../ansi-regex-4 node_modules/abc-X/node_modules/ansi-regex -> ../../ansi-regex-2 node_modules/abc-X/node_modules/cliui -> ../../cliui-X node_modules/abc-X/node_modules/inquirer -> ../../inquirer-X node_modules/abc-X/node_modules/string-width -> ../../string-width node_modules/abc-X/node_modules/@angular/compiler-cli -> ../../../@angular/compiler-cli-X

Quoted 1 line

How can you handle that with Yocto ? I am not sure but I think it is not possible.

It is possible and Yocto could do even more. If you take a look at the package ansi-regex you realize that the project increase the major version only because they change the minimum required nodejs version. This means you could replace all ansi-regex version by the last version, add some symbolic links to the ansi-regex package and provide all versions by the same package.

Regards
Stefan

#6 lore raw
Quoted 32 lines · Stefan Herbrechtsmeier

Hi Jean-Marie,

Am 07.10.19 um 14:16 schrieb Jean-marie Lemetayer:

Quoted 8 lines
Quoted 1 line

I thought about your idea of using Yocto to manage NPM package dependencies and I ran into an issue. NPM projects can have multiple dependencies on a single package, and sometimes with multiple versions. NPM will manage this by creating sub 'node_modules' tree.

Here is an example with a newly created angular application. The app depends on 3 different version of 'ansi-regex'. NPM will install the packages this way:
node_modules/ansi-regex @ 2.1.1
node_modules/cliui/node_modules/ansi-regex @ 3.0.0
node_modules/inquirer/node_modules/ansi-regex @ 4.1.0
node_modules/string-width/node_modules/ansi-regex @ 3.0.0
node_modules/@angular/compiler-cli/node_modules/ansi-regex @ 4.1.0

I use symbolic links instead of folders, append the major version to the folder name and move the folders into the main node_modules folder. At the moment I use one version per major version or first version unequal to zero:

node_modules/ansi-regex-2
node_modules/ansi-regex-3
node_modules/ansi-regex-4 node_modules/cliui-X/node_modules/ansi-regex -> ../../ansi-regex-3 node_modules/inquirer-X/node_modules/ansi-regex -> ../../ansi-regex-3 node_modules/string-width-X/node_modules/ansi-regex -> ../../ansi-regex-3 node_modules/@angular/compiler-cli-X/node_modules/ansi-regex -> ../../../ansi-regex-4 node_modules/abc-X/node_modules/ansi-regex -> ../../ansi-regex-2 node_modules/abc-X/node_modules/cliui -> ../../cliui-X node_modules/abc-X/node_modules/inquirer -> ../../inquirer-X node_modules/abc-X/node_modules/string-width -> ../../string-width node_modules/abc-X/node_modules/@angular/compiler-cli -> ../../../@angular/compiler-cli-X

Quoted 7 lines
Quoted 1 line

How can you handle that with Yocto ? I am not sure but I think it is not possible.

It is possible and Yocto could do even more. If you take a look at the package ansi-regex you realize that the project increase the major version only because they change the minimum required nodejs version. This means you could replace all ansi-regex version by the last version, add some symbolic links to the ansi-regex package and provide all versions by the same package.

I think it would be dangerous to assume this holds true for other packages too. In the semantic versioning world as npm packages expectexd to behave - and nodejs itself certainly does! - a change in the major version means a breaking change of some kind.

Quoted 8 lines

Regards
Stefan

Signature · 4 lines
_______________________________________________
yocto mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/yocto
Signature · 20 lines
———————————————
Josef Holzmayr
Software Developer Embedded Systems

Tel: +49 8444 9204-48
Fax: +49 8444 9204-50

R-S-I Elektrotechnik GmbH & Co. KG
Woelkestrasse 11
D-85301 Schweitenkirchen
www.rsi-elektrotechnik.de
———————————————
Amtsgericht Ingolstadt – GmbH: HRB 191328 – KG: HRA 170393
Geschäftsführer: Dr.-Ing. Michael Sorg, Dipl.-Ing. Franz Sorg
Ust-IdNr: DE 128592548 

_____________________________________________________________
Amtsgericht Ingolstadt - GmbH: HRB 191328 - KG: HRA 170363
Geschäftsführer: Dr.-Ing. Michael Sorg, Dipl.-Ing. Franz Sorg
USt-IdNr.: DE 128592548
#7 lore raw

Hi Josef,

Am 08.10.19 um 07:12 schrieb Josef Holzmayr:

Quoted 41 lines · Stefan Herbrechtsmeier
Quoted 32 lines

Hi Jean-Marie,

Am 07.10.19 um 14:16 schrieb Jean-marie Lemetayer:

Quoted 8 lines
Quoted 1 line

I thought about your idea of using Yocto to manage NPM package dependencies and I ran into an issue. NPM projects can have multiple dependencies on a single package, and sometimes with multiple versions. NPM will manage this by creating sub 'node_modules' tree.

Here is an example with a newly created angular application. The app depends on 3 different version of 'ansi-regex'. NPM will install the packages this way:
node_modules/ansi-regex @ 2.1.1
node_modules/cliui/node_modules/ansi-regex @ 3.0.0
node_modules/inquirer/node_modules/ansi-regex @ 4.1.0
node_modules/string-width/node_modules/ansi-regex @ 3.0.0
node_modules/@angular/compiler-cli/node_modules/ansi-regex @ 4.1.0

I use symbolic links instead of folders, append the major version to the folder name and move the folders into the main node_modules folder. At the moment I use one version per major version or first version unequal to zero:

node_modules/ansi-regex-2
node_modules/ansi-regex-3
node_modules/ansi-regex-4 node_modules/cliui-X/node_modules/ansi-regex -> ../../ansi-regex-3 node_modules/inquirer-X/node_modules/ansi-regex -> ../../ansi-regex-3 node_modules/string-width-X/node_modules/ansi-regex -> ../../ansi-regex-3 node_modules/@angular/compiler-cli-X/node_modules/ansi-regex -> ../../../ansi-regex-4 node_modules/abc-X/node_modules/ansi-regex -> ../../ansi-regex-2 node_modules/abc-X/node_modules/cliui -> ../../cliui-X node_modules/abc-X/node_modules/inquirer -> ../../inquirer-X node_modules/abc-X/node_modules/string-width -> ../../string-width node_modules/abc-X/node_modules/@angular/compiler-cli -> ../../../@angular/compiler-cli-X

Quoted 7 lines
Quoted 1 line

How can you handle that with Yocto ? I am not sure but I think it is not possible.

It is possible and Yocto could do even more. If you take a look at the package ansi-regex you realize that the project increase the major version only because they change the minimum required nodejs version. This means you could replace all ansi-regex version by the last version, add some symbolic links to the ansi-regex package and provide all versions by the same package.

Quoted 2 lines

I think it would be dangerous to assume this holds true for other packages too.

I don't assume this. It is a manual optimization of individual recipes. The OE package name (PN) for a NPM package consists of the NPM package name and the major version. An individual recipe could contain multiple RPROVIDES for different major version if the API is compatible.

In the semantic versioning world as npm packages expectexd

Quoted 2 lines

to behave - and nodejs itself certainly does! - a change in the major version means a breaking change of some kind.

And one of this breaking change is the update of the minimal Node.js version.

Regards
Stefan