scripts/cve: Avoid to do a complete clone of cve git repository
PATCH v2 1/2 scripts/cve: Avoid to do a complete clone of cve git repository
Just a simple clone and pull with --depth 1 should be enough to parse the cve and generate the pkg-stats report.
From a full clone and a depth-1 clone, and the size delta is 2.9GiB vs. 2.2GiB. The download size does change: from 983.55MiB down to 270.78MiB. it's a net time win too: 2m17s vs 1min7s (on a 100Mbps link).
---V1->V2:
- Add statistics from Yann E. Morin
- Use git pull --depth 1 for update the repo
--- support/scripts/cve.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
support/scripts/cve.py +2 -2
@@ -72,7 +72,7 @@ class CVE: print(f"Updating from {NVD_BASE_URL}") if os.path.exists(nvd_git_dir): subprocess.check_call( - ["git", "pull"], + ["git", "pull", "--depth", "1"], cwd=nvd_git_dir, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
@@ -82,7 +82,7 @@ class CVE: # happily clones into an empty directory. os.makedirs(nvd_git_dir) subprocess.check_call( - ["git", "clone", NVD_BASE_URL, nvd_git_dir], + ["git", "clone", "--depth", "1", NVD_BASE_URL, nvd_git_dir], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, )
Signature · 6 lines
2.43.0 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH v2 2/2 scripts/cve: Restart the clone if the pull generate an exception
If we are not able to pull from the directory, restart from a clean clone. This can happen for corrupt repository or unfinished download
---V1->V2: Adjust the commit message
--- support/scripts/cve.py | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-)
support/scripts/cve.py +14 -7
@@ -21,6 +21,7 @@ import datetime import os import distutils.version import json +import shutil import subprocess import sys import operator
@@ -69,15 +70,21 @@ class CVE: @staticmethod def download_nvd(nvd_git_dir): + done = False print(f"Updating from {NVD_BASE_URL}") if os.path.exists(nvd_git_dir): - subprocess.check_call( - ["git", "pull", "--depth", "1"], - cwd=nvd_git_dir, - stdout=subprocess.DEVNULL, - stderr=subprocess.DEVNULL, - ) - else: + try: + subprocess.check_call( + ["git", "pull", "--depth", "1"], + cwd=nvd_git_dir, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + done = True + except: + shutil.rmtree(nvd_git_dir) + + if (not done): # Create the directory and its parents; git # happily clones into an empty directory. os.makedirs(nvd_git_dir)
Signature · 6 lines
2.43.0 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH V2 2/2] scripts/cve: Restart the clone if the pull generate an exception
Quoted 26 lines · On Tue, 3 Sep 2024 10:57:45 +0200 Michael Trimarchi…
support/scripts/cve.py +2
-7@@ -21,6 +21,7 @@ import datetime import os import distutils.version import json +import shutil import subprocess import sys import operator@@ -69,15 +70,21 @@ class CVE: @staticmethod def download_nvd(nvd_git_dir): + done = False print(f"Updating from {NVD_BASE_URL}") if os.path.exists(nvd_git_dir): - subprocess.check_call( - ["git", "pull", "--depth", "1"], - cwd=nvd_git_dir, - stdout=subprocess.DEVNULL, - stderr=subprocess.DEVNULL, - ) - else:
support/scripts/cve.py class CVE: · line 85
Quoted 10 lines
support/scripts/cve.py +10
-0+ try: + subprocess.check_call( + ["git", "pull", "--depth", "1"], + cwd=nvd_git_dir, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + done = True + except: + shutil.rmtree(nvd_git_dir)
The thing I'm worried about is that you can also get a failure of "git pull" for example due to a network timeout or something like that, which doesn't need a full git clone, but just a "try again" later... and now we're going to wipe out the entire local clone, and try to clone everything again. Is that a good idea?
Also, perhaps we need to show an error message if the "git pull" failed, and saying we're falling back to a full clone, or something?
support/scripts/cve.py class CVE: · line 87
Quoted 1 line
support/scripts/cve.py +1
-0+ if (not done):
if not done:
is sufficient, we are not writing C code here :-)
Thanks!
Thomas
Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering and training https://bootlin.com _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH V2 2/2] scripts/cve: Restart the clone if the pull generate an exception
Hi Thomas
Quoted 50 lines · Thomas Petazzoni
Quoted 26 lines · On Tue, 3 Sep 2024 10:57:45 +0200 Michael Trimarchi…
support/scripts/cve.py +2
-7@@ -21,6 +21,7 @@ import datetime import os import distutils.version import json +import shutil import subprocess import sys import operator@@ -69,15 +70,21 @@ class CVE: @staticmethod def download_nvd(nvd_git_dir): + done = False print(f"Updating from {NVD_BASE_URL}") if os.path.exists(nvd_git_dir): - subprocess.check_call( - ["git", "pull", "--depth", "1"], - cwd=nvd_git_dir, - stdout=subprocess.DEVNULL, - stderr=subprocess.DEVNULL, - ) - else:support/scripts/cve.py class CVE: · line 85
Quoted 10 lines
support/scripts/cve.py +10
-0+ try: + subprocess.check_call( + ["git", "pull", "--depth", "1"], + cwd=nvd_git_dir, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + done = True + except: + shutil.rmtree(nvd_git_dir)The thing I'm worried about is that you can also get a failure of "git pull" for example due to a network timeout or something like that, which doesn't need a full git clone, but just a "try again" later... and now we're going to wipe out the entire local clone, and try to clone everything again. Is that a good idea?
Also, perhaps we need to show an error message if the "git pull" failed, and saying we're falling back to a full clone, or something?
Quoted 6 lines
support/scripts/cve.py class CVE: · line 87
Quoted 1 line
support/scripts/cve.py +1
-0+ if (not done):if not done:
is sufficient, we are not writing C code here :-)
Sorry, I will ask my colleague to write python and send a better strategy. I totally agree with you.
Anyway I have a plan to add more information on cve reporting, hope that you like the idea.
Mostly I'm playing with parser for jenkins and love to add buildroot too but I don't have enough
information to show
https://github.com/jenkinsci/analysis-model/pull/1085
Michael
Quoted 8 lines
Thanks!
Thomas
--
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering and training
https://bootlin.com
Signature · 15 lines
Michael Nazzareno Trimarchi Co-Founder & Chief Executive Officer M. +39 347 913 2170 [email protected] __________________________________ Amarula Solutions BV Joop Geesinkweg 125, 1114 AB, Amsterdam, NL T. +31 (0)85 111 9172 [email protected] www.amarulasolutions.com _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Thomas Petazzoni via buildroot
AppliedQuoted 7 lines · On Tue, 3 Sep 2024 10:57:44 +0200 Michael Trimarchi…
Just a simple clone and pull with --depth 1 should be enough to parse the cve and generate the pkg-stats report.
From a full clone and a depth-1 clone, and the size delta is 2.9GiB vs. 2.2GiB. The download size does change: from 983.55MiB down to 270.78MiB. it's a net time win too: 2m17s vs 1min7s (on a 100Mbps link).
Quoted 5 lines
Signed-off-by Michael Trimarchi <[email protected]>---
V1->V2:
- Add statistics from Yann E. Morin
- Use git pull --depth 1 for update the repo
Applied to next, thanks!
Thomas
Signature · 7 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Michael, All,
On 2024-09-03 20:50 +0200, Thomas Petazzoni via buildroot spake thusly:
Quoted 10 lines
Quoted 7 lines · On Tue, 3 Sep 2024 10:57:44 +0200 Michael Trimarchi…
Just a simple clone and pull with --depth 1 should be enough to parse the cve and generate the pkg-stats report.
From a full clone and a depth-1 clone, and the size delta is 2.9GiB vs. 2.2GiB. The download size does change: from 983.55MiB down to 270.78MiB. it's a net time win too: 2m17s vs 1min7s (on a 100Mbps link).
Quoted 7 lines
Quoted 5 lines
Signed-off-by Michael Trimarchi <[email protected]>---
V1->V2:
- Add statistics from Yann E. Morin
- Use git pull --depth 1 for update the repoApplied to next, thanks!
I don't understand: Michael said in the first iteration that we should drop the package:
https://lore.kernel.org/buildroot/CAOf5uw=m4LOk97OT1dTP=2-uP6QZ1WQyHfKYSQmWCirDaxXvgQ@mail.gmail.com/
Only the first clone is slow, the following calls will just pull (mostly nothing most of the time), so the optimisation is not really worth it.
Also, in the download backend for git, we stopped doing shallow clone because they were causing issues (but might not be applicable here).
Anyway, too late, that's been applied...
Regards,
Yann E. MORIN.
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Hi Yann
Quoted 26 lines · Yann E. MORIN
Michael, All,
On 2024-09-03 20:50 +0200, Thomas Petazzoni via buildroot spake thusly:
Quoted 10 lines
Quoted 7 lines · On Tue, 3 Sep 2024 10:57:44 +0200 Michael Trimarchi…
Just a simple clone and pull with --depth 1 should be enough to parse the cve and generate the pkg-stats report.
From a full clone and a depth-1 clone, and the size delta is 2.9GiB vs. 2.2GiB. The download size does change: from 983.55MiB down to 270.78MiB. it's a net time win too: 2m17s vs 1min7s (on a 100Mbps link).
Quoted 7 lines
Quoted 5 lines
Signed-off-by Michael Trimarchi <[email protected]>---
V1->V2:
- Add statistics from Yann E. Morin
- Use git pull --depth 1 for update the repoApplied to next, thanks!
I don't understand: Michael said in the first iteration that we should drop the package:
https://lore.kernel.org/buildroot/CAOf5uw=m4LOk97OT1dTP=2-uP6QZ1WQyHfKYSQmWCirDaxXvgQ@mail.gmail.com/
Quoted 6 lines
Only the first clone is slow, the following calls will just pull (mostly nothing most of the time), so the optimisation is not really worth it.
Also, in the download backend for git, we stopped doing shallow clone because they were causing issues (but might not be applicable here).
I have reposted it, fixing the pull. Yann if you really don't like it, people can drop, revert
Michael
Quoted 12 lines
Anyway, too late, that's been applied...
Regards,
Yann E. MORIN.--
.-----------------.--------------------.------------------.--------------------.| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |'------------------------------^-------^------------------^--------------------'
Signature · 15 lines
Michael Nazzareno Trimarchi Co-Founder & Chief Executive Officer M. +39 347 913 2170 [email protected] __________________________________ Amarula Solutions BV Joop Geesinkweg 125, 1114 AB, Amsterdam, NL T. +31 (0)85 111 9172 [email protected] www.amarulasolutions.com _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Hello Michael,
Quoted 7 lines · On Tue, 3 Sep 2024 10:57:44 +0200 Michael Trimarchi…
Just a simple clone and pull with --depth 1 should be enough to parse the cve and generate the pkg-stats report.
From a full clone and a depth-1 clone, and the size delta is 2.9GiB vs. 2.2GiB. The download size does change: from 983.55MiB down to 270.78MiB. it's a net time win too: 2m17s vs 1min7s (on a 100Mbps link).
Quoted 5 lines
Signed-off-by Michael Trimarchi <[email protected]>---
V1->V2:
- Add statistics from Yann E. Morin
- Use git pull --depth 1 for update the repo
I am sorry, but I had to revert this commit... as it doesn't work:
Updating from https://github.com/fkie-cad/nvd-json-data-feeds/
Traceback (most recent call last):
File "/home/buildroot/buildroot-stats/./support/scripts/pkg-stats", line 1346, in <module>
__main__()
File "/home/buildroot/buildroot-stats/./support/scripts/pkg-stats", line 1335, in __main__
check_package_cves(args.nvd_path, packages)
File "/home/buildroot/buildroot-stats/./support/scripts/pkg-stats", line 660, in check_package_cves
for cve in cvecheck.CVE.read_nvd_dir(nvd_path):
File "/home/buildroot/buildroot-stats/support/scripts/cve.py", line 105, in read_nvd_dir
CVE.download_nvd(nvd_git_dir)
File "/home/buildroot/buildroot-stats/support/scripts/cve.py", line 74, in download_nvd
subprocess.check_call( File "/usr/lib/python3.11/subprocess.py", line 413, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['git', 'pull', '--depth', '1']' returned non-zero exit status 128.
if I go to the machine in question:
buildroot@buildroot:~/nvd/git$ git pull --depth 1 remote: Enumerating objects: 164, done. remote: Counting objects: 100% (140/140), done. remote: Compressing objects: 100% (27/27), done. remote: Total 63 (delta 54), reused 44 (delta 36), pack-reused 0 (from 0) Unpacking objects: 100% (63/63), 18.15 KiB | 26.00 KiB/s, done. From https://github.com/fkie-cad/nvd-json-data-feeds + 51b091a6a4...7b69235976 main -> origin/main (forced update) hint: You have divergent branches and need to specify how to reconcile them. hint: You can do so by running one of the following commands sometime before hint: your next pull: hint: hint: git config pull.rebase false # merge hint: git config pull.rebase true # rebase hint: git config pull.ff only # fast-forward only hint: hint: You can replace "git config" with "git config --global" to set a default hint: preference for all repositories. You can also pass --rebase, --no-rebase, hint: or --ff-only on the command line to override the configured default per hint: invocation. fatal: Need to specify how to reconcile divergent branches.
I had already done a git reset --hard, and see if it was just a one-time issue, but nope. So there's something that doesn't work here.
In case that matters, this machine has:
$ git --version git version 2.39.2
Best regards,
Thomas Petazzoni
Signature · 7 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Hi Thomas
Quoted 61 lines · Thomas Petazzoni
Hello Michael,
Quoted 7 lines · On Tue, 3 Sep 2024 10:57:44 +0200 Michael Trimarchi…
Just a simple clone and pull with --depth 1 should be enough to parse the cve and generate the pkg-stats report.
From a full clone and a depth-1 clone, and the size delta is 2.9GiB vs. 2.2GiB. The download size does change: from 983.55MiB down to 270.78MiB. it's a net time win too: 2m17s vs 1min7s (on a 100Mbps link).
Quoted 5 lines
Signed-off-by Michael Trimarchi <[email protected]>---
V1->V2:
- Add statistics from Yann E. Morin
- Use git pull --depth 1 for update the repoI am sorry, but I had to revert this commit... as it doesn't work:
Updating from https://github.com/fkie-cad/nvd-json-data-feeds/
Traceback (most recent call last):
File "/home/buildroot/buildroot-stats/./support/scripts/pkg-stats", line 1346, in <module>__main__() File "/home/buildroot/buildroot-stats/./support/scripts/pkg-stats", line 1335, in __main__ check_package_cves(args.nvd_path, packages) File "/home/buildroot/buildroot-stats/./support/scripts/pkg-stats", line 660, in check_package_cves for cve in cvecheck.CVE.read_nvd_dir(nvd_path): File "/home/buildroot/buildroot-stats/support/scripts/cve.py", line 105, in read_nvd_dir CVE.download_nvd(nvd_git_dir) File "/home/buildroot/buildroot-stats/support/scripts/cve.py", line 74, in download_nvd subprocess.check_call(File "
/usr/lib/python3.11/subprocess.py", line 413, incheck_call
raiseCalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['git', 'pull', '--depth', '1']' returned non-zero exit status 128.if I go to the machine in question:
buildroot@buildroot:~/nvd/git$ git pull --depth 1 remote: Enumerating objects: 164, done. remote: Counting objects: 100% (140/140), done. remote: Compressing objects: 100% (27/27), done. remote: Total 63 (delta 54), reused 44 (delta 36), pack-reused 0 (from 0) Unpacking objects: 100% (63/63), 18.15 KiB | 26.00 KiB/s, done. From https://github.com/fkie-cad/nvd-json-data-feeds + 51b091a6a4...7b69235976 main -> origin/main (forced update) hint: You have divergent branches and need to specify how to reconcile them. hint: You can do so by running one of the following commands sometime before hint: your next pull: hint: hint: git config pull.rebase false # merge hint: git config pull.rebase true # rebase hint: git config pull.ff only # fast-forward only hint: hint: You can replace "git config" with "git config --global" to set a default hint: preference for all repositories. You can also pass --rebase, --no-rebase, hint: or --ff-only on the command line to override the configured default per hint: invocation. fatal: Need to specify how to reconcile divergent branches.
Quoted 8 lines
I had already done a git reset --hard, and see if it was just a one-time issue, but nope. So there's something that doesn't work here.
In case that matters, this machine has:
$ git --version git version 2.39.2
Yes sorry, I always use tools that are the latest, git version 2.43.0
Make sense. I will create a better setup for me
Michael
Quoted 7 lines
Best regards,
Thomas Petazzoni
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
Signature · 15 lines
Michael Nazzareno Trimarchi Co-Founder & Chief Executive Officer M. +39 347 913 2170 [email protected] __________________________________ Amarula Solutions BV Joop Geesinkweg 125, 1114 AB, Amsterdam, NL T. +31 (0)85 111 9172 [email protected] www.amarulasolutions.com _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot