Skip to contentSkip to conversation

scripts/cve: Avoid to do a complete clone of cve git repository

PATCH Applied Michael Trimarchi 3 participants 9 messages Sep 3, 2024 – Sep 12, 2024

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/2 scripts/cve: Avoid to do a complete clone of cve git repository

Just a simple clone and pull with --depth 1 should be enough to parse the cve and generate the pkg-stats report.

From a full clone and a depth-1 clone, and the size delta is 2.9GiB vs. 2.2GiB. The download size does change: from 983.55MiB down to 270.78MiB. it's a net time win too: 2m17s vs 1min7s (on a 100Mbps link).

---
V1->V2:
- Add statistics from Yann E. Morin
- Use git pull --depth 1 for update the repo

---
 support/scripts/cve.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
support/scripts/cve.py +2 -2
index e25825581e..dcb3a63925 100755
@@ -72,7 +72,7 @@ class CVE:
         print(f"Updating from {NVD_BASE_URL}")
         if os.path.exists(nvd_git_dir):
             subprocess.check_call(
-                ["git", "pull"],
+                ["git", "pull", "--depth", "1"],
                 cwd=nvd_git_dir,
                 stdout=subprocess.DEVNULL,
                 stderr=subprocess.DEVNULL,
@@ -82,7 +82,7 @@ class CVE:
             # happily clones into an empty directory.
             os.makedirs(nvd_git_dir)
             subprocess.check_call(
-                ["git", "clone", NVD_BASE_URL, nvd_git_dir],
+                ["git", "clone", "--depth", "1", NVD_BASE_URL, nvd_git_dir],
                 stdout=subprocess.DEVNULL,
                 stderr=subprocess.DEVNULL,
             )
Signature · 6 lines
2.43.0

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#1 lore raw

PATCH v2 2/2 scripts/cve: Restart the clone if the pull generate an exception

If we are not able to pull from the directory, restart from a clean clone. This can happen for corrupt repository or unfinished download

---
V1->V2: Adjust the commit message

---
 support/scripts/cve.py | 21 ++++++++++++++-------
 1 file changed, 14 insertions(+), 7 deletions(-)
support/scripts/cve.py +14 -7
index dcb3a63925..6cd9aab963 100755
@@ -21,6 +21,7 @@ import datetime
 import os
 import distutils.version
 import json
+import shutil
 import subprocess
 import sys
 import operator
@@ -69,15 +70,21 @@ class CVE:
 
     @staticmethod
     def download_nvd(nvd_git_dir):
+        done = False
         print(f"Updating from {NVD_BASE_URL}")
         if os.path.exists(nvd_git_dir):
-            subprocess.check_call(
-                ["git", "pull", "--depth", "1"],
-                cwd=nvd_git_dir,
-                stdout=subprocess.DEVNULL,
-                stderr=subprocess.DEVNULL,
-            )
-        else:
+            try:
+                subprocess.check_call(
+                    ["git", "pull", "--depth", "1"],
+                    cwd=nvd_git_dir,
+                    stdout=subprocess.DEVNULL,
+                    stderr=subprocess.DEVNULL,
+                )
+                done = True
+            except:
+                shutil.rmtree(nvd_git_dir)
+
+        if (not done):
             # Create the directory and its parents; git
             # happily clones into an empty directory.
             os.makedirs(nvd_git_dir)
Signature · 6 lines
2.43.0

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#2 lore raw

Re: [Buildroot] [PATCH V2 2/2] scripts/cve: Restart the clone if the pull generate an exception

Quoted 26 lines · On Tue, 3 Sep 2024 10:57:45 +0200 Michael Trimarchi…
support/scripts/cve.py +2 -7
index dcb3a63925..6cd9aab963 100755
@@ -21,6 +21,7 @@ import datetime
 import os
 import distutils.version
 import json
+import shutil
 import subprocess
 import sys
 import operator
@@ -69,15 +70,21 @@ class CVE:
 
     @staticmethod
     def download_nvd(nvd_git_dir):
+        done = False
         print(f"Updating from {NVD_BASE_URL}")
         if os.path.exists(nvd_git_dir):
-            subprocess.check_call(
-                ["git", "pull", "--depth", "1"],
-                cwd=nvd_git_dir,
-                stdout=subprocess.DEVNULL,
-                stderr=subprocess.DEVNULL,
-            )
-        else:

support/scripts/cve.py class CVE: · line 85

Quoted 10 lines
support/scripts/cve.py +10 -0
+            try:
+                subprocess.check_call(
+                    ["git", "pull", "--depth", "1"],
+                    cwd=nvd_git_dir,
+                    stdout=subprocess.DEVNULL,
+                    stderr=subprocess.DEVNULL,
+                )
+                done = True
+            except:
+                shutil.rmtree(nvd_git_dir)

The thing I'm worried about is that you can also get a failure of "git pull" for example due to a network timeout or something like that, which doesn't need a full git clone, but just a "try again" later... and now we're going to wipe out the entire local clone, and try to clone everything again. Is that a good idea?

Also, perhaps we need to show an error message if the "git pull" failed, and saying we're falling back to a full clone, or something?

support/scripts/cve.py class CVE: · line 87

Quoted 1 line
support/scripts/cve.py +1 -0
+        if (not done):

if not done:

is sufficient, we are not writing C code here :-)

Thanks!

Thomas

Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#3 lore raw

Re: [Buildroot] [PATCH V2 2/2] scripts/cve: Restart the clone if the pull generate an exception

Hi Thomas

Quoted 50 lines · Thomas Petazzoni
Quoted 26 lines · On Tue, 3 Sep 2024 10:57:45 +0200 Michael Trimarchi…
support/scripts/cve.py +2 -7
index dcb3a63925..6cd9aab963 100755
@@ -21,6 +21,7 @@ import datetime
 import os
 import distutils.version
 import json
+import shutil
 import subprocess
 import sys
 import operator
@@ -69,15 +70,21 @@ class CVE:
 
     @staticmethod
     def download_nvd(nvd_git_dir):
+        done = False
         print(f"Updating from {NVD_BASE_URL}")
         if os.path.exists(nvd_git_dir):
-            subprocess.check_call(
-                ["git", "pull", "--depth", "1"],
-                cwd=nvd_git_dir,
-                stdout=subprocess.DEVNULL,
-                stderr=subprocess.DEVNULL,
-            )
-        else:

support/scripts/cve.py class CVE: · line 85

Quoted 10 lines
support/scripts/cve.py +10 -0
+            try:
+                subprocess.check_call(
+                    ["git", "pull", "--depth", "1"],
+                    cwd=nvd_git_dir,
+                    stdout=subprocess.DEVNULL,
+                    stderr=subprocess.DEVNULL,
+                )
+                done = True
+            except:
+                shutil.rmtree(nvd_git_dir)

The thing I'm worried about is that you can also get a failure of "git pull" for example due to a network timeout or something like that, which doesn't need a full git clone, but just a "try again" later... and now we're going to wipe out the entire local clone, and try to clone everything again. Is that a good idea?

Also, perhaps we need to show an error message if the "git pull" failed, and saying we're falling back to a full clone, or something?

Quoted 6 lines

support/scripts/cve.py class CVE: · line 87

Quoted 1 line
support/scripts/cve.py +1 -0
+        if (not done):

if not done:

is sufficient, we are not writing C code here :-)

Sorry, I will ask my colleague to write python and send a better strategy. I totally agree with you.

Anyway I have a plan to add more information on cve reporting, hope that you like the idea.
Mostly I'm playing with parser for jenkins and love to add buildroot too but I don't have enough
information to show

https://github.com/jenkinsci/analysis-model/pull/1085

Michael

Quoted 8 lines

Thanks!

Thomas
--
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering and training
https://bootlin.com

Signature · 15 lines
Michael Nazzareno Trimarchi
Co-Founder & Chief Executive Officer
M. +39 347 913 2170
[email protected]
__________________________________

Amarula Solutions BV
Joop Geesinkweg 125, 1114 AB, Amsterdam, NL
T. +31 (0)85 111 9172
[email protected]
www.amarulasolutions.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#4 lore raw
In reply to Michael Trimarchi
Quoted 7 lines · On Tue, 3 Sep 2024 10:57:44 +0200 Michael Trimarchi…

Just a simple clone and pull with --depth 1 should be enough to parse the cve and generate the pkg-stats report.

From a full clone and a depth-1 clone, and the size delta is 2.9GiB vs. 2.2GiB. The download size does change: from 983.55MiB down to 270.78MiB. it's a net time win too: 2m17s vs 1min7s (on a 100Mbps link).

Quoted 5 lines

---
V1->V2:
- Add statistics from Yann E. Morin
- Use git pull --depth 1 for update the repo

Applied to next, thanks!

Thomas

Signature · 7 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#5 lore raw

Michael, All,

On 2024-09-03 20:50 +0200, Thomas Petazzoni via buildroot spake thusly:

Quoted 10 lines
Quoted 7 lines · On Tue, 3 Sep 2024 10:57:44 +0200 Michael Trimarchi…

Just a simple clone and pull with --depth 1 should be enough to parse the cve and generate the pkg-stats report.

From a full clone and a depth-1 clone, and the size delta is 2.9GiB vs. 2.2GiB. The download size does change: from 983.55MiB down to 270.78MiB. it's a net time win too: 2m17s vs 1min7s (on a 100Mbps link).

Quoted 7 lines
Quoted 5 lines

---
V1->V2:
- Add statistics from Yann E. Morin
- Use git pull --depth 1 for update the repo

Applied to next, thanks!

I don't understand: Michael said in the first iteration that we should drop the package:
https://lore.kernel.org/buildroot/CAOf5uw=m4LOk97OT1dTP=2-uP6QZ1WQyHfKYSQmWCirDaxXvgQ@mail.gmail.com/

Only the first clone is slow, the following calls will just pull (mostly nothing most of the time), so the optimisation is not really worth it.

Also, in the download backend for git, we stopped doing shallow clone because they were causing issues (but might not be applicable here).

Anyway, too late, that's been applied...

Regards,
Yann E. MORIN.

Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#6 lore raw

Hi Yann

Quoted 26 lines · Yann E. MORIN

Michael, All,

On 2024-09-03 20:50 +0200, Thomas Petazzoni via buildroot spake thusly:

Quoted 10 lines
Quoted 7 lines · On Tue, 3 Sep 2024 10:57:44 +0200 Michael Trimarchi…

Just a simple clone and pull with --depth 1 should be enough to parse the cve and generate the pkg-stats report.

From a full clone and a depth-1 clone, and the size delta is 2.9GiB vs. 2.2GiB. The download size does change: from 983.55MiB down to 270.78MiB. it's a net time win too: 2m17s vs 1min7s (on a 100Mbps link).

Quoted 7 lines
Quoted 5 lines

---
V1->V2:
- Add statistics from Yann E. Morin
- Use git pull --depth 1 for update the repo

Applied to next, thanks!

I don't understand: Michael said in the first iteration that we should drop the package:
https://lore.kernel.org/buildroot/CAOf5uw=m4LOk97OT1dTP=2-uP6QZ1WQyHfKYSQmWCirDaxXvgQ@mail.gmail.com/

Quoted 6 lines

Only the first clone is slow, the following calls will just pull (mostly nothing most of the time), so the optimisation is not really worth it.

Also, in the download backend for git, we stopped doing shallow clone because they were causing issues (but might not be applicable here).

I have reposted it, fixing the pull. Yann if you really don't like it, people can drop, revert

Michael

Quoted 12 lines

Anyway, too late, that's been applied...

Regards,
Yann E. MORIN.

--
.-----------------.--------------------.------------------.--------------------.

|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |

'------------------------------^-------^------------------^--------------------'

Signature · 15 lines
Michael Nazzareno Trimarchi
Co-Founder & Chief Executive Officer
M. +39 347 913 2170
[email protected]
__________________________________

Amarula Solutions BV
Joop Geesinkweg 125, 1114 AB, Amsterdam, NL
T. +31 (0)85 111 9172
[email protected]
www.amarulasolutions.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#7 lore raw
In reply to Michael Trimarchi

Hello Michael,

Quoted 7 lines · On Tue, 3 Sep 2024 10:57:44 +0200 Michael Trimarchi…

Just a simple clone and pull with --depth 1 should be enough to parse the cve and generate the pkg-stats report.

From a full clone and a depth-1 clone, and the size delta is 2.9GiB vs. 2.2GiB. The download size does change: from 983.55MiB down to 270.78MiB. it's a net time win too: 2m17s vs 1min7s (on a 100Mbps link).

Quoted 5 lines

---
V1->V2:
- Add statistics from Yann E. Morin
- Use git pull --depth 1 for update the repo

I am sorry, but I had to revert this commit... as it doesn't work:

Updating from https://github.com/fkie-cad/nvd-json-data-feeds/
Traceback (most recent call last):
File "/home/buildroot/buildroot-stats/./support/scripts/pkg-stats", line 1346, in <module>

    __main__()
  File "/home/buildroot/buildroot-stats/./support/scripts/pkg-stats", line 1335, in __main__
    check_package_cves(args.nvd_path, packages)
  File "/home/buildroot/buildroot-stats/./support/scripts/pkg-stats", line 660, in check_package_cves
    for cve in cvecheck.CVE.read_nvd_dir(nvd_path):
  File "/home/buildroot/buildroot-stats/support/scripts/cve.py", line 105, in read_nvd_dir
    CVE.download_nvd(nvd_git_dir)
  File "/home/buildroot/buildroot-stats/support/scripts/cve.py", line 74, in download_nvd
    subprocess.check_call(

File "/usr/lib/python3.11/subprocess.py", line 413, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['git', 'pull', '--depth', '1']' returned non-zero exit status 128.

if I go to the machine in question:

buildroot@buildroot:~/nvd/git$ git pull --depth 1
remote: Enumerating objects: 164, done.
remote: Counting objects: 100% (140/140), done.
remote: Compressing objects: 100% (27/27), done.
remote: Total 63 (delta 54), reused 44 (delta 36), pack-reused 0 (from 0)
Unpacking objects: 100% (63/63), 18.15 KiB | 26.00 KiB/s, done.
From https://github.com/fkie-cad/nvd-json-data-feeds
 + 51b091a6a4...7b69235976 main       -> origin/main  (forced update)
hint: You have divergent branches and need to specify how to reconcile them.
hint: You can do so by running one of the following commands sometime before
hint: your next pull:
hint: 
hint:   git config pull.rebase false  # merge
hint:   git config pull.rebase true   # rebase
hint:   git config pull.ff only       # fast-forward only
hint: 
hint: You can replace "git config" with "git config --global" to set a default
hint: preference for all repositories. You can also pass --rebase, --no-rebase,
hint: or --ff-only on the command line to override the configured default per
hint: invocation.
fatal: Need to specify how to reconcile divergent branches.

I had already done a git reset --hard, and see if it was just a one-time issue, but nope. So there's something that doesn't work here.

In case that matters, this machine has:

$ git --version
git version 2.39.2

Best regards,

Thomas Petazzoni

Signature · 7 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#8 lore raw

Hi Thomas

Quoted 61 lines · Thomas Petazzoni

Hello Michael,

Quoted 7 lines · On Tue, 3 Sep 2024 10:57:44 +0200 Michael Trimarchi…

Just a simple clone and pull with --depth 1 should be enough to parse the cve and generate the pkg-stats report.

From a full clone and a depth-1 clone, and the size delta is 2.9GiB vs. 2.2GiB. The download size does change: from 983.55MiB down to 270.78MiB. it's a net time win too: 2m17s vs 1min7s (on a 100Mbps link).

Quoted 5 lines

---
V1->V2:
- Add statistics from Yann E. Morin
- Use git pull --depth 1 for update the repo

I am sorry, but I had to revert this commit... as it doesn't work:

Updating from https://github.com/fkie-cad/nvd-json-data-feeds/
Traceback (most recent call last):
File "/home/buildroot/buildroot-stats/./support/scripts/pkg-stats", line 1346, in <module>

    __main__()
  File "/home/buildroot/buildroot-stats/./support/scripts/pkg-stats", line 1335, in __main__
    check_package_cves(args.nvd_path, packages)
  File "/home/buildroot/buildroot-stats/./support/scripts/pkg-stats", line 660, in check_package_cves
    for cve in cvecheck.CVE.read_nvd_dir(nvd_path):
  File "/home/buildroot/buildroot-stats/support/scripts/cve.py", line 105, in read_nvd_dir
    CVE.download_nvd(nvd_git_dir)
  File "/home/buildroot/buildroot-stats/support/scripts/cve.py", line 74, in download_nvd
    subprocess.check_call(

File "/usr/lib/python3.11/subprocess.py", line 413, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['git', 'pull', '--depth', '1']' returned non-zero exit status 128.

if I go to the machine in question:

buildroot@buildroot:~/nvd/git$ git pull --depth 1
remote: Enumerating objects: 164, done.
remote: Counting objects: 100% (140/140), done.
remote: Compressing objects: 100% (27/27), done.
remote: Total 63 (delta 54), reused 44 (delta 36), pack-reused 0 (from 0)
Unpacking objects: 100% (63/63), 18.15 KiB | 26.00 KiB/s, done.
From https://github.com/fkie-cad/nvd-json-data-feeds
 + 51b091a6a4...7b69235976 main       -> origin/main  (forced update)
hint: You have divergent branches and need to specify how to reconcile them.
hint: You can do so by running one of the following commands sometime before
hint: your next pull:
hint:
hint:   git config pull.rebase false  # merge
hint:   git config pull.rebase true   # rebase
hint:   git config pull.ff only       # fast-forward only
hint:
hint: You can replace "git config" with "git config --global" to set a default
hint: preference for all repositories. You can also pass --rebase, --no-rebase,
hint: or --ff-only on the command line to override the configured default per
hint: invocation.
fatal: Need to specify how to reconcile divergent branches.
Quoted 8 lines

I had already done a git reset --hard, and see if it was just a one-time issue, but nope. So there's something that doesn't work here.

In case that matters, this machine has:

$ git --version
git version 2.39.2

Yes sorry, I always use tools that are the latest, git version 2.43.0

Make sense. I will create a better setup for me

Michael

Quoted 7 lines

Best regards,

Thomas Petazzoni
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Signature · 15 lines
Michael Nazzareno Trimarchi
Co-Founder & Chief Executive Officer
M. +39 347 913 2170
[email protected]
__________________________________

Amarula Solutions BV
Joop Geesinkweg 125, 1114 AB, Amsterdam, NL
T. +31 (0)85 111 9172
[email protected]
www.amarulasolutions.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot