Skip to contentSkip to conversation

scripts/cve: Avoid to do a complete clone of cve git repository

PATCH Superseded Michael Trimarchi 2 participants 7 messages Sep 2, 2024

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 scripts/cve: Avoid to do a complete clone of cve git repository

Just a simple clone --depth 1 should be enough to parse the cve and generate the pkg-stats report. The space now does not take in account of the full history and only use 2.2Gb

---
 support/scripts/cve.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
support/scripts/cve.py +1 -1
index e25825581e..7d115a28b5 100755
@@ -82,7 +82,7 @@ class CVE:
             # happily clones into an empty directory.
             os.makedirs(nvd_git_dir)
             subprocess.check_call(
-                ["git", "clone", NVD_BASE_URL, nvd_git_dir],
+                ["git", "clone", "--depth", "1", NVD_BASE_URL, nvd_git_dir],
                 stdout=subprocess.DEVNULL,
                 stderr=subprocess.DEVNULL,
             )
Signature · 6 lines
2.43.0

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#1 lore raw

PATCH 2/2 scripts/cve: Restart the clone if the pull generate an exception

If we are not able to pull from the directory, restart from a clean clone

---
 support/scripts/cve.py | 21 ++++++++++++++-------
 1 file changed, 14 insertions(+), 7 deletions(-)
support/scripts/cve.py +14 -7
index 7d115a28b5..b205b72e65 100755
@@ -21,6 +21,7 @@ import datetime
 import os
 import distutils.version
 import json
+import shutil
 import subprocess
 import sys
 import operator
@@ -69,15 +70,21 @@ class CVE:
 
     @staticmethod
     def download_nvd(nvd_git_dir):
+        done = False
         print(f"Updating from {NVD_BASE_URL}")
         if os.path.exists(nvd_git_dir):
-            subprocess.check_call(
-                ["git", "pull"],
-                cwd=nvd_git_dir,
-                stdout=subprocess.DEVNULL,
-                stderr=subprocess.DEVNULL,
-            )
-        else:
+            try:
+                subprocess.check_call(
+                    ["git", "pull"],
+                    cwd=nvd_git_dir,
+                    stdout=subprocess.DEVNULL,
+                    stderr=subprocess.DEVNULL,
+                )
+                done = True
+            except:
+                shutil.rmtree(nvd_git_dir)
+
+        if (not done):
             # Create the directory and its parents; git
             # happily clones into an empty directory.
             os.makedirs(nvd_git_dir)
Signature · 6 lines
2.43.0

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#2 lore raw

Re: [Buildroot] [PATCH 2/2] scripts/cve: Restart the clone if the pull generate an exception

Michael, All,

On 2024-09-02 09:41 +0200, Michael Trimarchi spake thusly:

Quoted 2 lines

If we are not able to pull from the directory, restart from a clean clone

In what situations would such a problem arise?

Is it due to a shallow clone as proposed in the first patch? If so, then that's a good reason not to do a shallow clone.

Regards,
Yann E. MORIN.

Quoted 53 lines
---
 support/scripts/cve.py | 21 ++++++++++++++-------
 1 file changed, 14 insertions(+), 7 deletions(-)
support/scripts/cve.py +14 -7
index 7d115a28b5..b205b72e65 100755
@@ -21,6 +21,7 @@ import datetime
 import os
 import distutils.version
 import json
+import shutil
 import subprocess
 import sys
 import operator
@@ -69,15 +70,21 @@ class CVE:
 
     @staticmethod
     def download_nvd(nvd_git_dir):
+        done = False
         print(f"Updating from {NVD_BASE_URL}")
         if os.path.exists(nvd_git_dir):
-            subprocess.check_call(
-                ["git", "pull"],
-                cwd=nvd_git_dir,
-                stdout=subprocess.DEVNULL,
-                stderr=subprocess.DEVNULL,
-            )
-        else:
+            try:
+                subprocess.check_call(
+                    ["git", "pull"],
+                    cwd=nvd_git_dir,
+                    stdout=subprocess.DEVNULL,
+                    stderr=subprocess.DEVNULL,
+                )
+                done = True
+            except:
+                shutil.rmtree(nvd_git_dir)
+
+        if (not done):
             # Create the directory and its parents; git
             # happily clones into an empty directory.
             os.makedirs(nvd_git_dir)
Signature · 6 lines
2.43.0

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#3 lore raw

Re: [Buildroot] [PATCH 2/2] scripts/cve: Restart the clone if the pull generate an exception

Hi Yann

Quoted 12 lines · Yann E. MORIN

Michael, All,

On 2024-09-02 09:41 +0200, Michael Trimarchi spake thusly:

Quoted 2 lines

If we are not able to pull from the directory, restart from a clean clone

In what situations would such a problem arise?

Is it due to a shallow clone as proposed in the first patch? If so, then that's a good reason not to do a shallow clone.

You test if the dir exists but if you cancel the git clone without terminating it, you can not continue.
It's happen always and does not depend on the previous patch

Michael

Quoted 64 lines

Regards,
Yann E. MORIN.

Quoted 49 lines
---
 support/scripts/cve.py | 21 ++++++++++++++-------
 1 file changed, 14 insertions(+), 7 deletions(-)
support/scripts/cve.py +14 -8
index 7d115a28b5..b205b72e65 100755
@@ -21,6 +21,7 @@ import datetime
 import os
 import distutils.version
 import json
+import shutil
 import subprocess
 import sys
 import operator
@@ -69,15 +70,21 @@ class CVE:
 
     @staticmethod
     def download_nvd(nvd_git_dir):
+        done = False
         print(f"Updating from {NVD_BASE_URL}")
         if os.path.exists(nvd_git_dir):
-            subprocess.check_call(
-                ["git", "pull"],
-                cwd=nvd_git_dir,
-                stdout=subprocess.DEVNULL,
-                stderr=subprocess.DEVNULL,
-            )
-        else:
+            try:
+                subprocess.check_call(
+                    ["git", "pull"],
+                    cwd=nvd_git_dir,
+                    stdout=subprocess.DEVNULL,
+                    stderr=subprocess.DEVNULL,
+                )
+                done = True
+            except:
+                shutil.rmtree(nvd_git_dir)
+
+        if (not done):
             # Create the directory and its parents; git
             # happily clones into an empty directory.
             os.makedirs(nvd_git_dir)
--

2.43.0

Quoted 4 lines

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

--
.-----------------.--------------------.------------------.--------------------.

|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |

'------------------------------^-------^------------------^--------------------'

Signature · 15 lines
Michael Nazzareno Trimarchi
Co-Founder & Chief Executive Officer
M. +39 347 913 2170
[email protected]
__________________________________

Amarula Solutions BV
Joop Geesinkweg 125, 1114 AB, Amsterdam, NL
T. +31 (0)85 111 9172
[email protected]
www.amarulasolutions.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#4 lore raw
In reply to Michael Trimarchi

Michael, All,

On 2024-09-02 09:41 +0200, Michael Trimarchi spake thusly:

Quoted 3 lines

Just a simple clone --depth 1 should be enough to parse the cve and generate the pkg-stats report. The space now does not take in account of the full history and only use 2.2Gb

I just did both a full clone and a depth-1 clone, and the size delta is not that much: 2.9GiB vs. 2.2GiB, so we're still in the same league.

Still, the download size does change: from 983.55MiB down to 270.78MiB; the remote has to compress objects specifically for the depth-1 clone, which takes a bit of time, but there less deltas to resolve locally, so it's a net time win too: 2m17s vs 1min7s (on a 100Mbps link).

Is it eventually worth it? I like things to be simple rather than have complex heuristics, and storage nowadays is cheap, and bandwidth is only really paid on the first clone, since the following calls would only pull what's missing, which will eventually cause the local clone to grow to a size that would not differ by much if the first clone had been a full one.

So, I 'm a bit comnficted about that one, but would tend to say no... Leaving pending for other maintainers to chime in.

Regards,
Yann E. MORIN.

Quoted 25 lines
---
 support/scripts/cve.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
support/scripts/cve.py +1 -1
index e25825581e..7d115a28b5 100755
@@ -82,7 +82,7 @@ class CVE:
             # happily clones into an empty directory.
             os.makedirs(nvd_git_dir)
             subprocess.check_call(
-                ["git", "clone", NVD_BASE_URL, nvd_git_dir],
+                ["git", "clone", "--depth", "1", NVD_BASE_URL, nvd_git_dir],
                 stdout=subprocess.DEVNULL,
                 stderr=subprocess.DEVNULL,
             )
Signature · 6 lines
2.43.0

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#5 lore raw

HI Yann

Quoted 8 lines · Yann E. MORIN

Michael, All,

On 2024-09-02 09:41 +0200, Michael Trimarchi spake thusly:

Quoted 3 lines

Just a simple clone --depth 1 should be enough to parse the cve and generate the pkg-stats report. The space now does not take in account of the full history and only use 2.2Gb

Quoted 8 lines

I just did both a full clone and a depth-1 clone, and the size delta is not that much: 2.9GiB vs. 2.2GiB, so we're still in the same league.

Still, the download size does change: from 983.55MiB down to 270.78MiB; the remote has to compress objects specifically for the depth-1 clone, which takes a bit of time, but there less deltas to resolve locally, so it's a net time win too: 2m17s vs 1min7s (on a 100Mbps link).

If you have a dsl and download link of 10Mbps this makes really sense and this just tend
to increase in size. You 1 minutes can be ten 10 minutes on network congestion

Michael

Quoted 46 lines

Is it eventually worth it? I like things to be simple rather than have complex heuristics, and storage nowadays is cheap, and bandwidth is only really paid on the first clone, since the following calls would only pull what's missing, which will eventually cause the local clone to grow to a size that would not differ by much if the first clone had been a full one.

So, I 'm a bit comnficted about that one, but would tend to say no... Leaving pending for other maintainers to chime in.

Regards,
Yann E. MORIN.

Quoted 21 lines
---
 support/scripts/cve.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
support/scripts/cve.py +1 -2
index e25825581e..7d115a28b5 100755
@@ -82,7 +82,7 @@ class CVE:
             # happily clones into an empty directory.
             os.makedirs(nvd_git_dir)
             subprocess.check_call(
-                ["git", "clone", NVD_BASE_URL, nvd_git_dir],
+                ["git", "clone", "--depth", "1", NVD_BASE_URL, nvd_git_dir],
                 stdout=subprocess.DEVNULL,
                 stderr=subprocess.DEVNULL,
             )
--

2.43.0

Quoted 4 lines

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

--
.-----------------.--------------------.------------------.--------------------.

|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |

'------------------------------^-------^------------------^--------------------'

Signature · 15 lines
Michael Nazzareno Trimarchi
Co-Founder & Chief Executive Officer
M. +39 347 913 2170
[email protected]
__________________________________

Amarula Solutions BV
Joop Geesinkweg 125, 1114 AB, Amsterdam, NL
T. +31 (0)85 111 9172
[email protected]
www.amarulasolutions.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#6 lore raw

Hi Yann

Quoted 21 lines · Michael Nazzareno Trimarchi

HI Yann

Quoted 8 lines · Yann E. MORIN

Michael, All,

On 2024-09-02 09:41 +0200, Michael Trimarchi spake thusly:

Quoted 3 lines

Just a simple clone --depth 1 should be enough to parse the cve and generate the pkg-stats report. The space now does not take in account of the full history and only use 2.2Gb

Quoted 8 lines

I just did both a full clone and a depth-1 clone, and the size delta is not that much: 2.9GiB vs. 2.2GiB, so we're still in the same league.

Still, the download size does change: from 983.55MiB down to 270.78MiB; the remote has to compress objects specifically for the depth-1 clone, which takes a bit of time, but there less deltas to resolve locally, so it's a net time win too: 2m17s vs 1min7s (on a 100Mbps link).

Quoted 4 lines

If you have a dsl and download link of 10Mbps this makes really sense and this just tend
to increase in size. You 1 minutes can be ten 10 minutes on network congestion

I think that the real reason could that the next pull will be problematic. Yes, you can drop this patch

Anyway the second one does not depend on it

Michael

Quoted 63 lines

Michael

Quoted 41 lines

Is it eventually worth it? I like things to be simple rather than have complex heuristics, and storage nowadays is cheap, and bandwidth is only really paid on the first clone, since the following calls would only pull what's missing, which will eventually cause the local clone to grow to a size that would not differ by much if the first clone had been a full one.

So, I 'm a bit comnficted about that one, but would tend to say no... Leaving pending for other maintainers to chime in.

Regards,
Yann E. MORIN.

Quoted 21 lines
---
 support/scripts/cve.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
support/scripts/cve.py +1 -2
index e25825581e..7d115a28b5 100755
@@ -82,7 +82,7 @@ class CVE:
             # happily clones into an empty directory.
             os.makedirs(nvd_git_dir)
             subprocess.check_call(
-                ["git", "clone", NVD_BASE_URL, nvd_git_dir],
+                ["git", "clone", "--depth", "1", NVD_BASE_URL, nvd_git_dir],
                 stdout=subprocess.DEVNULL,
                 stderr=subprocess.DEVNULL,
             )
--

2.43.0

Quoted 4 lines

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

--
.-----------------.--------------------.------------------.--------------------.

Quoted 5 lines
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |

'------------------------------^-------^------------------^--------------------'

--
Michael Nazzareno Trimarchi
Co-Founder & Chief Executive Officer
M. +39 347 913 2170
[email protected]
__________________________________

Amarula Solutions BV
Joop Geesinkweg 125, 1114 AB, Amsterdam, NL
T. +31 (0)85 111 9172
[email protected]
www.amarulasolutions.com

Signature · 15 lines
Michael Nazzareno Trimarchi
Co-Founder & Chief Executive Officer
M. +39 347 913 2170
[email protected]
__________________________________

Amarula Solutions BV
Joop Geesinkweg 125, 1114 AB, Amsterdam, NL
T. +31 (0)85 111 9172
[email protected]
www.amarulasolutions.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot