scripts/cve: Avoid to do a complete clone of cve git repository
PATCH 1/2 scripts/cve: Avoid to do a complete clone of cve git repository
Just a simple clone --depth 1 should be enough to parse the cve and generate the pkg-stats report. The space now does not take in account of the full history and only use 2.2Gb
--- support/scripts/cve.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
support/scripts/cve.py +1 -1
@@ -82,7 +82,7 @@ class CVE: # happily clones into an empty directory. os.makedirs(nvd_git_dir) subprocess.check_call( - ["git", "clone", NVD_BASE_URL, nvd_git_dir], + ["git", "clone", "--depth", "1", NVD_BASE_URL, nvd_git_dir], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, )
Signature · 6 lines
2.43.0 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH 2/2 scripts/cve: Restart the clone if the pull generate an exception
If we are not able to pull from the directory, restart from a clean clone
--- support/scripts/cve.py | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-)
support/scripts/cve.py +14 -7
@@ -21,6 +21,7 @@ import datetime import os import distutils.version import json +import shutil import subprocess import sys import operator
@@ -69,15 +70,21 @@ class CVE: @staticmethod def download_nvd(nvd_git_dir): + done = False print(f"Updating from {NVD_BASE_URL}") if os.path.exists(nvd_git_dir): - subprocess.check_call( - ["git", "pull"], - cwd=nvd_git_dir, - stdout=subprocess.DEVNULL, - stderr=subprocess.DEVNULL, - ) - else: + try: + subprocess.check_call( + ["git", "pull"], + cwd=nvd_git_dir, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + done = True + except: + shutil.rmtree(nvd_git_dir) + + if (not done): # Create the directory and its parents; git # happily clones into an empty directory. os.makedirs(nvd_git_dir)
Signature · 6 lines
2.43.0 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH 2/2] scripts/cve: Restart the clone if the pull generate an exception
Michael, All,
On 2024-09-02 09:41 +0200, Michael Trimarchi spake thusly:
Quoted 2 lines
If we are not able to pull from the directory, restart from a clean clone
In what situations would such a problem arise?
Is it due to a shallow clone as proposed in the first patch? If so, then that's a good reason not to do a shallow clone.
Regards,
Yann E. MORIN.
Quoted 53 lines
Signed-off-by Michael Trimarchi <[email protected]>--- support/scripts/cve.py | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-)support/scripts/cve.py +14
-7@@ -21,6 +21,7 @@ import datetime import os import distutils.version import json +import shutil import subprocess import sys import operator@@ -69,15 +70,21 @@ class CVE: @staticmethod def download_nvd(nvd_git_dir): + done = False print(f"Updating from {NVD_BASE_URL}") if os.path.exists(nvd_git_dir): - subprocess.check_call( - ["git", "pull"], - cwd=nvd_git_dir, - stdout=subprocess.DEVNULL, - stderr=subprocess.DEVNULL, - ) - else: + try: + subprocess.check_call( + ["git", "pull"], + cwd=nvd_git_dir, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + done = True + except: + shutil.rmtree(nvd_git_dir) + + if (not done): # Create the directory and its parents; git # happily clones into an empty directory. os.makedirs(nvd_git_dir)Signature · 6 lines
2.43.0 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH 2/2] scripts/cve: Restart the clone if the pull generate an exception
Hi Yann
Quoted 12 lines · Yann E. MORIN
Michael, All,
On 2024-09-02 09:41 +0200, Michael Trimarchi spake thusly:
Quoted 2 lines
If we are not able to pull from the directory, restart from a clean clone
In what situations would such a problem arise?
Is it due to a shallow clone as proposed in the first patch? If so, then that's a good reason not to do a shallow clone.
You test if the dir exists but if you cancel the git clone without terminating it, you can not continue.
It's happen always and does not depend on the previous patch
Michael
Quoted 64 lines
Regards,
Yann E. MORIN.Quoted 49 lines
Signed-off-by Michael Trimarchi <[email protected]>--- support/scripts/cve.py | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-)support/scripts/cve.py +14
-8@@ -21,6 +21,7 @@ import datetime import os import distutils.version import json +import shutil import subprocess import sys import operator@@ -69,15 +70,21 @@ class CVE: @staticmethod def download_nvd(nvd_git_dir): + done = False print(f"Updating from {NVD_BASE_URL}") if os.path.exists(nvd_git_dir): - subprocess.check_call( - ["git", "pull"], - cwd=nvd_git_dir, - stdout=subprocess.DEVNULL, - stderr=subprocess.DEVNULL, - ) - else: + try: + subprocess.check_call( + ["git", "pull"], + cwd=nvd_git_dir, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + done = True + except: + shutil.rmtree(nvd_git_dir) + + if (not done): # Create the directory and its parents; git # happily clones into an empty directory. os.makedirs(nvd_git_dir) --2.43.0
Quoted 4 lines
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot--
.-----------------.--------------------.------------------.--------------------.| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |'------------------------------^-------^------------------^--------------------'
Signature · 15 lines
Michael Nazzareno Trimarchi Co-Founder & Chief Executive Officer M. +39 347 913 2170 [email protected] __________________________________ Amarula Solutions BV Joop Geesinkweg 125, 1114 AB, Amsterdam, NL T. +31 (0)85 111 9172 [email protected] www.amarulasolutions.com _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Michael, All,
On 2024-09-02 09:41 +0200, Michael Trimarchi spake thusly:
Quoted 3 lines
Just a simple clone --depth 1 should be enough to parse the cve and generate the pkg-stats report. The space now does not take in account of the full history and only use 2.2Gb
I just did both a full clone and a depth-1 clone, and the size delta is not that much: 2.9GiB vs. 2.2GiB, so we're still in the same league.
Still, the download size does change: from 983.55MiB down to 270.78MiB; the remote has to compress objects specifically for the depth-1 clone, which takes a bit of time, but there less deltas to resolve locally, so it's a net time win too: 2m17s vs 1min7s (on a 100Mbps link).
Is it eventually worth it? I like things to be simple rather than have complex heuristics, and storage nowadays is cheap, and bandwidth is only really paid on the first clone, since the following calls would only pull what's missing, which will eventually cause the local clone to grow to a size that would not differ by much if the first clone had been a full one.
So, I 'm a bit comnficted about that one, but would tend to say no... Leaving pending for other maintainers to chime in.
Regards,
Yann E. MORIN.
Quoted 25 lines
Signed-off-by Michael Trimarchi <[email protected]>--- support/scripts/cve.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)support/scripts/cve.py +1
-1@@ -82,7 +82,7 @@ class CVE: # happily clones into an empty directory. os.makedirs(nvd_git_dir) subprocess.check_call( - ["git", "clone", NVD_BASE_URL, nvd_git_dir], + ["git", "clone", "--depth", "1", NVD_BASE_URL, nvd_git_dir], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, )Signature · 6 lines
2.43.0 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
HI Yann
Quoted 8 lines · Yann E. MORIN
Michael, All,
On 2024-09-02 09:41 +0200, Michael Trimarchi spake thusly:
Quoted 3 lines
Just a simple clone --depth 1 should be enough to parse the cve and generate the pkg-stats report. The space now does not take in account of the full history and only use 2.2Gb
Quoted 8 lines
I just did both a full clone and a depth-1 clone, and the size delta is not that much: 2.9GiB vs. 2.2GiB, so we're still in the same league.
Still, the download size does change: from 983.55MiB down to 270.78MiB; the remote has to compress objects specifically for the depth-1 clone, which takes a bit of time, but there less deltas to resolve locally, so it's a net time win too: 2m17s vs 1min7s (on a 100Mbps link).
If you have a dsl and download link of 10Mbps this makes really sense and this just tend
to increase in size. You 1 minutes can be ten 10 minutes on network congestion
Michael
Quoted 46 lines
Is it eventually worth it? I like things to be simple rather than have complex heuristics, and storage nowadays is cheap, and bandwidth is only really paid on the first clone, since the following calls would only pull what's missing, which will eventually cause the local clone to grow to a size that would not differ by much if the first clone had been a full one.
So, I 'm a bit comnficted about that one, but would tend to say no... Leaving pending for other maintainers to chime in.
Regards,
Yann E. MORIN.Quoted 21 lines
Signed-off-by Michael Trimarchi <[email protected]>--- support/scripts/cve.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)support/scripts/cve.py +1
-2@@ -82,7 +82,7 @@ class CVE: # happily clones into an empty directory. os.makedirs(nvd_git_dir) subprocess.check_call( - ["git", "clone", NVD_BASE_URL, nvd_git_dir], + ["git", "clone", "--depth", "1", NVD_BASE_URL, nvd_git_dir], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, ) --2.43.0
Quoted 4 lines
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot--
.-----------------.--------------------.------------------.--------------------.| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |'------------------------------^-------^------------------^--------------------'
Signature · 15 lines
Michael Nazzareno Trimarchi Co-Founder & Chief Executive Officer M. +39 347 913 2170 [email protected] __________________________________ Amarula Solutions BV Joop Geesinkweg 125, 1114 AB, Amsterdam, NL T. +31 (0)85 111 9172 [email protected] www.amarulasolutions.com _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Hi Yann
Quoted 21 lines · Michael Nazzareno Trimarchi
HI Yann
Quoted 8 lines · Yann E. MORIN
Michael, All,
On 2024-09-02 09:41 +0200, Michael Trimarchi spake thusly:
Quoted 3 lines
Just a simple clone --depth 1 should be enough to parse the cve and generate the pkg-stats report. The space now does not take in account of the full history and only use 2.2Gb
Quoted 8 lines
I just did both a full clone and a depth-1 clone, and the size delta is not that much: 2.9GiB vs. 2.2GiB, so we're still in the same league.
Still, the download size does change: from 983.55MiB down to 270.78MiB; the remote has to compress objects specifically for the depth-1 clone, which takes a bit of time, but there less deltas to resolve locally, so it's a net time win too: 2m17s vs 1min7s (on a 100Mbps link).
Quoted 4 lines
If you have a dsl and download link of 10Mbps this makes really sense and this just tend
to increase in size. You 1 minutes can be ten 10 minutes on network congestion
I think that the real reason could that the next pull will be problematic. Yes, you can drop this patch
Anyway the second one does not depend on it
Michael
Quoted 63 lines
Michael
Quoted 41 lines
Is it eventually worth it? I like things to be simple rather than have complex heuristics, and storage nowadays is cheap, and bandwidth is only really paid on the first clone, since the following calls would only pull what's missing, which will eventually cause the local clone to grow to a size that would not differ by much if the first clone had been a full one.
So, I 'm a bit comnficted about that one, but would tend to say no... Leaving pending for other maintainers to chime in.
Regards,
Yann E. MORIN.Quoted 21 lines
Signed-off-by Michael Trimarchi <[email protected]>--- support/scripts/cve.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)support/scripts/cve.py +1
-2@@ -82,7 +82,7 @@ class CVE: # happily clones into an empty directory. os.makedirs(nvd_git_dir) subprocess.check_call( - ["git", "clone", NVD_BASE_URL, nvd_git_dir], + ["git", "clone", "--depth", "1", NVD_BASE_URL, nvd_git_dir], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, ) --2.43.0
Quoted 4 lines
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot--
.-----------------.--------------------.------------------.--------------------.Quoted 5 lines
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |'------------------------------^-------^------------------^--------------------'
--
Michael Nazzareno Trimarchi
Co-Founder & Chief Executive Officer
M. +39 347 913 2170
[email protected]
__________________________________Amarula Solutions BV
Joop Geesinkweg 125, 1114 AB, Amsterdam, NL
T. +31 (0)85 111 9172
[email protected]
www.amarulasolutions.com
Signature · 15 lines
Michael Nazzareno Trimarchi Co-Founder & Chief Executive Officer M. +39 347 913 2170 [email protected] __________________________________ Amarula Solutions BV Joop Geesinkweg 125, 1114 AB, Amsterdam, NL T. +31 (0)85 111 9172 [email protected] www.amarulasolutions.com _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot