cve-update-db: Use std library instead of urllib3

From: Pierre Le Magourou <[email protected]>

urllib3 was used in this recipe but it was not set as a dependency. As it is not specifically needed, rewrite the recipe with urllib from the standard library.

Signed-off-by Pierre Le Magourou <[email protected]>

---
  meta/recipes-core/meta/cve-update-db.bb | 10 ++++------
  1 file changed, 4 insertions(+), 6 deletions(-)

meta/recipes-core/meta/cve-update-db.bb +4 -6

index 522fd23807..1f48820cc6 100644
@@ -19,7 +19,7 @@ python do_populate_cve_db() {
      Update NVD database with json data feed
      """
 
-    import sqlite3, urllib3, shutil, gzip, re
+    import sqlite3, urllib, shutil, gzip, re
      from datetime import date
 
      BASE_URL = "https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-"

@@ -33,16 +33,14 @@ python do_populate_cve_db() {
 
      initialize_db(c)
 
-    http = urllib3.PoolManager()
-
      for year in range(YEAR_START, date.today().year + 1):
          year_url = BASE_URL + str(year)
          meta_url = year_url + ".meta"
          json_url = year_url + ".json.gz"
 
          # Retrieve meta last modified date
-        with http.request('GET', meta_url, preload_content=False) as r:
-            date_line = str(r.data.splitlines()[0])
+        with urllib.request.urlopen(meta_url) as r:
+            date_line = str(r.read().splitlines()[0])
              last_modified = re.search('lastModifiedDate:(.*)', date_line).group(1)
 
          # Compare with current db last modified date

@@ -50,7 +48,7 @@ python do_populate_cve_db() {
          meta = c.fetchone()
          if not meta or meta[0] != last_modified:
              # Update db with current year json file
-            with http.request('GET', json_url, preload_content=False) as r, open(JSON_TMPFILE, 'wb') as tmpfile:
+            with urllib.request.urlopen(json_url) as r, open(JSON_TMPFILE, 'wb') as tmpfile:
                  shutil.copyfileobj(r, tmpfile)
              with gzip.open(JSON_TMPFILE, 'rt') as jsonfile:
                  update_db(c, jsonfile)

A recent do_populate_cve_db failure:
https://autobuilder.yoctoproject.org/typhoon/#/builders/23/builds/1005/steps/7/logs/errors

Could you please help look at what's going on?

Regards,
Chen Qi

Quoted 49 lines · Pierre Le Magourou

From: Pierre Le Magourou <[email protected]>

urllib3 was used in this recipe but it was not set as a dependency. As it is not specifically needed, rewrite the recipe with urllib from the standard library.

Signed-off-by Pierre Le Magourou <[email protected]>

---
  meta/recipes-core/meta/cve-update-db.bb | 10 ++++------
  1 file changed, 4 insertions(+), 6 deletions(-)

meta/recipes-core/meta/cve-update-db.bb +3 -4

index 522fd23807..1f48820cc6 100644
@@ -19,7 +19,7 @@ python do_populate_cve_db() {
      Update NVD database with json data feed
      """
  -    import sqlite3, urllib3, shutil, gzip, re
+    import sqlite3, urllib, shutil, gzip, re
      from datetime import date
        BASE_URL = "https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-"

@@ -33,16 +33,14 @@ python do_populate_cve_db() {
        initialize_db(c)
  -    http = urllib3.PoolManager()
-
      for year in range(YEAR_START, date.today().year + 1):
          year_url = BASE_URL + str(year)
          meta_url = year_url + ".meta"
          json_url = year_url + ".json.gz"
            # Retrieve meta last modified date
-        with http.request('GET', meta_url, preload_content=False) as r:
-            date_line = str(r.data.splitlines()[0])
+        with urllib.request.urlopen(meta_url) as r:
+            date_line = str(r.read().splitlines()[0])
              last_modified = re.search('lastModifiedDate:(.*)', 
 date_line).group(1)
            # Compare with current db last modified date

@@ -50,7 +48,7 @@ python do_populate_cve_db() {
          meta = c.fetchone()
          if not meta or meta[0] != last_modified:
              # Update db with current year json file
-            with http.request('GET', json_url, 

preload_content=False) as r, open(JSON_TMPFILE, 'wb') as tmpfile:
+ with urllib.request.urlopen(json_url) as r,
open(JSON_TMPFILE, 'wb') as tmpfile:
shutil.copyfileobj(r, tmpfile)
with gzip.open(JSON_TMPFILE, 'rt') as jsonfile:
update_db(c, jsonfile)

I just tired to open the web but failed. The web seems down.

Regards,
Chen Qi

Quoted 5 lines · ChenQi

A recent do_populate_cve_db failure:
https://autobuilder.yoctoproject.org/typhoon/#/builders/23/builds/1005/steps/7/logs/errors

Could you please help look at what's going on?

Quoted 5 lines · ChenQi

Quoted 4 lines

A recent do_populate_cve_db failure:
https://autobuilder.yoctoproject.org/typhoon/#/builders/23/builds/1005/steps/7/logs/errors

Quoted 1 line

Quoted 1 line

Could you please help look at what's going on?

Quoted 2 lines · ChenQi

I just tired to open the web but failed. The web seems down.

We were making some changes to the autobuilder to speed it up (changing database backends), its back now.

Hi,

Quoted 5 lines

Quoted 5 lines · ChenQi

Quoted 4 lines

A recent do_populate_cve_db failure:
https://autobuilder.yoctoproject.org/typhoon/#/builders/23/builds/1005/steps/7/logs/errors

Quoted 9 lines

Quoted 1 line

Quoted 1 line

Could you please help look at what's going on?

Quoted 2 lines · ChenQi

I just tired to open the web but failed. The web seems down.

We were making some changes to the autobuilder to speed it up (changing
database backends), its back now.

The NVD website is down, cve-update-db should not fail in that case. I
need to update it, so it prints a warning instead.
I am also wondering if we should mirror NVD json data feeds, to avoid too many requests from Yocto users. (but I don't know how to do that)