Skip to contentSkip to conversation

Upgrade to Django 3.2.5 LTS and Celery 5

PATCH Superseded Tim Orling 2 participants 17 messages Jul 18, 2021 – Jul 20, 2021

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Tim Orling

author
#0 lore raw

cover Upgrade to Django 3.2.5 LTS and Celery 5

The current code base uses Django 2.2.x which will go out of extended support in April 2022, but also holds us back from updating a number of dependencies.

Update to the current Django 3.2.5 LTS and also the latest Celery 5.1.2. Celery 4 has not had any commits since 2020 and is unlikely to be getting much more attention from the developers.

While we are at it, upgrade all of our dependencies to the latest versions.

The following changes since commit 2314fb2811bc3753d9dd4459747d8d30369321ad:

Add SPDX license headers to layerindex-web source files (2021-04-21 08:22:51 +1200)

are available in the Git repository at:

git://github.com/moto-timo/layerindex-web timo/django-3.2.5
https://github.com/moto-timo/layerindex-web/tree/timo/django-3.2.5

Tim Orling (15):
dockersetup.py: fix EMAIL_USE_SSL/TLS
requirements.txt: fix range check
Dockerfile: cleanup, install python3-wheel
dockersetup.py: letsencrypt 2048 bit rsa
layerindex/*: make all shebangs python3
requirements.txt: bump to Django 3.0
Updates for Django 3.0
requirements.txt: bump to Django 3.1
docker-compose: bump mariadb to 10.3
requirements.txt: update all to latest
about.html: update contributors
requirements.txt: bump to Django 3.2 LTS
settings: set DEFAULT_AUTO_FIELD
docker-compose: fix celery
README.devel: update versions

 Dockerfile                                    |  4 +-
 README.devel                                  |  6 +--
 docker-compose.yml                            |  4 +-
 docker/settings.py                            |  6 ++-
 dockersetup.py                                |  6 +--
 layerindex/bulkchange.py                      |  2 +-
 .../0045_layerbranch_classicrecipe.py         | 19 ++++++++
 layerindex/recipedesc.py                      |  2 +-
 layerindex/tools/import_classic_wiki.py       |  2 +-
 layerindex/tools/import_wiki_layers.py        |  2 +-
 layerindex/tools/site_name.py                 |  2 +-
 layerindex/tools/update_classic_status.py     |  2 +-
 layerindex/update_layer.py                    |  2 +-
 requirements.txt                              | 48 +++++++++----------
 settings.py                                   |  6 ++-
 templates/base.html                           |  2 +-
 templates/layerindex/about.html               |  2 +
 17 files changed, 72 insertions(+), 45 deletions(-)
 create mode 100644 layerindex/migrations/0045_layerbranch_classicrecipe.py
Signature · 1 line
2.32.0
author
#1 lore raw

PATCH 1/15 dockersetup.py: fix EMAIL_USE_SSL/TLS

Need to concatenate str(boolean), not bool

---
 dockersetup.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
dockersetup.py +2 -2
index e45a846..04dec91 100755
@@ -353,13 +353,13 @@ def edit_dockercompose(hostname, dbpassword, dbapassword, secretkey, rmqpassword
         elif '- "EMAIL_USE_SSL' in line:
             format = line[0:line.find('- "EMAIL_USE_SSL')].replace("#", "")
             if email_ssl:
-                newlines.append(format + '- "EMAIL_USE_SSL=' + email_ssl + '"\n')
+                newlines.append(format + '- "EMAIL_USE_SSL=' + str(email_ssl) + '"\n')
             else:
                 newlines.append(format + '#- "EMAIL_USE_SSL=<set this here if needed>"\n')
         elif '- "EMAIL_USE_TLS' in line:
             format = line[0:line.find('- "EMAIL_USE_TLS')].replace("#", "")
             if email_tls:
-                newlines.append(format + '- "EMAIL_USE_TLS=' + email_tls + '"\n')
+                newlines.append(format + '- "EMAIL_USE_TLS=' + str(email_tls) + '"\n')
             else:
                 newlines.append(format + '#- "EMAIL_USE_TLS=<set this here if needed>"\n')
         elif "ports:" in line:
Signature · 1 line
2.32.0
author
#2 lore raw

PATCH 2/15 requirements.txt: fix range check

In reply to Tim Orling

The range check needs a comma between >=2.2 and <2.3, otherwise we end up with Django 3.2.5 installed and we are not ready for that yet.

---
 requirements.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
requirements.txt +1 -1
index c7399ff..1c2c4a1 100644
@@ -3,7 +3,7 @@ beautifulsoup4==4.8.1
 billiard==3.6.1.0
 celery==4.3.0
 confusable-homoglyphs==3.2.0
-Django>=2.2<2.3
+Django>=2.2,<2.3
 django-appconf==1.0.3
 django-axes==4.5.4
 django-bootstrap-pagination==1.7.1
Signature · 1 line
2.32.0
author
#3 lore raw

PATCH 3/15 Dockerfile: cleanup, install python3-wheel

In reply to Tim Orling

Remove duplicate python3-pip, etc. lines
Need python3-wheel for bdist_wheel command

---
 Dockerfile | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
Dockerfile +1 -3
index 8c4c847..f6623fa 100644
@@ -24,10 +24,8 @@ RUN apt-get update \
 	python3-pip \
 	python3-mysqldb \
 	python3-dev \
-	python3-pip \
-	python3-mysqldb \
-	python3-dev \
 	python3-pil \
+        python3-wheel \
 	libjpeg-dev \
 	libmariadbclient-dev \
 	locales \
Signature · 1 line
2.32.0
author
#4 lore raw

PATCH 4/15 dockersetup.py: letsencrypt 2048 bit rsa

In reply to Tim Orling

The minimum length for an RSA pem is 2048 for the dummy cert.

---
 dockersetup.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
dockersetup.py +1 -1
index 04dec91..6a0ab67 100755
@@ -483,7 +483,7 @@ def setup_https(hostname, http_port, https_port, letsencrypt, cert, cert_key, em
             os.makedirs(local_letsencrypt_cert_dir)
         keyfile = os.path.join(letsencrypt_cert_subdir, 'privkey.pem')
         certfile = os.path.join(letsencrypt_cert_subdir, 'fullchain.pem')
-        return_code = subprocess.call(['openssl', 'req', '-x509', '-nodes', '-newkey', 'rsa:1024', '-days', '1', '-keyout', os.path.join(local_cert_dir, keyfile), '-out', os.path.join(local_cert_dir, certfile), '-subj', '/CN=localhost'], shell=False)
+        return_code = subprocess.call(['openssl', 'req', '-x509', '-nodes', '-newkey', 'rsa:2048', '-days', '1', '-keyout', os.path.join(local_cert_dir, keyfile), '-out', os.path.join(local_cert_dir, certfile), '-subj', '/CN=localhost'], shell=False)
         if return_code != 0:
             print("Dummy certificate generation failed")
             sys.exit(1)
Signature · 1 line
2.32.0
author
#5 lore raw

PATCH 5/15 layerindex/*: make all shebangs python3

In reply to Tim Orling

Several scripts still had /usr/bin/env python

---
 layerindex/bulkchange.py                  | 2 +-
 layerindex/recipedesc.py                  | 2 +-
 layerindex/tools/import_classic_wiki.py   | 2 +-
 layerindex/tools/import_wiki_layers.py    | 2 +-
 layerindex/tools/site_name.py             | 2 +-
 layerindex/tools/update_classic_status.py | 2 +-
 layerindex/update_layer.py                | 2 +-
 7 files changed, 7 insertions(+), 7 deletions(-)
layerindex/bulkchange.py +1 -1
index 6382053..0ec4058 100644
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 # layerindex-web - bulk change implementation
 #
layerindex/recipedesc.py +1 -1
index be47f6e..ee7f2fe 100644
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 # Test script
 #
layerindex/tools/import_classic_wiki.py +1 -1
index aa33b7a..bd7ab9a 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 # Import migration info from OE-Classic recipes wiki page into OE
 # layer index database
layerindex/tools/import_wiki_layers.py +1 -1
index 707d90e..2c82411 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 # Import layer index wiki page into database
 #
layerindex/tools/site_name.py +1 -1
index e1cff02..0f2782d 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 # Updates site name in Django database
 #
layerindex/tools/update_classic_status.py +1 -1
index 0bcf2e7..aa5d2ef 100755
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 # Update cover info for OE-Classic / other distro recipes in OE layer index database
 #
layerindex/update_layer.py +1 -1
index 42f99be..f51884d 100644
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 # Update layer index database for a single layer
 #
Signature · 1 line
2.32.0
author
#6 lore raw

PATCH 6/15 requirements.txt: bump to Django 3.0

In reply to Tim Orling

Incremental upgrade towards 3.2 LTS version.

---
 requirements.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
requirements.txt +1 -1
index 1c2c4a1..ac69589 100644
@@ -3,7 +3,7 @@ beautifulsoup4==4.8.1
 billiard==3.6.1.0
 celery==4.3.0
 confusable-homoglyphs==3.2.0
-Django>=2.2,<2.3
+Django>=3.0,<3.1
 django-appconf==1.0.3
 django-axes==4.5.4
 django-bootstrap-pagination==1.7.1
Signature · 1 line
2.32.0
author
#7 lore raw

PATCH 7/15 Updates for Django 3.0

In reply to Tim Orling

* Update requirements.txt versions
- Mostly update to latest pre-Django 3.1 versions
* Fix deprecated axes.backends.AxesModelBackend
- settings.py
- docker/settings.py
* Fix template syntax 'staticfiles' -> 'static'
- base.html
* Add migrations for layerbranch classicrecipe

---
 docker/settings.py                            |  3 ++-
 .../0045_layerbranch_classicrecipe.py         | 19 +++++++++++++++
 requirements.txt                              | 24 +++++++++----------
 settings.py                                   |  3 ++-
 templates/base.html                           |  2 +-
 5 files changed, 36 insertions(+), 15 deletions(-)
 create mode 100644 layerindex/migrations/0045_layerbranch_classicrecipe.py
docker/settings.py +2 -1
index 7330ab9..894974f 100644
@@ -102,6 +102,7 @@ MIDDLEWARE = (
     'django.contrib.auth.middleware.AuthenticationMiddleware',
     'django.contrib.messages.middleware.MessageMiddleware',
     'django.middleware.clickjacking.XFrameOptionsMiddleware',
+    'axes.middleware.AxesMiddleware',
     'layerindex.middleware.NonAtomicRevisionMiddleware',
 )
 
@@ -161,7 +162,7 @@ INSTALLED_APPS = (
 )
 
 AUTHENTICATION_BACKENDS = [
-    'axes.backends.AxesModelBackend',
+    'axes.backends.AxesBackend',
     'django.contrib.auth.backends.ModelBackend',
 ]
 
layerindex/migrations/0045_layerbranch_classicrecipe.py new file +19 -0
new file mode 100644
index 0000000..cb805c9
@@ -0,0 +1,19 @@
+# Generated by Django 3.0.14 on 2021-07-18 00:03
+
+from django.db import migrations, models
+import django.db.models.deletion
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('layerindex', '0044_extendedprovides'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='classicrecipe',
+            name='cover_layerbranch',
+            field=models.ForeignKey(blank=True, limit_choices_to={'branch__name': 'master'}, null=True, on_delete=django.db.models.deletion.SET_NULL, to='layerindex.LayerBranch', verbose_name='Covering layer'),
+        ),
+    ]
requirements.txt +12 -12
index ac69589..895994b 100644
@@ -1,24 +1,24 @@
-amqp==2.5.2
-beautifulsoup4==4.8.1
+amqp==2.6.1
+beautifulsoup4==4.9.3
 billiard==3.6.1.0
 celery==4.3.0
 confusable-homoglyphs==3.2.0
 Django>=3.0,<3.1
-django-appconf==1.0.3
-django-axes==4.5.4
+django-appconf==1.0.4
+django-axes==5.13.1
 django-bootstrap-pagination==1.7.1
-django-cors-headers==3.0.2
-django-ipware==2.1.0
+django-cors-headers==3.2.1
+django-ipware==3.0.2
 django-ranged-response==0.2.0
-django-registration==3.0.1
-django-reversion==2.0.13
-django-reversion-compare==0.8.6
-django-simple-captcha==0.5.12
-djangorestframework==3.9.4
+django-registration==3.1.2
+django-reversion==3.0.9
+django-reversion-compare==0.14.0
+django-simple-captcha==0.5.14
+djangorestframework==3.12.4
 gitdb2==2.0.6
 GitPython==2.1.13
 kombu==4.6.3
-mysqlclient==1.4.4
+mysqlclient==1.4.6
 Pillow==6.2.1
 pytz==2019.3
 six==1.12.0
settings.py +2 -1
index 1dd6ebc..d03d15c 100644
@@ -102,6 +102,7 @@ MIDDLEWARE = (
     'django.contrib.auth.middleware.AuthenticationMiddleware',
     'django.contrib.messages.middleware.MessageMiddleware',
     'django.middleware.clickjacking.XFrameOptionsMiddleware',
+    'axes.middleware.AxesMiddleware',
     'layerindex.middleware.NonAtomicRevisionMiddleware',
 )
 
@@ -161,7 +162,7 @@ INSTALLED_APPS = (
 )
 
 AUTHENTICATION_BACKENDS = [
-    'axes.backends.AxesModelBackend',
+    'axes.backends.AxesBackend',
     'django.contrib.auth.backends.ModelBackend',
 ]
 
templates/base.html +1 -1
index a465a68..f00c976 100644
@@ -9,7 +9,7 @@
 
 
 {% load i18n %}
-{% load staticfiles %}
+{% load static %}
 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
     "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
Signature · 1 line
2.32.0
author
#8 lore raw

PATCH 8/15 requirements.txt: bump to Django 3.1

In reply to Tim Orling

Another incremental upgrade towards 3.2 LTS

---
 requirements.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
requirements.txt +1 -1
index 895994b..865960c 100644
@@ -3,7 +3,7 @@ beautifulsoup4==4.9.3
 billiard==3.6.1.0
 celery==4.3.0
 confusable-homoglyphs==3.2.0
-Django>=3.0,<3.1
+Django>=3.1,<3.2
 django-appconf==1.0.4
 django-axes==5.13.1
 django-bootstrap-pagination==1.7.1
Signature · 1 line
2.32.0
author
#9 lore raw

PATCH 9/15 docker-compose: bump mariadb to 10.3

In reply to Tim Orling

10.2 was based on Ubuntu bionic 18.04,
bumping to 10.3 as it is the first release
on Ubuntu focal 20.04

---
 docker-compose.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
docker-compose.yml +1 -1
index 44260b6..3e16e95 100644
@@ -1,7 +1,7 @@
 version: '3'
 services:
   layersdb:
-    image: mariadb:10.2
+    image: mariadb:10.3
     command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci --wait_timeout=28800 --max_allowed_packet=128M
     environment:
      - "MYSQL_DATABASE=layersdb"
Signature · 1 line
2.32.0
author
#10 lore raw

PATCH 10/15 requirements.txt: update all to latest

In reply to Tim Orling

Since we are on a "modern" version of Python in Debian buster container (3.7) and a modern version of Django (3.1.x) we should be able to run with the latest of all dependencies.

---
 requirements.txt | 34 +++++++++++++++++-----------------
 1 file changed, 17 insertions(+), 17 deletions(-)
requirements.txt +17 -17
index 865960c..7704ac0 100644
@@ -1,27 +1,27 @@
-amqp==2.6.1
+amqp==5.0.6
 beautifulsoup4==4.9.3
-billiard==3.6.1.0
-celery==4.3.0
+billiard==3.6.4.0
+celery==5.1.2
 confusable-homoglyphs==3.2.0
 Django>=3.1,<3.2
 django-appconf==1.0.4
-django-axes==5.13.1
+django-axes==5.20.0
 django-bootstrap-pagination==1.7.1
-django-cors-headers==3.2.1
+django-cors-headers==3.7.0
 django-ipware==3.0.2
 django-ranged-response==0.2.0
-django-registration==3.1.2
-django-reversion==3.0.9
+django-registration==3.2
+django-reversion==4.0.0
 django-reversion-compare==0.14.0
 django-simple-captcha==0.5.14
 djangorestframework==3.12.4
-gitdb2==2.0.6
-GitPython==2.1.13
-kombu==4.6.3
-mysqlclient==1.4.6
-Pillow==6.2.1
-pytz==2019.3
-six==1.12.0
-smmap2==2.0.5
-soupsieve==1.9.4
-vine==1.3.0
+gitdb==4.0.7
+GitPython==3.1.18
+kombu==5.1.0
+mysqlclient==2.0.3
+Pillow==8.3.1
+pytz==2021.1
+six==1.16.0
+smmap==4.0.0
+soupsieve==2.2.1
+vine==5.0.0
Signature · 1 line
2.32.0
author
#11 lore raw

PATCH 11/15 about.html: update contributors

In reply to Tim Orling

* Add
- Amber Elliot
- Tim Orling

---
 templates/layerindex/about.html | 2 ++
 1 file changed, 2 insertions(+)
templates/layerindex/about.html +2 -0
index b6b461a..f79bf22 100644
@@ -51,6 +51,8 @@
 <li>Marius Avram</li>
 <li>Saul Wold</li>
 <li>Aníbal Limón</li>
+<li>Amber Elliot</li>
+<li>Tim Orling</li>
 </ul>
 
 <p>The code is Open Source and can be found on <a href="http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/">git.yoctoproject.org</a>. Patches welcome!</p>
Signature · 1 line
2.32.0
author
#12 lore raw

PATCH 12/15 requirements.txt: bump to Django 3.2 LTS

In reply to Tim Orling

Upgrade to the latest Django 3.2.x LTS for
extended support up until April 2024.

---
 requirements.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
requirements.txt +1 -1
index 7704ac0..41a0b1e 100644
@@ -3,7 +3,7 @@ beautifulsoup4==4.9.3
 billiard==3.6.4.0
 celery==5.1.2
 confusable-homoglyphs==3.2.0
-Django>=3.1,<3.2
+Django>=3.2,<3.3
 django-appconf==1.0.4
 django-axes==5.20.0
 django-bootstrap-pagination==1.7.1
Signature · 1 line
2.32.0
author
#13 lore raw

PATCH 13/15 settings: set DEFAULT_AUTO_FIELD

In reply to Tim Orling

New in Django 3.2

To quiet warnings, set DEFAULT_AUTO_FIELD to the
default value 'django.db.models.AutoField'

NOTE: The default value for newly created Django 3.2
projects is django.db.models.BigAutoField, but this causes the need for a migration in 'captcha'.

---
 docker/settings.py | 3 +++
 settings.py        | 3 +++
 2 files changed, 6 insertions(+)
docker/settings.py +3 -0
index 894974f..a3f007d 100644
@@ -140,6 +140,9 @@ TEMPLATES = [
     },
 ]
 
+# New in Django 3.2
+DEFAULT_AUTO_FIELD = 'django.db.models.AutoField'
+
 INSTALLED_APPS = (
     'django.contrib.auth',
     'django.contrib.contenttypes',
settings.py +3 -0
index d03d15c..a69eb53 100644
@@ -140,6 +140,9 @@ TEMPLATES = [
     },
 ]
 
+# New in Django 3.2
+DEFAULT_AUTO_FIELD = 'django.db.models.AutoField'
+
 INSTALLED_APPS = (
     'django.contrib.auth',
     'django.contrib.contenttypes',
Signature · 1 line
2.32.0
author
#14 lore raw

PATCH 14/15 docker-compose: fix celery

In reply to Tim Orling

In celery 5, the --workdir argument must come
before the 'worker' subcommand.

Without this, celery cannot load the layerindex
module and this causes the celery container to
continually restart.

---
 docker-compose.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
docker-compose.yml +1 -1
index 3e16e95..2dfff41 100644
@@ -80,7 +80,7 @@ services:
      #- "EMAIL_USE_TLS=<set this here if needed>"
      #- "DEBUG=1"
     restart: unless-stopped
-    command: /usr/local/bin/celery -A layerindex.tasks worker --loglevel=info --workdir=/opt/layerindex
+    command: /usr/local/bin/celery --workdir=/opt/layerindex --app layerindex.tasks worker --loglevel=INFO
   #layerscertbot:
   #  image: certbot/certbot
   #  volumes:
Signature · 1 line
2.32.0
author
#15 lore raw

PATCH 15/15 README.devel: update versions

In reply to Tim Orling

This updated code-base should be compatible with Django 3.1 and obviously 3.2 (but not 3.0 or earlier).

Django 3.2 requires Python 3.6+.

With the upgrade to Celery 5, it is safer to recommend RabbitMQ 3.8.x, since that is what we are using.

---
 README.devel | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
README.devel +3 -3
index b535304..cbf54bf 100644
@@ -7,9 +7,9 @@ covered in the main README.
 
 In order to run this application standalone, you will need:
 
-* Python 3.4+
-* Django 1.11.x (not compatible with 2.0 or newer)
-* RabbitMQ 3.7.x
+* Python 3.6+
+* Django 3.2.x (not compatible with 3.0 or older)
+* RabbitMQ 3.8.x
 * For production usage, a web server set up to host Django applications
   (not needed for local-only testing/development)
 * A database system supported by Django (SQLite, MariaDB/MySQL, etc.).
Signature · 1 line
2.32.0
#16 lore raw
In reply to Tim Orling
Quoted 10 lines · Tim Orling

The current code base uses Django 2.2.x which will go out of extended support in April 2022, but also holds us back from updating a number of dependencies.

Update to the current Django 3.2.5 LTS and also the latest Celery 5.1.2. Celery 4 has not had any commits since 2020 and is unlikely to be getting much more attention from the developers.

While we are at it, upgrade all of our dependencies to the latest versions.

I don't have the skills to review this in detail but it sounds like a good step forward for many different reasons, thanks!

Cheers,

Richard