Upgrade to Django 3.2.5 LTS and Celery 5
cover Upgrade to Django 3.2.5 LTS and Celery 5
The current code base uses Django 2.2.x which will go out of extended support in April 2022, but also holds us back from updating a number of dependencies.
Update to the current Django 3.2.5 LTS and also the latest Celery 5.1.2. Celery 4 has not had any commits since 2020 and is unlikely to be getting much more attention from the developers.
While we are at it, upgrade all of our dependencies to the latest versions.
The following changes since commit 2314fb2811bc3753d9dd4459747d8d30369321ad:
Add SPDX license headers to layerindex-web source files (2021-04-21 08:22:51 +1200)
are available in the Git repository at:
git://github.com/moto-timo/layerindex-web timo/django-3.2.5
https://github.com/moto-timo/layerindex-web/tree/timo/django-3.2.5
Tim Orling (15):
dockersetup.py: fix EMAIL_USE_SSL/TLS
requirements.txt: fix range check
Dockerfile: cleanup, install python3-wheel
dockersetup.py: letsencrypt 2048 bit rsa
layerindex/*: make all shebangs python3
requirements.txt: bump to Django 3.0
Updates for Django 3.0
requirements.txt: bump to Django 3.1
docker-compose: bump mariadb to 10.3
requirements.txt: update all to latest
about.html: update contributors
requirements.txt: bump to Django 3.2 LTS
settings: set DEFAULT_AUTO_FIELD
docker-compose: fix celery
README.devel: update versions
Dockerfile | 4 +- README.devel | 6 +-- docker-compose.yml | 4 +- docker/settings.py | 6 ++- dockersetup.py | 6 +-- layerindex/bulkchange.py | 2 +- .../0045_layerbranch_classicrecipe.py | 19 ++++++++ layerindex/recipedesc.py | 2 +- layerindex/tools/import_classic_wiki.py | 2 +- layerindex/tools/import_wiki_layers.py | 2 +- layerindex/tools/site_name.py | 2 +- layerindex/tools/update_classic_status.py | 2 +- layerindex/update_layer.py | 2 +- requirements.txt | 48 +++++++++---------- settings.py | 6 ++- templates/base.html | 2 +- templates/layerindex/about.html | 2 + 17 files changed, 72 insertions(+), 45 deletions(-) create mode 100644 layerindex/migrations/0045_layerbranch_classicrecipe.py
Signature · 1 line
2.32.0
PATCH 1/15 dockersetup.py: fix EMAIL_USE_SSL/TLS
Need to concatenate str(boolean), not bool
--- dockersetup.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
dockersetup.py +2 -2
@@ -353,13 +353,13 @@ def edit_dockercompose(hostname, dbpassword, dbapassword, secretkey, rmqpassword elif '- "EMAIL_USE_SSL' in line: format = line[0:line.find('- "EMAIL_USE_SSL')].replace("#", "") if email_ssl: - newlines.append(format + '- "EMAIL_USE_SSL=' + email_ssl + '"\n') + newlines.append(format + '- "EMAIL_USE_SSL=' + str(email_ssl) + '"\n') else: newlines.append(format + '#- "EMAIL_USE_SSL=<set this here if needed>"\n') elif '- "EMAIL_USE_TLS' in line: format = line[0:line.find('- "EMAIL_USE_TLS')].replace("#", "") if email_tls: - newlines.append(format + '- "EMAIL_USE_TLS=' + email_tls + '"\n') + newlines.append(format + '- "EMAIL_USE_TLS=' + str(email_tls) + '"\n') else: newlines.append(format + '#- "EMAIL_USE_TLS=<set this here if needed>"\n') elif "ports:" in line:
Signature · 1 line
2.32.0
PATCH 2/15 requirements.txt: fix range check
In reply to Tim OrlingThe range check needs a comma between >=2.2 and <2.3, otherwise we end up with Django 3.2.5 installed and we are not ready for that yet.
--- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
requirements.txt +1 -1
@@ -3,7 +3,7 @@ beautifulsoup4==4.8.1 billiard==3.6.1.0 celery==4.3.0 confusable-homoglyphs==3.2.0 -Django>=2.2<2.3 +Django>=2.2,<2.3 django-appconf==1.0.3 django-axes==4.5.4 django-bootstrap-pagination==1.7.1
Signature · 1 line
2.32.0
PATCH 3/15 Dockerfile: cleanup, install python3-wheel
In reply to Tim OrlingRemove duplicate python3-pip, etc. lines
Need python3-wheel for bdist_wheel command
--- Dockerfile | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
Dockerfile +1 -3
@@ -24,10 +24,8 @@ RUN apt-get update \ python3-pip \ python3-mysqldb \ python3-dev \ - python3-pip \ - python3-mysqldb \ - python3-dev \ python3-pil \ + python3-wheel \ libjpeg-dev \ libmariadbclient-dev \ locales \
Signature · 1 line
2.32.0
PATCH 4/15 dockersetup.py: letsencrypt 2048 bit rsa
In reply to Tim OrlingThe minimum length for an RSA pem is 2048 for the dummy cert.
--- dockersetup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
dockersetup.py +1 -1
@@ -483,7 +483,7 @@ def setup_https(hostname, http_port, https_port, letsencrypt, cert, cert_key, em os.makedirs(local_letsencrypt_cert_dir) keyfile = os.path.join(letsencrypt_cert_subdir, 'privkey.pem') certfile = os.path.join(letsencrypt_cert_subdir, 'fullchain.pem') - return_code = subprocess.call(['openssl', 'req', '-x509', '-nodes', '-newkey', 'rsa:1024', '-days', '1', '-keyout', os.path.join(local_cert_dir, keyfile), '-out', os.path.join(local_cert_dir, certfile), '-subj', '/CN=localhost'], shell=False) + return_code = subprocess.call(['openssl', 'req', '-x509', '-nodes', '-newkey', 'rsa:2048', '-days', '1', '-keyout', os.path.join(local_cert_dir, keyfile), '-out', os.path.join(local_cert_dir, certfile), '-subj', '/CN=localhost'], shell=False) if return_code != 0: print("Dummy certificate generation failed") sys.exit(1)
Signature · 1 line
2.32.0
PATCH 5/15 layerindex/*: make all shebangs python3
In reply to Tim OrlingSeveral scripts still had /usr/bin/env python
--- layerindex/bulkchange.py | 2 +- layerindex/recipedesc.py | 2 +- layerindex/tools/import_classic_wiki.py | 2 +- layerindex/tools/import_wiki_layers.py | 2 +- layerindex/tools/site_name.py | 2 +- layerindex/tools/update_classic_status.py | 2 +- layerindex/update_layer.py | 2 +- 7 files changed, 7 insertions(+), 7 deletions(-)
layerindex/bulkchange.py +1 -1
@@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!/usr/bin/env python3 # layerindex-web - bulk change implementation #
layerindex/recipedesc.py +1 -1
@@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!/usr/bin/env python3 # Test script #
layerindex/tools/import_classic_wiki.py +1 -1
@@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!/usr/bin/env python3 # Import migration info from OE-Classic recipes wiki page into OE # layer index database
layerindex/tools/import_wiki_layers.py +1 -1
@@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!/usr/bin/env python3 # Import layer index wiki page into database #
layerindex/tools/site_name.py +1 -1
@@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!/usr/bin/env python3 # Updates site name in Django database #
layerindex/tools/update_classic_status.py +1 -1
@@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!/usr/bin/env python3 # Update cover info for OE-Classic / other distro recipes in OE layer index database #
layerindex/update_layer.py +1 -1
@@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!/usr/bin/env python3 # Update layer index database for a single layer #
Signature · 1 line
2.32.0
PATCH 6/15 requirements.txt: bump to Django 3.0
In reply to Tim OrlingIncremental upgrade towards 3.2 LTS version.
--- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
requirements.txt +1 -1
@@ -3,7 +3,7 @@ beautifulsoup4==4.8.1 billiard==3.6.1.0 celery==4.3.0 confusable-homoglyphs==3.2.0 -Django>=2.2,<2.3 +Django>=3.0,<3.1 django-appconf==1.0.3 django-axes==4.5.4 django-bootstrap-pagination==1.7.1
Signature · 1 line
2.32.0
PATCH 7/15 Updates for Django 3.0
In reply to Tim Orling* Update requirements.txt versions
- Mostly update to latest pre-Django 3.1 versions
* Fix deprecated axes.backends.AxesModelBackend
- settings.py
- docker/settings.py
* Fix template syntax 'staticfiles' -> 'static'
- base.html
* Add migrations for layerbranch classicrecipe
--- docker/settings.py | 3 ++- .../0045_layerbranch_classicrecipe.py | 19 +++++++++++++++ requirements.txt | 24 +++++++++---------- settings.py | 3 ++- templates/base.html | 2 +- 5 files changed, 36 insertions(+), 15 deletions(-) create mode 100644 layerindex/migrations/0045_layerbranch_classicrecipe.py
docker/settings.py +2 -1
@@ -102,6 +102,7 @@ MIDDLEWARE = ( 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware', + 'axes.middleware.AxesMiddleware', 'layerindex.middleware.NonAtomicRevisionMiddleware', )
@@ -161,7 +162,7 @@ INSTALLED_APPS = ( ) AUTHENTICATION_BACKENDS = [ - 'axes.backends.AxesModelBackend', + 'axes.backends.AxesBackend', 'django.contrib.auth.backends.ModelBackend', ]
layerindex/migrations/0045_layerbranch_classicrecipe.py new file +19 -0
@@ -0,0 +1,19 @@ +# Generated by Django 3.0.14 on 2021-07-18 00:03 + +from django.db import migrations, models +import django.db.models.deletion + + +class Migration(migrations.Migration): + + dependencies = [ + ('layerindex', '0044_extendedprovides'), + ] + + operations = [ + migrations.AlterField( + model_name='classicrecipe', + name='cover_layerbranch', + field=models.ForeignKey(blank=True, limit_choices_to={'branch__name': 'master'}, null=True, on_delete=django.db.models.deletion.SET_NULL, to='layerindex.LayerBranch', verbose_name='Covering layer'), + ), + ]
requirements.txt +12 -12
@@ -1,24 +1,24 @@ -amqp==2.5.2 -beautifulsoup4==4.8.1 +amqp==2.6.1 +beautifulsoup4==4.9.3 billiard==3.6.1.0 celery==4.3.0 confusable-homoglyphs==3.2.0 Django>=3.0,<3.1 -django-appconf==1.0.3 -django-axes==4.5.4 +django-appconf==1.0.4 +django-axes==5.13.1 django-bootstrap-pagination==1.7.1 -django-cors-headers==3.0.2 -django-ipware==2.1.0 +django-cors-headers==3.2.1 +django-ipware==3.0.2 django-ranged-response==0.2.0 -django-registration==3.0.1 -django-reversion==2.0.13 -django-reversion-compare==0.8.6 -django-simple-captcha==0.5.12 -djangorestframework==3.9.4 +django-registration==3.1.2 +django-reversion==3.0.9 +django-reversion-compare==0.14.0 +django-simple-captcha==0.5.14 +djangorestframework==3.12.4 gitdb2==2.0.6 GitPython==2.1.13 kombu==4.6.3 -mysqlclient==1.4.4 +mysqlclient==1.4.6 Pillow==6.2.1 pytz==2019.3 six==1.12.0
settings.py +2 -1
@@ -102,6 +102,7 @@ MIDDLEWARE = ( 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware', + 'axes.middleware.AxesMiddleware', 'layerindex.middleware.NonAtomicRevisionMiddleware', )
@@ -161,7 +162,7 @@ INSTALLED_APPS = ( ) AUTHENTICATION_BACKENDS = [ - 'axes.backends.AxesModelBackend', + 'axes.backends.AxesBackend', 'django.contrib.auth.backends.ModelBackend', ]
templates/base.html +1 -1
@@ -9,7 +9,7 @@ {% load i18n %} -{% load staticfiles %} +{% load static %} <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
Signature · 1 line
2.32.0
PATCH 8/15 requirements.txt: bump to Django 3.1
In reply to Tim OrlingAnother incremental upgrade towards 3.2 LTS
--- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
requirements.txt +1 -1
@@ -3,7 +3,7 @@ beautifulsoup4==4.9.3 billiard==3.6.1.0 celery==4.3.0 confusable-homoglyphs==3.2.0 -Django>=3.0,<3.1 +Django>=3.1,<3.2 django-appconf==1.0.4 django-axes==5.13.1 django-bootstrap-pagination==1.7.1
Signature · 1 line
2.32.0
PATCH 9/15 docker-compose: bump mariadb to 10.3
In reply to Tim Orling10.2 was based on Ubuntu bionic 18.04,
bumping to 10.3 as it is the first release
on Ubuntu focal 20.04
--- docker-compose.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
docker-compose.yml +1 -1
@@ -1,7 +1,7 @@ version: '3' services: layersdb: - image: mariadb:10.2 + image: mariadb:10.3 command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci --wait_timeout=28800 --max_allowed_packet=128M environment: - "MYSQL_DATABASE=layersdb"
Signature · 1 line
2.32.0
PATCH 10/15 requirements.txt: update all to latest
In reply to Tim OrlingSince we are on a "modern" version of Python in Debian buster container (3.7) and a modern version of Django (3.1.x) we should be able to run with the latest of all dependencies.
--- requirements.txt | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-)
requirements.txt +17 -17
@@ -1,27 +1,27 @@ -amqp==2.6.1 +amqp==5.0.6 beautifulsoup4==4.9.3 -billiard==3.6.1.0 -celery==4.3.0 +billiard==3.6.4.0 +celery==5.1.2 confusable-homoglyphs==3.2.0 Django>=3.1,<3.2 django-appconf==1.0.4 -django-axes==5.13.1 +django-axes==5.20.0 django-bootstrap-pagination==1.7.1 -django-cors-headers==3.2.1 +django-cors-headers==3.7.0 django-ipware==3.0.2 django-ranged-response==0.2.0 -django-registration==3.1.2 -django-reversion==3.0.9 +django-registration==3.2 +django-reversion==4.0.0 django-reversion-compare==0.14.0 django-simple-captcha==0.5.14 djangorestframework==3.12.4 -gitdb2==2.0.6 -GitPython==2.1.13 -kombu==4.6.3 -mysqlclient==1.4.6 -Pillow==6.2.1 -pytz==2019.3 -six==1.12.0 -smmap2==2.0.5 -soupsieve==1.9.4 -vine==1.3.0 +gitdb==4.0.7 +GitPython==3.1.18 +kombu==5.1.0 +mysqlclient==2.0.3 +Pillow==8.3.1 +pytz==2021.1 +six==1.16.0 +smmap==4.0.0 +soupsieve==2.2.1 +vine==5.0.0
Signature · 1 line
2.32.0
PATCH 11/15 about.html: update contributors
In reply to Tim Orling* Add
- Amber Elliot
- Tim Orling
--- templates/layerindex/about.html | 2 ++ 1 file changed, 2 insertions(+)
templates/layerindex/about.html +2 -0
@@ -51,6 +51,8 @@ <li>Marius Avram</li> <li>Saul Wold</li> <li>Aníbal Limón</li> +<li>Amber Elliot</li> +<li>Tim Orling</li> </ul> <p>The code is Open Source and can be found on <a href="http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/">git.yoctoproject.org</a>. Patches welcome!</p>
Signature · 1 line
2.32.0
PATCH 12/15 requirements.txt: bump to Django 3.2 LTS
In reply to Tim OrlingUpgrade to the latest Django 3.2.x LTS for
extended support up until April 2024.
--- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
requirements.txt +1 -1
@@ -3,7 +3,7 @@ beautifulsoup4==4.9.3 billiard==3.6.4.0 celery==5.1.2 confusable-homoglyphs==3.2.0 -Django>=3.1,<3.2 +Django>=3.2,<3.3 django-appconf==1.0.4 django-axes==5.20.0 django-bootstrap-pagination==1.7.1
Signature · 1 line
2.32.0
PATCH 13/15 settings: set DEFAULT_AUTO_FIELD
In reply to Tim OrlingNew in Django 3.2
To quiet warnings, set DEFAULT_AUTO_FIELD to the
default value 'django.db.models.AutoField'
NOTE: The default value for newly created Django 3.2
projects is django.db.models.BigAutoField, but this causes the need for a migration in 'captcha'.
--- docker/settings.py | 3 +++ settings.py | 3 +++ 2 files changed, 6 insertions(+)
docker/settings.py +3 -0
@@ -140,6 +140,9 @@ TEMPLATES = [ }, ] +# New in Django 3.2 +DEFAULT_AUTO_FIELD = 'django.db.models.AutoField' + INSTALLED_APPS = ( 'django.contrib.auth', 'django.contrib.contenttypes',
settings.py +3 -0
@@ -140,6 +140,9 @@ TEMPLATES = [ }, ] +# New in Django 3.2 +DEFAULT_AUTO_FIELD = 'django.db.models.AutoField' + INSTALLED_APPS = ( 'django.contrib.auth', 'django.contrib.contenttypes',
Signature · 1 line
2.32.0
PATCH 14/15 docker-compose: fix celery
In reply to Tim OrlingIn celery 5, the --workdir argument must come
before the 'worker' subcommand.
Without this, celery cannot load the layerindex
module and this causes the celery container to
continually restart.
--- docker-compose.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
docker-compose.yml +1 -1
@@ -80,7 +80,7 @@ services: #- "EMAIL_USE_TLS=<set this here if needed>" #- "DEBUG=1" restart: unless-stopped - command: /usr/local/bin/celery -A layerindex.tasks worker --loglevel=info --workdir=/opt/layerindex + command: /usr/local/bin/celery --workdir=/opt/layerindex --app layerindex.tasks worker --loglevel=INFO #layerscertbot: # image: certbot/certbot # volumes:
Signature · 1 line
2.32.0
PATCH 15/15 README.devel: update versions
In reply to Tim OrlingThis updated code-base should be compatible with Django 3.1 and obviously 3.2 (but not 3.0 or earlier).
Django 3.2 requires Python 3.6+.
With the upgrade to Celery 5, it is safer to recommend RabbitMQ 3.8.x, since that is what we are using.
--- README.devel | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
README.devel +3 -3
@@ -7,9 +7,9 @@ covered in the main README. In order to run this application standalone, you will need: -* Python 3.4+ -* Django 1.11.x (not compatible with 2.0 or newer) -* RabbitMQ 3.7.x +* Python 3.6+ +* Django 3.2.x (not compatible with 3.0 or older) +* RabbitMQ 3.8.x * For production usage, a web server set up to host Django applications (not needed for local-only testing/development) * A database system supported by Django (SQLite, MariaDB/MySQL, etc.).
Signature · 1 line
2.32.0
Quoted 10 lines · Tim Orling
The current code base uses Django 2.2.x which will go out of extended support in April 2022, but also holds us back from updating a number of dependencies.
Update to the current Django 3.2.5 LTS and also the latest Celery 5.1.2. Celery 4 has not had any commits since 2020 and is unlikely to be getting much more attention from the developers.
While we are at it, upgrade all of our dependencies to the latest versions.
I don't have the skills to review this in detail but it sounds like a good step forward for many different reasons, thanks!
Cheers,
Richard