layerindex-web : adding asynchronous task execution
PATCH 1/1 layerindex-web : adding asynchronous task execution
Hello!
This is a work-in-progress solution to issue 11197 within layerindex-web: adding asynchronous task execution. I used RabbitMQ and Celery. I'm still working with Paul Eggleton on adding install instructions for RabbitMQ to the site's Dockerfile.
Previously, when a user submitted a layer, Django would synchronously send emails to users able to publish layers, such that if an email bounced or otherwise failed, the user submitting the layer would see an error. This patch sends those emails as asynchronous tasks, so any sending errors don't reach the user submitting a layer.
Thanks for reviewing this! Here's a brief changelog:
- updated readme to include RabbitMQ as a dependency
- updated settings.py to include RABBIT_BROKER and RABBIT_BACKEND
- added celery to requirements.txt
- defined celery tasks in layerindex/tasks.py
- separated layerindex/update#main into two functions: main (which parses sys.argv) and do_update (which accepts options as keyword arguments)
- updated layerindex/views.py to use the send_email task to asynchronously message anyone able to publish layers when a new layer is submitted
--- README | 5 +- layerindex/tasks.py | 31 +++++++++++ layerindex/update.py | 143 +++++++++++++++++++++++++++------------------------ layerindex/views.py | 7 ++- requirements.txt | 1 + settings.py | 4 ++ 6 files changed, 120 insertions(+), 71 deletions(-) create mode 100644 layerindex/tasks.py
README +4 -1
@@ -14,6 +14,7 @@ In order to make use of this application you will need: * Python 3.4+ * Django 1.8.x - tested with 1.8.17; newer versions may work, but the application has not been tested with 1.9 or newer. +* RabbitMQ 3.6.x - tested with 3.6.10. * For production usage, a web server set up to host Django applications (not needed for local-only testing) * A database supported by Django (SQLite, MySQL, etc.). Django takes
@@ -41,7 +42,9 @@ Setup instructions: 1. Edit settings.py to specify a database, EMAIL_HOST, SECRET_KEY and other settings specific to your installation. Ensure you set LAYER_FETCH_DIR to an absolute path to a location with sufficient - space for fetching layer repositories. + space for fetching layer repositories. Modify RABBIT_BROKER + and RABBIT_BACKEND to reflect the settings used by your RabbitMQ + server. 2. Run the following commands within the layerindex-web directory to initialise the database:
layerindex/tasks.py new file +31 -0
@@ -0,0 +1,31 @@ +from celery import Celery +from django.core.mail import EmailMessage +import os +import time + +try: + from update import do_update +except ImportError: + from .update import do_update + +try: + import settings +except ImportError: + # not in a full django env, so settings is inaccessible. + # setup django to access settings. + from utils import setup_django + setup_django() + import settings + +tasks = Celery('layerindex', + broker=settings.RABBIT_BROKER, + backend=settings.RABBIT_BACKEND) + [email protected] +def update(**options): + return do_update(**options) + [email protected] +def send_email(subject, text_content, from_email=settings.DEFAULT_FROM_EMAIL, to_emails=[]): + msg = EmailMessage(subject, text_content, from_email, to_emails) + msg.send()
layerindex/update.py +77 -66
221 lines not on this page · raw message
layerindex/views.py +3 -4
@@ -19,7 +19,6 @@ from layerindex.forms import EditLayerForm, LayerMaintainerFormSet, EditNoteForm from django.db import transaction from django.contrib.auth.models import User, Permission from django.db.models import Q, Count, Sum -from django.core.mail import EmailMessage from django.template.loader import get_template from django.template import Context from django.utils.decorators import method_decorator
@@ -28,6 +27,7 @@ from django.contrib import messages from reversion.models import Revision from . import utils from . import simplesearch +from . import tasks import settings from django.dispatch import receiver import reversion
@@ -163,7 +163,7 @@ def edit_layer_view(request, template_name, branch='master', slug=None): # Send email plaintext = get_template('layerindex/submitemail.txt') perm = Permission.objects.get(codename='publish_layer') - users = User.objects.filter(Q(groups__permissions=perm) | Q(user_permissions=perm) ).distinct() + users = User.objects.filter(Q(groups__permissions=perm) | Q(user_permissions=perm) | Q(is_superuser=True) ).distinct() for user in users: if user.first_name: user_name = user.first_name
@@ -178,8 +178,7 @@ def edit_layer_view(request, template_name, branch='master', slug=None): from_email = settings.SUBMIT_EMAIL_FROM to_email = user.email text_content = plaintext.render(d) - msg = EmailMessage(subject, text_content, from_email, [to_email]) - msg.send() + tasks.send_email(subject, text_content, from_email, [to_email]) return HttpResponseRedirect(reverse('submit_layer_thanks')) messages.success(request, 'Layer %s saved successfully.' % layeritem.name) if return_url:
requirements.txt +1 -0
@@ -1,3 +1,4 @@ +celery==3.1.25 Django==1.8.17 django-cors-headers==1.1.0 django-nvd3==0.9.7
settings.py +4 -0
@@ -217,3 +217,7 @@ UPDATE_PURGE_DAYS = 30 # Settings for layer submission feature SUBMIT_EMAIL_FROM = '[email protected]' SUBMIT_EMAIL_SUBJECT = 'OE Layerindex layer submission' + +# RabbitMQ settings +RABBIT_BROKER = 'amqp://' +RABBIT_BACKEND = 'rpc://'
Signature · 1 line
2.7.4
Hi Diana,
Sorry it's taken me a ridiculous amount of time to get back to you on this!
Quoted 6 lines · CEST Diana Thayer
This is a work-in-progress solution to issue 11197 within layerindex-web:
adding asynchronous task execution.
I used RabbitMQ and Celery. I'm still working with Paul Eggleton on adding install instructions for RabbitMQ to
the site's Dockerfile.
Quoted 6 lines
Previously, when a user submitted a layer, Django would synchronously send emails to users able to publish layers,
such that if an email bounced or otherwise failed, the user submitting the layer would see an error.
This patch sends those emails as asynchronous tasks, so any sending errors don't reach the user submitting a layer.
I had a chance to try your patch out today, although I had to apply it partially - I figured a good test would just be the email part. I installed rabbitmq and ensured it got started. I started the web interface and submitted a dummy layer, and the email sent as expected. I then tried killing my local SMTP server and the layer submission returned an error page, which is what happened previously - it looks to me as if the sending is still being done synchronously. The celery worker isn't reporting anything so I can only assume the django app is not trying to connect to it.
I've pushed just the changes I applied to a new branch here:
http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/log/?h=paule/celery
Any idea what's going wrong? Did I mess up in applying the changes, or misconfigure something else?
Thanks,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 24 lines · CEST Paul Eggleton
Quoted 6 lines · CEST Diana Thayer
This is a work-in-progress solution to issue 11197 within layerindex-web:
adding asynchronous task execution.
I used RabbitMQ and Celery. I'm still working with Paul Eggleton on adding install instructions for RabbitMQ to
the site's Dockerfile.Quoted 6 lines
Previously, when a user submitted a layer, Django would synchronously send emails to users able to publish layers,
such that if an email bounced or otherwise failed, the user submitting the layer would see an error.
This patch sends those emails as asynchronous tasks, so any sending errors don't reach the user submitting a layer.I had a chance to try your patch out today, although I had to apply it partially - I figured a good test would just be the email part. I installed rabbitmq and ensured it got started. I started the web interface and submitted a dummy layer, and the email sent as expected. I then tried killing my local SMTP server and the layer submission returned an error page, which is what happened previously - it looks to me as if the sending is still being done synchronously. The celery worker isn't reporting anything so I can only assume the django app is not trying to connect to it.
Quoted 6 lines
I've pushed just the changes I applied to a new branch here:
http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/log/?h=paule/celery
Any idea what's going wrong? Did I mess up in applying the changes, or misconfigure something else?
I figured out the issue - in order to have the task executed asynchronously you need to call .delay(<args>) on the task object rather than the task function itself. That leads to a "django.core.exceptions.ImproperlyConfigured" error reported by the worker since in my configuration the "import settings" succeeds but Django isn't set up properly, but at least it's actually executing the task via the worker now.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Great! Thanks for testing it out and identifying that issue.
I've put together a PR on GitHub
<https://github.com/garbados/layerindex-web/pull/1> that includes some changes that appear to fix the issue you're describing. It also updates the readme to document how to configure the connection to the RabbitMQ server, and how to start the Celery worker. You can see all the file changes here
<https://github.com/garbados/layerindex-web/pull/1/files>.
Quoted 57 lines · Paul Eggleton
Quoted 2 lines · CEST Paul Eggleton
Quoted 1 line · CEST Diana Thayer
This is a work-in-progress solution to issue 11197 within
layerindex-web:
Quoted 2 lines
Quoted 2 lines
adding asynchronous task execution.
I used RabbitMQ and Celery. I'm still working with Paul Eggleton onadding
Quoted 4 lines
Quoted 4 lines
install instructions for RabbitMQ to the site's Dockerfile.
Previously, when a user submitted a layer, Django would synchronously
send
Quoted 2 lines
Quoted 2 lines
emails to users able to publish layers,
such that if an email bounced or otherwise failed, the user submittingthe
Quoted 2 lines
Quoted 2 lines
layer would see an error.
This patch sends those emails as asynchronous tasks, so any sendingerrors
Quoted 4 lines
Quoted 1 line
don't reach the user submitting a layer.
I had a chance to try your patch out today, although I had to apply it partially - I figured a good test would just be the email part. I
installed
Quoted 4 lines
rabbitmq and ensured it got started. I started the web interface and submitted a dummy layer, and the email sent as expected. I then tried killing my local SMTP server and the layer submission returned an error page, which is what happened previously - it looks to me as if the
sending
Quoted 7 lines
is still being done synchronously. The celery worker isn't reporting anything so I can only assume the django app is not trying to connect to it.
I've pushed just the changes I applied to a new branch here:
log/?h=paule/celery
Quoted 3 lines
Any idea what's going wrong? Did I mess up in applying the changes, or misconfigure something else?
I figured out the issue - in order to have the task executed asynchronously you need to call .delay(<args>) on the task object rather than the task function itself. That leads to a "django.core.exceptions. ImproperlyConfigured"
error reported by the worker since in my configuration the "import settings"
succeeds but Django isn't set up properly, but at least it's actually executing the task via the worker now.Cheers,
Paul--
Paul Eggleton
Intel Open Source Technology Centre
Hello! Did you get a chance to look at the PR
<https://github.com/garbados/layerindex-web/pull/1>?
Quoted 73 lines · Diana Thayer
Great! Thanks for testing it out and identifying that issue.
I've put together a PR on GitHub
<https://github.com/garbados/layerindex-web/pull/1> that includes some changes that appear to fix the issue you're describing. It also updates the readme to document how to configure the connection to the RabbitMQ server, and how to start the Celery worker. You can see all the file changes here
<https://github.com/garbados/layerindex-web/pull/1/files>.Quoted 59 lines · Paul Eggleton
Quoted 2 lines · CEST Paul Eggleton
Quoted 1 line · CEST Diana Thayer
This is a work-in-progress solution to issue 11197 within
layerindex-web:
Quoted 2 lines
Quoted 2 lines
adding asynchronous task execution.
I used RabbitMQ and Celery. I'm still working with Paul Eggleton onadding
Quoted 4 lines
Quoted 4 lines
install instructions for RabbitMQ to the site's Dockerfile.
Previously, when a user submitted a layer, Django would synchronously
send
Quoted 2 lines
Quoted 2 lines
emails to users able to publish layers,
such that if an email bounced or otherwise failed, the usersubmitting the
Quoted 2 lines
Quoted 2 lines
layer would see an error.
This patch sends those emails as asynchronous tasks, so any sendingerrors
Quoted 4 lines
Quoted 1 line
don't reach the user submitting a layer.
I had a chance to try your patch out today, although I had to apply it partially - I figured a good test would just be the email part. I
installed
Quoted 4 lines
rabbitmq and ensured it got started. I started the web interface and submitted a dummy layer, and the email sent as expected. I then tried killing my local SMTP server and the layer submission returned an error page, which is what happened previously - it looks to me as if the
sending
Quoted 2 lines
is still being done synchronously. The celery worker isn't reporting anything so I can only assume the django app is not trying to connect
to
Quoted 5 lines
it.
I've pushed just the changes I applied to a new branch here:
http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/log
/?h=paule/celery
Quoted 3 lines
Any idea what's going wrong? Did I mess up in applying the changes, or misconfigure something else?
I figured out the issue - in order to have the task executed asynchronously
you need to call .delay(<args>) on the task object rather than the task function itself. That leads to a "django.core.exceptions.Improp erlyConfigured"
error reported by the worker since in my configuration the "import settings"
succeeds but Django isn't set up properly, but at least it's actually executing the task via the worker now.Cheers,
Paul--
Paul Eggleton
Intel Open Source Technology Centre
Hi Diana
I did finally manage to get to it today - sorry about the delay. I've sorted out the settings issue and dropped the related item from the todo list. Assuming that didn't break anything on your end, could you please squash all of these into one commit with you as the author and an appropriate shortlog / commit message and then send a new patch? We'd probably want to mention that the mechanism this establishes will be used later for executing other asynchronous tasks.
Thanks,
Paul
Quoted 71 lines · NZST Diana Thayer
Hello! Did you get a chance to look at the PR
<https://github.com/garbados/layerindex-web/pull/1>?Quoted 66 lines · Diana Thayer
Great! Thanks for testing it out and identifying that issue.
I've put together a PR on GitHub
<https://github.com/garbados/layerindex-web/pull/1> that includes some changes that appear to fix the issue you're describing. It also updates the readme to document how to configure the connection to the RabbitMQ server, and how to start the Celery worker. You can see all the file changes here
<https://github.com/garbados/layerindex-web/pull/1/files>.Quoted 54 lines · Paul Eggleton
Quoted 2 lines · CEST Paul Eggleton
Quoted 1 line · CEST Diana Thayer
This is a work-in-progress solution to issue 11197 within
layerindex-web:
Quoted 2 lines
Quoted 2 lines
adding asynchronous task execution.
I used RabbitMQ and Celery. I'm still working with Paul Eggleton onadding
Quoted 4 lines
Quoted 4 lines
install instructions for RabbitMQ to the site's Dockerfile.
Previously, when a user submitted a layer, Django would synchronously
send
Quoted 2 lines
Quoted 2 lines
emails to users able to publish layers,
such that if an email bounced or otherwise failed, the usersubmitting the
Quoted 2 lines
Quoted 2 lines
layer would see an error.
This patch sends those emails as asynchronous tasks, so any sendingerrors
Quoted 4 lines
Quoted 1 line
don't reach the user submitting a layer.
I had a chance to try your patch out today, although I had to apply it partially - I figured a good test would just be the email part. I
installed
Quoted 4 lines
rabbitmq and ensured it got started. I started the web interface and submitted a dummy layer, and the email sent as expected. I then tried killing my local SMTP server and the layer submission returned an error page, which is what happened previously - it looks to me as if the
sending
Quoted 2 lines
is still being done synchronously. The celery worker isn't reporting anything so I can only assume the django app is not trying to connect
to
Quoted 5 lines
it.
I've pushed just the changes I applied to a new branch here:
http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/log
/?h=paule/celery
Quoted 3 lines
Any idea what's going wrong? Did I mess up in applying the changes, or misconfigure something else?
I figured out the issue - in order to have the task executed asynchronously
you need to call .delay(<args>) on the task object rather than the task function itself. That leads to a "django.core.exceptions.Improp erlyConfigured"
error reported by the worker since in my configuration the "import settings"
succeeds but Django isn't set up properly, but at least it's actually executing the task via the worker now.Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 4 lines · NZST Paul Eggleton
Hi Diana
I did finally manage to get to it today - sorry about the delay. I've sorted out the settings issue and dropped the related item from the todo list.
Oops, I neglected to mention I pushed these minor changes to the same paule/celery branch.
Quoted 85 lines
Assuming that didn't break anything on your end, could you please squash all of these into one commit with you as the author and an appropriate shortlog / commit message and then send a new patch? We'd probably want to mention that the mechanism this establishes will be used later for executing other asynchronous tasks.
Thanks,
PaulQuoted 71 lines · NZST Diana Thayer
Hello! Did you get a chance to look at the PR
<https://github.com/garbados/layerindex-web/pull/1>?Quoted 66 lines · Diana Thayer
Great! Thanks for testing it out and identifying that issue.
I've put together a PR on GitHub
<https://github.com/garbados/layerindex-web/pull/1> that includes some changes that appear to fix the issue you're describing. It also updates the readme to document how to configure the connection to the RabbitMQ server, and how to start the Celery worker. You can see all the file changes here
<https://github.com/garbados/layerindex-web/pull/1/files>.Quoted 54 lines · Paul Eggleton
Quoted 2 lines · CEST Paul Eggleton
Quoted 1 line · CEST Diana Thayer
This is a work-in-progress solution to issue 11197 within
layerindex-web:
Quoted 2 lines
Quoted 2 lines
adding asynchronous task execution.
I used RabbitMQ and Celery. I'm still working with Paul Eggleton onadding
Quoted 4 lines
Quoted 4 lines
install instructions for RabbitMQ to the site's Dockerfile.
Previously, when a user submitted a layer, Django would synchronously
send
Quoted 2 lines
Quoted 2 lines
emails to users able to publish layers,
such that if an email bounced or otherwise failed, the usersubmitting the
Quoted 2 lines
Quoted 2 lines
layer would see an error.
This patch sends those emails as asynchronous tasks, so any sendingerrors
Quoted 4 lines
Quoted 1 line
don't reach the user submitting a layer.
I had a chance to try your patch out today, although I had to apply it partially - I figured a good test would just be the email part. I
installed
Quoted 4 lines
rabbitmq and ensured it got started. I started the web interface and submitted a dummy layer, and the email sent as expected. I then tried killing my local SMTP server and the layer submission returned an error page, which is what happened previously - it looks to me as if the
sending
Quoted 2 lines
is still being done synchronously. The celery worker isn't reporting anything so I can only assume the django app is not trying to connect
to
Quoted 5 lines
it.
I've pushed just the changes I applied to a new branch here:
http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/log
/?h=paule/celery
Quoted 3 lines
Any idea what's going wrong? Did I mess up in applying the changes, or misconfigure something else?
I figured out the issue - in order to have the task executed asynchronously
you need to call .delay(<args>) on the task object rather than the task function itself. That leads to a "django.core.exceptions.Improp erlyConfigured"
error reported by the worker since in my configuration the "import settings"
succeeds but Django isn't set up properly, but at least it's actually executing the task via the worker now.Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre