Skip to contentSkip to conversation

layerindex-web : adding asynchronous task execution

PATCH In review Diana Thayer 2 participants 7 messages Jun 27, 2017 – Aug 23, 2017

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 layerindex-web : adding asynchronous task execution

Hello!

This is a work-in-progress solution to issue 11197 within layerindex-web: adding asynchronous task execution. I used RabbitMQ and Celery. I'm still working with Paul Eggleton on adding install instructions for RabbitMQ to the site's Dockerfile.

Previously, when a user submitted a layer, Django would synchronously send emails to users able to publish layers, such that if an email bounced or otherwise failed, the user submitting the layer would see an error. This patch sends those emails as asynchronous tasks, so any sending errors don't reach the user submitting a layer.

Thanks for reviewing this! Here's a brief changelog:

- updated readme to include RabbitMQ as a dependency
- updated settings.py to include RABBIT_BROKER and RABBIT_BACKEND
- added celery to requirements.txt
- defined celery tasks in layerindex/tasks.py
- separated layerindex/update#main into two functions: main (which parses sys.argv) and do_update (which accepts options as keyword arguments)
- updated layerindex/views.py to use the send_email task to asynchronously message anyone able to publish layers when a new layer is submitted

---
 README               |   5 +-
 layerindex/tasks.py  |  31 +++++++++++
 layerindex/update.py | 143 +++++++++++++++++++++++++++------------------------
 layerindex/views.py  |   7 ++-
 requirements.txt     |   1 +
 settings.py          |   4 ++
 6 files changed, 120 insertions(+), 71 deletions(-)
 create mode 100644 layerindex/tasks.py
README +4 -1
index 62f739d..cfcc37f 100644
@@ -14,6 +14,7 @@ In order to make use of this application you will need:
 * Python 3.4+
 * Django 1.8.x - tested with 1.8.17; newer versions may work, but
   the application has not been tested with 1.9 or newer.
+* RabbitMQ 3.6.x - tested with 3.6.10.
 * For production usage, a web server set up to host Django applications
   (not needed for local-only testing)
 * A database supported by Django (SQLite, MySQL, etc.). Django takes
@@ -41,7 +42,9 @@ Setup instructions:
 1. Edit settings.py to specify a database, EMAIL_HOST, SECRET_KEY and
    other settings specific to your installation. Ensure you set
    LAYER_FETCH_DIR to an absolute path to a location with sufficient
-   space for fetching layer repositories.
+   space for fetching layer repositories. Modify RABBIT_BROKER
+   and RABBIT_BACKEND to reflect the settings used by your RabbitMQ
+   server.
 
 2. Run the following commands within the layerindex-web directory to
    initialise the database:
layerindex/tasks.py new file +31 -0
new file mode 100644
index 0000000..9bb4701
@@ -0,0 +1,31 @@
+from celery import Celery
+from django.core.mail import EmailMessage
+import os
+import time
+
+try:
+  from update import do_update
+except ImportError:
+  from .update import do_update
+
+try:
+  import settings
+except ImportError:
+  # not in a full django env, so settings is inaccessible.
+  # setup django to access settings.
+  from utils import setup_django
+  setup_django()
+  import settings
+
+tasks = Celery('layerindex',
+    broker=settings.RABBIT_BROKER,
+    backend=settings.RABBIT_BACKEND)
+
[email protected]
+def update(**options):
+  return do_update(**options)
+
[email protected]
+def send_email(subject, text_content, from_email=settings.DEFAULT_FROM_EMAIL, to_emails=[]):
+  msg = EmailMessage(subject, text_content, from_email, to_emails)
+  msg.send()
layerindex/update.py +77 -66

221 lines not on this page · raw message

layerindex/views.py +3 -4
index 65a536a..c7c20b8 100644
@@ -19,7 +19,6 @@ from layerindex.forms import EditLayerForm, LayerMaintainerFormSet, EditNoteForm
 from django.db import transaction
 from django.contrib.auth.models import User, Permission
 from django.db.models import Q, Count, Sum
-from django.core.mail import EmailMessage
 from django.template.loader import get_template
 from django.template import Context
 from django.utils.decorators import method_decorator
@@ -28,6 +27,7 @@ from django.contrib import messages
 from reversion.models import Revision
 from . import utils
 from . import simplesearch
+from . import tasks
 import settings
 from django.dispatch import receiver
 import reversion
@@ -163,7 +163,7 @@ def edit_layer_view(request, template_name, branch='master', slug=None):
                     # Send email
                     plaintext = get_template('layerindex/submitemail.txt')
                     perm = Permission.objects.get(codename='publish_layer')
-                    users = User.objects.filter(Q(groups__permissions=perm) | Q(user_permissions=perm) ).distinct()
+                    users = User.objects.filter(Q(groups__permissions=perm) | Q(user_permissions=perm) | Q(is_superuser=True) ).distinct()
                     for user in users:
                         if user.first_name:
                             user_name = user.first_name
@@ -178,8 +178,7 @@ def edit_layer_view(request, template_name, branch='master', slug=None):
                         from_email = settings.SUBMIT_EMAIL_FROM
                         to_email = user.email
                         text_content = plaintext.render(d)
-                        msg = EmailMessage(subject, text_content, from_email, [to_email])
-                        msg.send()
+                        tasks.send_email(subject, text_content, from_email, [to_email])
                     return HttpResponseRedirect(reverse('submit_layer_thanks'))
             messages.success(request, 'Layer %s saved successfully.' % layeritem.name)
             if return_url:
requirements.txt +1 -0
index f80e5d4..7cd741b 100644
@@ -1,3 +1,4 @@
+celery==3.1.25
 Django==1.8.17
 django-cors-headers==1.1.0
 django-nvd3==0.9.7
settings.py +4 -0
index 0ecf90b..94d698b 100644
@@ -217,3 +217,7 @@ UPDATE_PURGE_DAYS = 30
 # Settings for layer submission feature
 SUBMIT_EMAIL_FROM = '[email protected]'
 SUBMIT_EMAIL_SUBJECT = 'OE Layerindex layer submission'
+
+# RabbitMQ settings
+RABBIT_BROKER = 'amqp://'
+RABBIT_BACKEND = 'rpc://'
Signature · 1 line
2.7.4
#1 lore raw

Hi Diana,

Sorry it's taken me a ridiculous amount of time to get back to you on this!

Quoted 6 lines · CEST Diana Thayer

This is a work-in-progress solution to issue 11197 within layerindex-web:
adding asynchronous task execution.
I used RabbitMQ and Celery. I'm still working with Paul Eggleton on adding install instructions for RabbitMQ to
the site's Dockerfile.

Quoted 6 lines

Previously, when a user submitted a layer, Django would synchronously send emails to users able to publish layers,
such that if an email bounced or otherwise failed, the user submitting the layer would see an error.
This patch sends those emails as asynchronous tasks, so any sending errors don't reach the user submitting a layer.

I had a chance to try your patch out today, although I had to apply it partially - I figured a good test would just be the email part. I installed rabbitmq and ensured it got started. I started the web interface and submitted a dummy layer, and the email sent as expected. I then tried killing my local SMTP server and the layer submission returned an error page, which is what happened previously - it looks to me as if the sending is still being done synchronously. The celery worker isn't reporting anything so I can only assume the django app is not trying to connect to it.

I've pushed just the changes I applied to a new branch here:

http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/log/?h=paule/celery

Any idea what's going wrong? Did I mess up in applying the changes, or misconfigure something else?

Thanks,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#2 lore raw
Quoted 24 lines · CEST Paul Eggleton
Quoted 6 lines · CEST Diana Thayer

This is a work-in-progress solution to issue 11197 within layerindex-web:
adding asynchronous task execution.
I used RabbitMQ and Celery. I'm still working with Paul Eggleton on adding install instructions for RabbitMQ to
the site's Dockerfile.

Quoted 6 lines

Previously, when a user submitted a layer, Django would synchronously send emails to users able to publish layers,
such that if an email bounced or otherwise failed, the user submitting the layer would see an error.
This patch sends those emails as asynchronous tasks, so any sending errors don't reach the user submitting a layer.

I had a chance to try your patch out today, although I had to apply it partially - I figured a good test would just be the email part. I installed rabbitmq and ensured it got started. I started the web interface and submitted a dummy layer, and the email sent as expected. I then tried killing my local SMTP server and the layer submission returned an error page, which is what happened previously - it looks to me as if the sending is still being done synchronously. The celery worker isn't reporting anything so I can only assume the django app is not trying to connect to it.

Quoted 6 lines

I've pushed just the changes I applied to a new branch here:

http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/log/?h=paule/celery

Any idea what's going wrong? Did I mess up in applying the changes, or misconfigure something else?

I figured out the issue - in order to have the task executed asynchronously you need to call .delay(<args>) on the task object rather than the task function itself. That leads to a "django.core.exceptions.ImproperlyConfigured" error reported by the worker since in my configuration the "import settings" succeeds but Django isn't set up properly, but at least it's actually executing the task via the worker now.

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#3 lore raw

Great! Thanks for testing it out and identifying that issue.

I've put together a PR on GitHub
<https://github.com/garbados/layerindex-web/pull/1> that includes some changes that appear to fix the issue you're describing. It also updates the readme to document how to configure the connection to the RabbitMQ server, and how to start the Celery worker. You can see all the file changes here
<https://github.com/garbados/layerindex-web/pull/1/files>.

Quoted 57 lines · Paul Eggleton
Quoted 2 lines · CEST Paul Eggleton
Quoted 1 line · CEST Diana Thayer

This is a work-in-progress solution to issue 11197 within

layerindex-web:

Quoted 2 lines
Quoted 2 lines

adding asynchronous task execution.
I used RabbitMQ and Celery. I'm still working with Paul Eggleton on

adding

Quoted 4 lines
Quoted 4 lines

install instructions for RabbitMQ to the site's Dockerfile.

Previously, when a user submitted a layer, Django would synchronously

send

Quoted 2 lines
Quoted 2 lines

emails to users able to publish layers,
such that if an email bounced or otherwise failed, the user submitting

the

Quoted 2 lines
Quoted 2 lines

layer would see an error.
This patch sends those emails as asynchronous tasks, so any sending

errors

Quoted 4 lines
Quoted 1 line

don't reach the user submitting a layer.

I had a chance to try your patch out today, although I had to apply it partially - I figured a good test would just be the email part. I

installed

Quoted 4 lines

rabbitmq and ensured it got started. I started the web interface and submitted a dummy layer, and the email sent as expected. I then tried killing my local SMTP server and the layer submission returned an error page, which is what happened previously - it looks to me as if the

sending

Quoted 7 lines

is still being done synchronously. The celery worker isn't reporting anything so I can only assume the django app is not trying to connect to it.

I've pushed just the changes I applied to a new branch here:

http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/

log/?h=paule/celery

Quoted 3 lines

Any idea what's going wrong? Did I mess up in applying the changes, or misconfigure something else?

I figured out the issue - in order to have the task executed asynchronously you need to call .delay(<args>) on the task object rather than the task function itself. That leads to a "django.core.exceptions. ImproperlyConfigured"
error reported by the worker since in my configuration the "import settings"
succeeds but Django isn't set up properly, but at least it's actually executing the task via the worker now.

Cheers,
Paul

--

Paul Eggleton
Intel Open Source Technology Centre

#4 lore raw

Hello! Did you get a chance to look at the PR
<https://github.com/garbados/layerindex-web/pull/1>?

Quoted 73 lines · Diana Thayer

Great! Thanks for testing it out and identifying that issue.

I've put together a PR on GitHub
<https://github.com/garbados/layerindex-web/pull/1> that includes some changes that appear to fix the issue you're describing. It also updates the readme to document how to configure the connection to the RabbitMQ server, and how to start the Celery worker. You can see all the file changes here
<https://github.com/garbados/layerindex-web/pull/1/files>.

Quoted 59 lines · Paul Eggleton
Quoted 2 lines · CEST Paul Eggleton
Quoted 1 line · CEST Diana Thayer

This is a work-in-progress solution to issue 11197 within

layerindex-web:

Quoted 2 lines
Quoted 2 lines

adding asynchronous task execution.
I used RabbitMQ and Celery. I'm still working with Paul Eggleton on

adding

Quoted 4 lines
Quoted 4 lines

install instructions for RabbitMQ to the site's Dockerfile.

Previously, when a user submitted a layer, Django would synchronously

send

Quoted 2 lines
Quoted 2 lines

emails to users able to publish layers,
such that if an email bounced or otherwise failed, the user

submitting the

Quoted 2 lines
Quoted 2 lines

layer would see an error.
This patch sends those emails as asynchronous tasks, so any sending

errors

Quoted 4 lines
Quoted 1 line

don't reach the user submitting a layer.

I had a chance to try your patch out today, although I had to apply it partially - I figured a good test would just be the email part. I

installed

Quoted 4 lines

rabbitmq and ensured it got started. I started the web interface and submitted a dummy layer, and the email sent as expected. I then tried killing my local SMTP server and the layer submission returned an error page, which is what happened previously - it looks to me as if the

sending

Quoted 2 lines

is still being done synchronously. The celery worker isn't reporting anything so I can only assume the django app is not trying to connect

to

Quoted 5 lines

it.

I've pushed just the changes I applied to a new branch here:

http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/log

/?h=paule/celery

Quoted 3 lines

Any idea what's going wrong? Did I mess up in applying the changes, or misconfigure something else?

I figured out the issue - in order to have the task executed asynchronously
you need to call .delay(<args>) on the task object rather than the task function itself. That leads to a "django.core.exceptions.Improp erlyConfigured"
error reported by the worker since in my configuration the "import settings"
succeeds but Django isn't set up properly, but at least it's actually executing the task via the worker now.

Cheers,
Paul

--

Paul Eggleton
Intel Open Source Technology Centre

#5 lore raw

Hi Diana

I did finally manage to get to it today - sorry about the delay. I've sorted out the settings issue and dropped the related item from the todo list. Assuming that didn't break anything on your end, could you please squash all of these into one commit with you as the author and an appropriate shortlog / commit message and then send a new patch? We'd probably want to mention that the mechanism this establishes will be used later for executing other asynchronous tasks.

Thanks,
Paul

Quoted 71 lines · NZST Diana Thayer

Hello! Did you get a chance to look at the PR
<https://github.com/garbados/layerindex-web/pull/1>?

Quoted 66 lines · Diana Thayer

Great! Thanks for testing it out and identifying that issue.

I've put together a PR on GitHub
<https://github.com/garbados/layerindex-web/pull/1> that includes some changes that appear to fix the issue you're describing. It also updates the readme to document how to configure the connection to the RabbitMQ server, and how to start the Celery worker. You can see all the file changes here
<https://github.com/garbados/layerindex-web/pull/1/files>.

Quoted 54 lines · Paul Eggleton
Quoted 2 lines · CEST Paul Eggleton
Quoted 1 line · CEST Diana Thayer

This is a work-in-progress solution to issue 11197 within

layerindex-web:

Quoted 2 lines
Quoted 2 lines

adding asynchronous task execution.
I used RabbitMQ and Celery. I'm still working with Paul Eggleton on

adding

Quoted 4 lines
Quoted 4 lines

install instructions for RabbitMQ to the site's Dockerfile.

Previously, when a user submitted a layer, Django would synchronously

send

Quoted 2 lines
Quoted 2 lines

emails to users able to publish layers,
such that if an email bounced or otherwise failed, the user

submitting the

Quoted 2 lines
Quoted 2 lines

layer would see an error.
This patch sends those emails as asynchronous tasks, so any sending

errors

Quoted 4 lines
Quoted 1 line

don't reach the user submitting a layer.

I had a chance to try your patch out today, although I had to apply it partially - I figured a good test would just be the email part. I

installed

Quoted 4 lines

rabbitmq and ensured it got started. I started the web interface and submitted a dummy layer, and the email sent as expected. I then tried killing my local SMTP server and the layer submission returned an error page, which is what happened previously - it looks to me as if the

sending

Quoted 2 lines

is still being done synchronously. The celery worker isn't reporting anything so I can only assume the django app is not trying to connect

to

Quoted 5 lines

it.

I've pushed just the changes I applied to a new branch here:

http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/log

/?h=paule/celery

Quoted 3 lines

Any idea what's going wrong? Did I mess up in applying the changes, or misconfigure something else?

I figured out the issue - in order to have the task executed asynchronously
you need to call .delay(<args>) on the task object rather than the task function itself. That leads to a "django.core.exceptions.Improp erlyConfigured"
error reported by the worker since in my configuration the "import settings"
succeeds but Django isn't set up properly, but at least it's actually executing the task via the worker now.

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#6 lore raw
Quoted 4 lines · NZST Paul Eggleton

Hi Diana

I did finally manage to get to it today - sorry about the delay. I've sorted out the settings issue and dropped the related item from the todo list.

Oops, I neglected to mention I pushed these minor changes to the same paule/celery branch.

Quoted 85 lines

Assuming that didn't break anything on your end, could you please squash all of these into one commit with you as the author and an appropriate shortlog / commit message and then send a new patch? We'd probably want to mention that the mechanism this establishes will be used later for executing other asynchronous tasks.

Thanks,
Paul

Quoted 71 lines · NZST Diana Thayer

Hello! Did you get a chance to look at the PR
<https://github.com/garbados/layerindex-web/pull/1>?

Quoted 66 lines · Diana Thayer

Great! Thanks for testing it out and identifying that issue.

I've put together a PR on GitHub
<https://github.com/garbados/layerindex-web/pull/1> that includes some changes that appear to fix the issue you're describing. It also updates the readme to document how to configure the connection to the RabbitMQ server, and how to start the Celery worker. You can see all the file changes here
<https://github.com/garbados/layerindex-web/pull/1/files>.

Quoted 54 lines · Paul Eggleton
Quoted 2 lines · CEST Paul Eggleton
Quoted 1 line · CEST Diana Thayer

This is a work-in-progress solution to issue 11197 within

layerindex-web:

Quoted 2 lines
Quoted 2 lines

adding asynchronous task execution.
I used RabbitMQ and Celery. I'm still working with Paul Eggleton on

adding

Quoted 4 lines
Quoted 4 lines

install instructions for RabbitMQ to the site's Dockerfile.

Previously, when a user submitted a layer, Django would synchronously

send

Quoted 2 lines
Quoted 2 lines

emails to users able to publish layers,
such that if an email bounced or otherwise failed, the user

submitting the

Quoted 2 lines
Quoted 2 lines

layer would see an error.
This patch sends those emails as asynchronous tasks, so any sending

errors

Quoted 4 lines
Quoted 1 line

don't reach the user submitting a layer.

I had a chance to try your patch out today, although I had to apply it partially - I figured a good test would just be the email part. I

installed

Quoted 4 lines

rabbitmq and ensured it got started. I started the web interface and submitted a dummy layer, and the email sent as expected. I then tried killing my local SMTP server and the layer submission returned an error page, which is what happened previously - it looks to me as if the

sending

Quoted 2 lines

is still being done synchronously. The celery worker isn't reporting anything so I can only assume the django app is not trying to connect

to

Quoted 5 lines

it.

I've pushed just the changes I applied to a new branch here:

http://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/log

/?h=paule/celery

Quoted 3 lines

Any idea what's going wrong? Did I mess up in applying the changes, or misconfigure something else?

I figured out the issue - in order to have the task executed asynchronously
you need to call .delay(<args>) on the task object rather than the task function itself. That leads to a "django.core.exceptions.Improp erlyConfigured"
error reported by the worker since in my configuration the "import settings"
succeeds but Django isn't set up properly, but at least it's actually executing the task via the worker now.

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre