Mixing
How to dynamically add/stop workers from Python code with Celery
I am working with a dockerized Django system which processes buffers for users. Let's say I have 3 users and one of them decides to queue thousands of buffers. I do not want the other 2 to have to wait before all of those are completed.
Would it be possible to dynamically start a worker who only processes tasks which are in the queue assigned to that user? I know you can dynamically decide the queue in which a task should be placed upon calling the task.
I tried starting workers from within my Django project on startup. But I seem to fail to start more than just one. I attempted doing this in my celery config at first:
import os
from celery import Celery
from celery.bin import worker
from django.conf import settings
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings")
app = Celery('project1')
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
app.conf.ONCE = {
'settings': {
'url': 'redis://' + 'redis',
'blocking': True,
'default_timeout': 60 * 60,
'blocking_timeout': 86400
}
}
print("Starting worker...")
for e in Customer.objects.all():
worker = worker.worker(app=app)
worker.run(queues=["Queue" + e.id.__str__(),])
Is this even an legitimate way of doing this? If so how can I create workers from my Python source code?
If this is not the way to go, how can I dynamically create or delete workers for my users?
python django docker celery
asked Jan 2 at 14:28
God SerenaGod Serena
288
add a comment |
I am working with a dockerized Django system which processes buffers for users. Let's say I have 3 users and one of them decides to queue thousands of buffers. I do not want the other 2 to have to wait before all of those are completed.
Would it be possible to dynamically start a worker who only processes tasks which are in the queue assigned to that user? I know you can dynamically decide the queue in which a task should be placed upon calling the task.
I tried starting workers from within my Django project on startup. But I seem to fail to start more than just one. I attempted doing this in my celery config at first:
import os
from celery import Celery
from celery.bin import worker
from django.conf import settings
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings")
app = Celery('project1')
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
app.conf.ONCE = {
'settings': {
'url': 'redis://' + 'redis',
'blocking': True,
'default_timeout': 60 * 60,
'blocking_timeout': 86400
}
}
print("Starting worker...")
for e in Customer.objects.all():
worker = worker.worker(app=app)
worker.run(queues=["Queue" + e.id.__str__(),])
Is this even an legitimate way of doing this? If so how can I create workers from my Python source code?
If this is not the way to go, how can I dynamically create or delete workers for my users?
python django docker celery
asked Jan 2 at 14:28
God SerenaGod Serena
288
add a comment |
I am working with a dockerized Django system which processes buffers for users. Let's say I have 3 users and one of them decides to queue thousands of buffers. I do not want the other 2 to have to wait before all of those are completed.
Would it be possible to dynamically start a worker who only processes tasks which are in the queue assigned to that user? I know you can dynamically decide the queue in which a task should be placed upon calling the task.
I tried starting workers from within my Django project on startup. But I seem to fail to start more than just one. I attempted doing this in my celery config at first:
import os
from celery import Celery
from celery.bin import worker
from django.conf import settings
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings")
app = Celery('project1')
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
app.conf.ONCE = {
'settings': {
'url': 'redis://' + 'redis',
'blocking': True,
'default_timeout': 60 * 60,
'blocking_timeout': 86400
}
}
print("Starting worker...")
for e in Customer.objects.all():
worker = worker.worker(app=app)
worker.run(queues=["Queue" + e.id.__str__(),])
Is this even an legitimate way of doing this? If so how can I create workers from my Python source code?
If this is not the way to go, how can I dynamically create or delete workers for my users?
python django docker celery
asked Jan 2 at 14:28
God SerenaGod Serena
288
I am working with a dockerized Django system which processes buffers for users. Let's say I have 3 users and one of them decides to queue thousands of buffers. I do not want the other 2 to have to wait before all of those are completed.
Would it be possible to dynamically start a worker who only processes tasks which are in the queue assigned to that user? I know you can dynamically decide the queue in which a task should be placed upon calling the task.
I tried starting workers from within my Django project on startup. But I seem to fail to start more than just one. I attempted doing this in my celery config at first:
import os
from celery import Celery
from celery.bin import worker
from django.conf import settings
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings")
app = Celery('project1')
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
app.conf.ONCE = {
'settings': {
'url': 'redis://' + 'redis',
'blocking': True,
'default_timeout': 60 * 60,
'blocking_timeout': 86400
}
}
print("Starting worker...")
for e in Customer.objects.all():
worker = worker.worker(app=app)
worker.run(queues=["Queue" + e.id.__str__(),])
Is this even an legitimate way of doing this? If so how can I create workers from my Python source code?
If this is not the way to go, how can I dynamically create or delete workers for my users?
python django docker celery
python django docker celery
asked Jan 2 at 14:28
God SerenaGod Serena
288
asked Jan 2 at 14:28
God SerenaGod Serena
288
asked Jan 2 at 14:28
God SerenaGod Serena
288
asked Jan 2 at 14:28
God SerenaGod Serena
288
asked Jan 2 at 14:28
God SerenaGod Serena
288
288
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
The code you have is a legitimate way to create a new worker. You would probably also want to investigate detaching the worker when you start it. That said, if you have lots of users this implementation is limited by two facts (1) all of the workers run on the same machine and (2) the caller does not control how / when the worker will stop.
answered Jan 3 at 7:02
2ps2ps
8,10721131
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54008101%2fhow-to-dynamically-add-stop-workers-from-python-code-with-celery%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
The code you have is a legitimate way to create a new worker. You would probably also want to investigate detaching the worker when you start it. That said, if you have lots of users this implementation is limited by two facts (1) all of the workers run on the same machine and (2) the caller does not control how / when the worker will stop.
answered Jan 3 at 7:02
2ps2ps
8,10721131
add a comment |
The code you have is a legitimate way to create a new worker. You would probably also want to investigate detaching the worker when you start it. That said, if you have lots of users this implementation is limited by two facts (1) all of the workers run on the same machine and (2) the caller does not control how / when the worker will stop.
answered Jan 3 at 7:02
2ps2ps
8,10721131
add a comment |
The code you have is a legitimate way to create a new worker. You would probably also want to investigate detaching the worker when you start it. That said, if you have lots of users this implementation is limited by two facts (1) all of the workers run on the same machine and (2) the caller does not control how / when the worker will stop.
answered Jan 3 at 7:02
2ps2ps
8,10721131
The code you have is a legitimate way to create a new worker. You would probably also want to investigate detaching the worker when you start it. That said, if you have lots of users this implementation is limited by two facts (1) all of the workers run on the same machine and (2) the caller does not control how / when the worker will stop.
answered Jan 3 at 7:02
2ps2ps
8,10721131
answered Jan 3 at 7:02
2ps2ps
8,10721131
answered Jan 3 at 7:02
2ps2ps
8,10721131
answered Jan 3 at 7:02
2ps2ps
8,10721131
8,10721131
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54008101%2fhow-to-dynamically-add-stop-workers-from-python-code-with-celery%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
