Mixing
Celery shared_task parameters wrong parsing
2
i'm using celery 4.2.0 with Django to schedule new tasks in rabbitmq and it randomly fails when it comes to parse arguments.
My code looks like that:
for param in serializer.validated_data['parameters']:
run_task.delay(param_name=param)
Where parameters is list of int and it is ensured with serializer validation.
@shared_task
def run_task(param_name: int) -> None:
obj= Obj.objects.filter(param=param_name)
...
Under UTs when i invoke this run_task with CELERY_TASK_ALWAYS_EAGER=True everything works. On real enviroment this tasks sometimes runs just OK and sometimes it fails with server error:
File "/usr/lib/python3.6/site-packages/rest_framework/decorators.py", line 53, in handler
return func(*args, **kwargs)
File "/opt/app/compliance/jobs/views.py", line 30, in create_task
run_task.delay(param_name=param)
File "/usr/lib/python3.6/site-packages/celery/app/task.py", line 408, in delay
return self.apply_async(args, kwargs)
File "/usr/lib/python3.6/site-packages/celery/app/task.py", line 535, in apply_async
**options
File "/usr/lib/python3.6/site-packages/celery/app/base.py", line 741, in send_task
with self.producer_or_acquire(producer) as P:
File "/usr/lib/python3.6/site-packages/celery/app/base.py", line 876, in producer_or_acquire
producer, self.producer_pool.acquire, block=True,
File "/usr/lib/python3.6/site-packages/celery/app/base.py", line 1246, in producer_pool
return self.amqp.producer_pool
File "/usr/lib/python3.6/site-packages/celery/app/amqp.py", line 612, in producer_pool
self.app.connection_for_write()]
File "/usr/lib/python3.6/site-packages/kombu/utils/collections.py", line 34, in __getitem__
h = eqhash(key)
File "/usr/lib/python3.6/site-packages/kombu/utils/collections.py", line 25, in eqhash
return o.__eqhash__()
File "/usr/lib/python3.6/site-packages/kombu/connection.py", line 629, in __eqhash__
repr(self.transport_options))
File "/usr/lib/python3.6/site-packages/kombu/utils/collections.py", line 16, in __init__
self.hashvalue = hash(seq)
TypeError: unhashable type: 'set'
I understand error itself but how come it appears here when I pass only one simple and parsable argument - not set. Does anyone know how to make it work persistently?
Thanks!
python django rabbitmq celery
asked Nov 21 '18 at 9:07
user3043817user3043817
162
add a comment |
2
i'm using celery 4.2.0 with Django to schedule new tasks in rabbitmq and it randomly fails when it comes to parse arguments.
My code looks like that:
for param in serializer.validated_data['parameters']:
run_task.delay(param_name=param)
Where parameters is list of int and it is ensured with serializer validation.
@shared_task
def run_task(param_name: int) -> None:
obj= Obj.objects.filter(param=param_name)
...
Under UTs when i invoke this run_task with CELERY_TASK_ALWAYS_EAGER=True everything works. On real enviroment this tasks sometimes runs just OK and sometimes it fails with server error:
File "/usr/lib/python3.6/site-packages/rest_framework/decorators.py", line 53, in handler
return func(*args, **kwargs)
File "/opt/app/compliance/jobs/views.py", line 30, in create_task
run_task.delay(param_name=param)
File "/usr/lib/python3.6/site-packages/celery/app/task.py", line 408, in delay
return self.apply_async(args, kwargs)
File "/usr/lib/python3.6/site-packages/celery/app/task.py", line 535, in apply_async
**options
File "/usr/lib/python3.6/site-packages/celery/app/base.py", line 741, in send_task
with self.producer_or_acquire(producer) as P:
File "/usr/lib/python3.6/site-packages/celery/app/base.py", line 876, in producer_or_acquire
producer, self.producer_pool.acquire, block=True,
File "/usr/lib/python3.6/site-packages/celery/app/base.py", line 1246, in producer_pool
return self.amqp.producer_pool
File "/usr/lib/python3.6/site-packages/celery/app/amqp.py", line 612, in producer_pool
self.app.connection_for_write()]
File "/usr/lib/python3.6/site-packages/kombu/utils/collections.py", line 34, in __getitem__
h = eqhash(key)
File "/usr/lib/python3.6/site-packages/kombu/utils/collections.py", line 25, in eqhash
return o.__eqhash__()
File "/usr/lib/python3.6/site-packages/kombu/connection.py", line 629, in __eqhash__
repr(self.transport_options))
File "/usr/lib/python3.6/site-packages/kombu/utils/collections.py", line 16, in __init__
self.hashvalue = hash(seq)
TypeError: unhashable type: 'set'
I understand error itself but how come it appears here when I pass only one simple and parsable argument - not set. Does anyone know how to make it work persistently?
Thanks!
python django rabbitmq celery
asked Nov 21 '18 at 9:07
user3043817user3043817
162
Can you print the value ofparamin the failing cases?
– Daniel Roseman
Nov 21 '18 at 9:11
Just line before sending i have:LOGGER.info('Sending request for param %d', param)and it producesSending request for param 17
– user3043817
Nov 21 '18 at 9:30
add a comment |
2
2
2
i'm using celery 4.2.0 with Django to schedule new tasks in rabbitmq and it randomly fails when it comes to parse arguments.
My code looks like that:
for param in serializer.validated_data['parameters']:
run_task.delay(param_name=param)
Where parameters is list of int and it is ensured with serializer validation.
@shared_task
def run_task(param_name: int) -> None:
obj= Obj.objects.filter(param=param_name)
...
Under UTs when i invoke this run_task with CELERY_TASK_ALWAYS_EAGER=True everything works. On real enviroment this tasks sometimes runs just OK and sometimes it fails with server error:
File "/usr/lib/python3.6/site-packages/rest_framework/decorators.py", line 53, in handler
return func(*args, **kwargs)
File "/opt/app/compliance/jobs/views.py", line 30, in create_task
run_task.delay(param_name=param)
File "/usr/lib/python3.6/site-packages/celery/app/task.py", line 408, in delay
return self.apply_async(args, kwargs)
File "/usr/lib/python3.6/site-packages/celery/app/task.py", line 535, in apply_async
**options
File "/usr/lib/python3.6/site-packages/celery/app/base.py", line 741, in send_task
with self.producer_or_acquire(producer) as P:
File "/usr/lib/python3.6/site-packages/celery/app/base.py", line 876, in producer_or_acquire
producer, self.producer_pool.acquire, block=True,
File "/usr/lib/python3.6/site-packages/celery/app/base.py", line 1246, in producer_pool
return self.amqp.producer_pool
File "/usr/lib/python3.6/site-packages/celery/app/amqp.py", line 612, in producer_pool
self.app.connection_for_write()]
File "/usr/lib/python3.6/site-packages/kombu/utils/collections.py", line 34, in __getitem__
h = eqhash(key)
File "/usr/lib/python3.6/site-packages/kombu/utils/collections.py", line 25, in eqhash
return o.__eqhash__()
File "/usr/lib/python3.6/site-packages/kombu/connection.py", line 629, in __eqhash__
repr(self.transport_options))
File "/usr/lib/python3.6/site-packages/kombu/utils/collections.py", line 16, in __init__
self.hashvalue = hash(seq)
TypeError: unhashable type: 'set'
I understand error itself but how come it appears here when I pass only one simple and parsable argument - not set. Does anyone know how to make it work persistently?
Thanks!
python django rabbitmq celery
asked Nov 21 '18 at 9:07
user3043817user3043817
162
i'm using celery 4.2.0 with Django to schedule new tasks in rabbitmq and it randomly fails when it comes to parse arguments.
My code looks like that:
for param in serializer.validated_data['parameters']:
run_task.delay(param_name=param)
Where parameters is list of int and it is ensured with serializer validation.
@shared_task
def run_task(param_name: int) -> None:
obj= Obj.objects.filter(param=param_name)
...
Under UTs when i invoke this run_task with CELERY_TASK_ALWAYS_EAGER=True everything works. On real enviroment this tasks sometimes runs just OK and sometimes it fails with server error:
File "/usr/lib/python3.6/site-packages/rest_framework/decorators.py", line 53, in handler
return func(*args, **kwargs)
File "/opt/app/compliance/jobs/views.py", line 30, in create_task
run_task.delay(param_name=param)
File "/usr/lib/python3.6/site-packages/celery/app/task.py", line 408, in delay
return self.apply_async(args, kwargs)
File "/usr/lib/python3.6/site-packages/celery/app/task.py", line 535, in apply_async
**options
File "/usr/lib/python3.6/site-packages/celery/app/base.py", line 741, in send_task
with self.producer_or_acquire(producer) as P:
File "/usr/lib/python3.6/site-packages/celery/app/base.py", line 876, in producer_or_acquire
producer, self.producer_pool.acquire, block=True,
File "/usr/lib/python3.6/site-packages/celery/app/base.py", line 1246, in producer_pool
return self.amqp.producer_pool
File "/usr/lib/python3.6/site-packages/celery/app/amqp.py", line 612, in producer_pool
self.app.connection_for_write()]
File "/usr/lib/python3.6/site-packages/kombu/utils/collections.py", line 34, in __getitem__
h = eqhash(key)
File "/usr/lib/python3.6/site-packages/kombu/utils/collections.py", line 25, in eqhash
return o.__eqhash__()
File "/usr/lib/python3.6/site-packages/kombu/connection.py", line 629, in __eqhash__
repr(self.transport_options))
File "/usr/lib/python3.6/site-packages/kombu/utils/collections.py", line 16, in __init__
self.hashvalue = hash(seq)
TypeError: unhashable type: 'set'
I understand error itself but how come it appears here when I pass only one simple and parsable argument - not set. Does anyone know how to make it work persistently?
Thanks!
python django rabbitmq celery
python django rabbitmq celery
asked Nov 21 '18 at 9:07
user3043817user3043817
162
asked Nov 21 '18 at 9:07
user3043817user3043817
162
asked Nov 21 '18 at 9:07
user3043817user3043817
162
asked Nov 21 '18 at 9:07
user3043817user3043817
162
asked Nov 21 '18 at 9:07
user3043817user3043817
162
162
Can you print the value ofparamin the failing cases?
– Daniel Roseman
Nov 21 '18 at 9:11
Just line before sending i have:LOGGER.info('Sending request for param %d', param)and it producesSending request for param 17
– user3043817
Nov 21 '18 at 9:30
add a comment |
Can you print the value ofparamin the failing cases?
– Daniel Roseman
Nov 21 '18 at 9:11
Just line before sending i have:LOGGER.info('Sending request for param %d', param)and it producesSending request for param 17
– user3043817
Nov 21 '18 at 9:30
Can you print the value of
param in the failing cases?– Daniel Roseman
Nov 21 '18 at 9:11
Can you print the value of
param in the failing cases?– Daniel Roseman
Nov 21 '18 at 9:11
Just line before sending i have:
LOGGER.info('Sending request for param %d', param) and it produces Sending request for param 17– user3043817
Nov 21 '18 at 9:30
Just line before sending i have:
LOGGER.info('Sending request for param %d', param) and it produces Sending request for param 17– user3043817
Nov 21 '18 at 9:30
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53408540%2fcelery-shared-task-parameters-wrong-parsing%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53408540%2fcelery-shared-task-parameters-wrong-parsing%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Can you print the value of
paramin the failing cases?– Daniel Roseman
Nov 21 '18 at 9:11
Just line before sending i have:
LOGGER.info('Sending request for param %d', param)and it producesSending request for param 17– user3043817
Nov 21 '18 at 9:30