Mixing
![How to prove that $n^{n-2} = O(2^{n^2})$ [closed]](https://lh4.googleusercontent.com/-704sC3B_mEI/AAAAAAAAAAI/AAAAAAAAAx4/DuAQiT4vCt8/s72-c/photo.jpg?sz=32)
Function takes too much time to process in loop from database(3000 times)
I am calling a function in loop of 3000 time . I am calling that function and getting some dictionary from that function but it takes time . I am fetching data from database which gives me approximately 3000 rows and i am looping that rows and calling function in that loop which fetch data from database and returns dictionary but it takes time .
Code:
def test(request, uni_id):
try:
Obj = get_object_or_404(tabl_name, id=uni_id)
except:
Obj = None
dict = {}
if Obj:outlet_info
dict['data1'] = Obj.id
dict['data2'] = Obj.name
dict['data3'] = Obj.eg
dict['data4'] = Obj.access
return dict
cursor.execute('''SELECT cd.name, cd.no,ofk.demo_id
FROM `main_table` as myo
LEFT JOIN `table1` as emt ON emt.some_id = myo.some1_id
LEFT JOIN `table2` as ofk ON ofk.id = myo.kit_id
LEFT JOIN `table3` as cd ON cd.eg_id = myo.eg_id
WHERE emt.type='test''''
result = dictfetchall(cursor)
tmp_list, tmp_dict = , {}
for res in result:
tmp_dict['name'] = res['name']
tmp_dict['no'] = res['no']
info = test(request,res['demo_id'])
tmp_dict['data1'] = info['data1']
tmp_list.append(tmp_dict.copy())
Here I am getting demo_id from query and passing that to another query using function test to fetch data therefore it is taking too much time .
Can anyone tell me how to improve the speed or include demo_id to main query and fetch data from main query ?
python mysql django
asked Jan 2 at 7:14
sachin dubeysachin dubey
584223
add a comment |
I am calling a function in loop of 3000 time . I am calling that function and getting some dictionary from that function but it takes time . I am fetching data from database which gives me approximately 3000 rows and i am looping that rows and calling function in that loop which fetch data from database and returns dictionary but it takes time .
Code:
def test(request, uni_id):
try:
Obj = get_object_or_404(tabl_name, id=uni_id)
except:
Obj = None
dict = {}
if Obj:outlet_info
dict['data1'] = Obj.id
dict['data2'] = Obj.name
dict['data3'] = Obj.eg
dict['data4'] = Obj.access
return dict
cursor.execute('''SELECT cd.name, cd.no,ofk.demo_id
FROM `main_table` as myo
LEFT JOIN `table1` as emt ON emt.some_id = myo.some1_id
LEFT JOIN `table2` as ofk ON ofk.id = myo.kit_id
LEFT JOIN `table3` as cd ON cd.eg_id = myo.eg_id
WHERE emt.type='test''''
result = dictfetchall(cursor)
tmp_list, tmp_dict = , {}
for res in result:
tmp_dict['name'] = res['name']
tmp_dict['no'] = res['no']
info = test(request,res['demo_id'])
tmp_dict['data1'] = info['data1']
tmp_list.append(tmp_dict.copy())
Here I am getting demo_id from query and passing that to another query using function test to fetch data therefore it is taking too much time .
Can anyone tell me how to improve the speed or include demo_id to main query and fetch data from main query ?
python mysql django
asked Jan 2 at 7:14
sachin dubeysachin dubey
584223
add a comment |
I am calling a function in loop of 3000 time . I am calling that function and getting some dictionary from that function but it takes time . I am fetching data from database which gives me approximately 3000 rows and i am looping that rows and calling function in that loop which fetch data from database and returns dictionary but it takes time .
Code:
def test(request, uni_id):
try:
Obj = get_object_or_404(tabl_name, id=uni_id)
except:
Obj = None
dict = {}
if Obj:outlet_info
dict['data1'] = Obj.id
dict['data2'] = Obj.name
dict['data3'] = Obj.eg
dict['data4'] = Obj.access
return dict
cursor.execute('''SELECT cd.name, cd.no,ofk.demo_id
FROM `main_table` as myo
LEFT JOIN `table1` as emt ON emt.some_id = myo.some1_id
LEFT JOIN `table2` as ofk ON ofk.id = myo.kit_id
LEFT JOIN `table3` as cd ON cd.eg_id = myo.eg_id
WHERE emt.type='test''''
result = dictfetchall(cursor)
tmp_list, tmp_dict = , {}
for res in result:
tmp_dict['name'] = res['name']
tmp_dict['no'] = res['no']
info = test(request,res['demo_id'])
tmp_dict['data1'] = info['data1']
tmp_list.append(tmp_dict.copy())
Here I am getting demo_id from query and passing that to another query using function test to fetch data therefore it is taking too much time .
Can anyone tell me how to improve the speed or include demo_id to main query and fetch data from main query ?
python mysql django
asked Jan 2 at 7:14
sachin dubeysachin dubey
584223
I am calling a function in loop of 3000 time . I am calling that function and getting some dictionary from that function but it takes time . I am fetching data from database which gives me approximately 3000 rows and i am looping that rows and calling function in that loop which fetch data from database and returns dictionary but it takes time .
Code:
def test(request, uni_id):
try:
Obj = get_object_or_404(tabl_name, id=uni_id)
except:
Obj = None
dict = {}
if Obj:outlet_info
dict['data1'] = Obj.id
dict['data2'] = Obj.name
dict['data3'] = Obj.eg
dict['data4'] = Obj.access
return dict
cursor.execute('''SELECT cd.name, cd.no,ofk.demo_id
FROM `main_table` as myo
LEFT JOIN `table1` as emt ON emt.some_id = myo.some1_id
LEFT JOIN `table2` as ofk ON ofk.id = myo.kit_id
LEFT JOIN `table3` as cd ON cd.eg_id = myo.eg_id
WHERE emt.type='test''''
result = dictfetchall(cursor)
tmp_list, tmp_dict = , {}
for res in result:
tmp_dict['name'] = res['name']
tmp_dict['no'] = res['no']
info = test(request,res['demo_id'])
tmp_dict['data1'] = info['data1']
tmp_list.append(tmp_dict.copy())
Here I am getting demo_id from query and passing that to another query using function test to fetch data therefore it is taking too much time .
Can anyone tell me how to improve the speed or include demo_id to main query and fetch data from main query ?
python mysql django
python mysql django
asked Jan 2 at 7:14
sachin dubeysachin dubey
584223
asked Jan 2 at 7:14
sachin dubeysachin dubey
584223
asked Jan 2 at 7:14
sachin dubeysachin dubey
584223
asked Jan 2 at 7:14
sachin dubeysachin dubey
584223
asked Jan 2 at 7:14
sachin dubeysachin dubey
584223
584223
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
You are making 3000 db calls(by calling get_object_or_404 3k times) that is definitely going to slow. Instead you should try fetching objects via tablemanager.filter(id__in=id_list). Prepare id_list in loop and pass that list to your test function.
answered Jan 2 at 7:18
ArpitArpit
11.3k21837
ok thanks for reply . Let me try this
– sachin dubey
Jan 2 at 7:19
also if you need only few fields try using.values(id,name,eg,success). That would be memory efficient.
– Arpit
Jan 2 at 7:21
1
@sachin dubey alternative, if you could build that entire logic intoSQL-query itself, that would be great too: just fire a single query and get final results (no intermediate processing required from your side) [I haven't been through your query and code thoroughly, so can't say if that's even possible]
– y2k-shubham
Jan 2 at 7:21
Ya i am trying to do that too @y2k-shubham
– sachin dubey
Jan 2 at 7:24
why don't you use the Django ORM notation?. Just wondering.
– imolitor
Jan 2 at 18:23
|
show 1 more comment
I included all the things in query. now i don't need to call function test
cursor.execute('''SELECT (SELECT CONCAT_WS(',',id,name,eg,access)
from tabl_name where id=ofk.demo_id) AS result
,cd.name, cd.no,ofk.demo_id
FROM `main_table` as myo
LEFT JOIN `table1` as emt ON emt.some_id = myo.some1_id
LEFT JOIN `table2` as ofk ON ofk.id = myo.kit_id
LEFT JOIN `table3` as cd ON cd.eg_id = myo.eg_id
WHERE emt.type="test" '''
result = dictfetchall(cursor) # function to get data in form of dictionary
tmp_list, tmp_dict = , {}
for res in result:
try:
info = res['result'].decode("utf-8").split(",") # converting res['result'] from bytes to string
except:
info = res['result'].split(",") # res['result'] is in string form
tmp_dict['name'] = res['name']
tmp_dict['no'] = res['no']
tmp_dict['data1'] = info[0] # id
tmp_list.append(tmp_dict.copy())
This thing makes my code 10 times faster
answered Jan 5 at 10:47
sachin dubeysachin dubey
584223
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54002581%2ffunction-takes-too-much-time-to-process-in-loop-from-database3000-times%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
You are making 3000 db calls(by calling get_object_or_404 3k times) that is definitely going to slow. Instead you should try fetching objects via tablemanager.filter(id__in=id_list). Prepare id_list in loop and pass that list to your test function.
answered Jan 2 at 7:18
ArpitArpit
11.3k21837
ok thanks for reply . Let me try this
– sachin dubey
Jan 2 at 7:19
also if you need only few fields try using.values(id,name,eg,success). That would be memory efficient.
– Arpit
Jan 2 at 7:21
1
@sachin dubey alternative, if you could build that entire logic intoSQL-query itself, that would be great too: just fire a single query and get final results (no intermediate processing required from your side) [I haven't been through your query and code thoroughly, so can't say if that's even possible]
– y2k-shubham
Jan 2 at 7:21
Ya i am trying to do that too @y2k-shubham
– sachin dubey
Jan 2 at 7:24
why don't you use the Django ORM notation?. Just wondering.
– imolitor
Jan 2 at 18:23
|
show 1 more comment
You are making 3000 db calls(by calling get_object_or_404 3k times) that is definitely going to slow. Instead you should try fetching objects via tablemanager.filter(id__in=id_list). Prepare id_list in loop and pass that list to your test function.
answered Jan 2 at 7:18
ArpitArpit
11.3k21837
ok thanks for reply . Let me try this
– sachin dubey
Jan 2 at 7:19
also if you need only few fields try using.values(id,name,eg,success). That would be memory efficient.
– Arpit
Jan 2 at 7:21
1
@sachin dubey alternative, if you could build that entire logic intoSQL-query itself, that would be great too: just fire a single query and get final results (no intermediate processing required from your side) [I haven't been through your query and code thoroughly, so can't say if that's even possible]
– y2k-shubham
Jan 2 at 7:21
Ya i am trying to do that too @y2k-shubham
– sachin dubey
Jan 2 at 7:24
why don't you use the Django ORM notation?. Just wondering.
– imolitor
Jan 2 at 18:23
|
show 1 more comment
You are making 3000 db calls(by calling get_object_or_404 3k times) that is definitely going to slow. Instead you should try fetching objects via tablemanager.filter(id__in=id_list). Prepare id_list in loop and pass that list to your test function.
answered Jan 2 at 7:18
ArpitArpit
11.3k21837
You are making 3000 db calls(by calling get_object_or_404 3k times) that is definitely going to slow. Instead you should try fetching objects via tablemanager.filter(id__in=id_list). Prepare id_list in loop and pass that list to your test function.
answered Jan 2 at 7:18
ArpitArpit
11.3k21837
answered Jan 2 at 7:18
ArpitArpit
11.3k21837
answered Jan 2 at 7:18
ArpitArpit
11.3k21837
answered Jan 2 at 7:18
ArpitArpit
11.3k21837
11.3k21837
ok thanks for reply . Let me try this
– sachin dubey
Jan 2 at 7:19
also if you need only few fields try using.values(id,name,eg,success). That would be memory efficient.
– Arpit
Jan 2 at 7:21
1
@sachin dubey alternative, if you could build that entire logic intoSQL-query itself, that would be great too: just fire a single query and get final results (no intermediate processing required from your side) [I haven't been through your query and code thoroughly, so can't say if that's even possible]
– y2k-shubham
Jan 2 at 7:21
Ya i am trying to do that too @y2k-shubham
– sachin dubey
Jan 2 at 7:24
why don't you use the Django ORM notation?. Just wondering.
– imolitor
Jan 2 at 18:23
|
show 1 more comment
ok thanks for reply . Let me try this
– sachin dubey
Jan 2 at 7:19
also if you need only few fields try using.values(id,name,eg,success). That would be memory efficient.
– Arpit
Jan 2 at 7:21
1
@sachin dubey alternative, if you could build that entire logic intoSQL-query itself, that would be great too: just fire a single query and get final results (no intermediate processing required from your side) [I haven't been through your query and code thoroughly, so can't say if that's even possible]
– y2k-shubham
Jan 2 at 7:21
Ya i am trying to do that too @y2k-shubham
– sachin dubey
Jan 2 at 7:24
why don't you use the Django ORM notation?. Just wondering.
– imolitor
Jan 2 at 18:23
ok thanks for reply . Let me try this
– sachin dubey
Jan 2 at 7:19
ok thanks for reply . Let me try this
– sachin dubey
Jan 2 at 7:19
also if you need only few fields try using
.values(id,name,eg,success). That would be memory efficient.– Arpit
Jan 2 at 7:21
also if you need only few fields try using
.values(id,name,eg,success). That would be memory efficient.– Arpit
Jan 2 at 7:21
1
1
@sachin dubey alternative, if you could build that entire logic into
SQL-query itself, that would be great too: just fire a single query and get final results (no intermediate processing required from your side) [I haven't been through your query and code thoroughly, so can't say if that's even possible]– y2k-shubham
Jan 2 at 7:21
@sachin dubey alternative, if you could build that entire logic into
SQL-query itself, that would be great too: just fire a single query and get final results (no intermediate processing required from your side) [I haven't been through your query and code thoroughly, so can't say if that's even possible]– y2k-shubham
Jan 2 at 7:21
Ya i am trying to do that too @y2k-shubham
– sachin dubey
Jan 2 at 7:24
Ya i am trying to do that too @y2k-shubham
– sachin dubey
Jan 2 at 7:24
why don't you use the Django ORM notation?. Just wondering.
– imolitor
Jan 2 at 18:23
why don't you use the Django ORM notation?. Just wondering.
– imolitor
Jan 2 at 18:23
|
show 1 more comment
I included all the things in query. now i don't need to call function test
cursor.execute('''SELECT (SELECT CONCAT_WS(',',id,name,eg,access)
from tabl_name where id=ofk.demo_id) AS result
,cd.name, cd.no,ofk.demo_id
FROM `main_table` as myo
LEFT JOIN `table1` as emt ON emt.some_id = myo.some1_id
LEFT JOIN `table2` as ofk ON ofk.id = myo.kit_id
LEFT JOIN `table3` as cd ON cd.eg_id = myo.eg_id
WHERE emt.type="test" '''
result = dictfetchall(cursor) # function to get data in form of dictionary
tmp_list, tmp_dict = , {}
for res in result:
try:
info = res['result'].decode("utf-8").split(",") # converting res['result'] from bytes to string
except:
info = res['result'].split(",") # res['result'] is in string form
tmp_dict['name'] = res['name']
tmp_dict['no'] = res['no']
tmp_dict['data1'] = info[0] # id
tmp_list.append(tmp_dict.copy())
This thing makes my code 10 times faster
answered Jan 5 at 10:47
sachin dubeysachin dubey
584223
add a comment |
I included all the things in query. now i don't need to call function test
cursor.execute('''SELECT (SELECT CONCAT_WS(',',id,name,eg,access)
from tabl_name where id=ofk.demo_id) AS result
,cd.name, cd.no,ofk.demo_id
FROM `main_table` as myo
LEFT JOIN `table1` as emt ON emt.some_id = myo.some1_id
LEFT JOIN `table2` as ofk ON ofk.id = myo.kit_id
LEFT JOIN `table3` as cd ON cd.eg_id = myo.eg_id
WHERE emt.type="test" '''
result = dictfetchall(cursor) # function to get data in form of dictionary
tmp_list, tmp_dict = , {}
for res in result:
try:
info = res['result'].decode("utf-8").split(",") # converting res['result'] from bytes to string
except:
info = res['result'].split(",") # res['result'] is in string form
tmp_dict['name'] = res['name']
tmp_dict['no'] = res['no']
tmp_dict['data1'] = info[0] # id
tmp_list.append(tmp_dict.copy())
This thing makes my code 10 times faster
answered Jan 5 at 10:47
sachin dubeysachin dubey
584223
add a comment |
I included all the things in query. now i don't need to call function test
cursor.execute('''SELECT (SELECT CONCAT_WS(',',id,name,eg,access)
from tabl_name where id=ofk.demo_id) AS result
,cd.name, cd.no,ofk.demo_id
FROM `main_table` as myo
LEFT JOIN `table1` as emt ON emt.some_id = myo.some1_id
LEFT JOIN `table2` as ofk ON ofk.id = myo.kit_id
LEFT JOIN `table3` as cd ON cd.eg_id = myo.eg_id
WHERE emt.type="test" '''
result = dictfetchall(cursor) # function to get data in form of dictionary
tmp_list, tmp_dict = , {}
for res in result:
try:
info = res['result'].decode("utf-8").split(",") # converting res['result'] from bytes to string
except:
info = res['result'].split(",") # res['result'] is in string form
tmp_dict['name'] = res['name']
tmp_dict['no'] = res['no']
tmp_dict['data1'] = info[0] # id
tmp_list.append(tmp_dict.copy())
This thing makes my code 10 times faster
answered Jan 5 at 10:47
sachin dubeysachin dubey
584223
I included all the things in query. now i don't need to call function test
cursor.execute('''SELECT (SELECT CONCAT_WS(',',id,name,eg,access)
from tabl_name where id=ofk.demo_id) AS result
,cd.name, cd.no,ofk.demo_id
FROM `main_table` as myo
LEFT JOIN `table1` as emt ON emt.some_id = myo.some1_id
LEFT JOIN `table2` as ofk ON ofk.id = myo.kit_id
LEFT JOIN `table3` as cd ON cd.eg_id = myo.eg_id
WHERE emt.type="test" '''
result = dictfetchall(cursor) # function to get data in form of dictionary
tmp_list, tmp_dict = , {}
for res in result:
try:
info = res['result'].decode("utf-8").split(",") # converting res['result'] from bytes to string
except:
info = res['result'].split(",") # res['result'] is in string form
tmp_dict['name'] = res['name']
tmp_dict['no'] = res['no']
tmp_dict['data1'] = info[0] # id
tmp_list.append(tmp_dict.copy())
This thing makes my code 10 times faster
answered Jan 5 at 10:47
sachin dubeysachin dubey
584223
edited Jan 7 at 5:31
answered Jan 5 at 10:47
sachin dubeysachin dubey
584223
answered Jan 5 at 10:47
sachin dubeysachin dubey
584223
answered Jan 5 at 10:47
sachin dubeysachin dubey
584223
584223
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54002581%2ffunction-takes-too-much-time-to-process-in-loop-from-database3000-times%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
