MySQLdb returns not all arguments converted with “on duplicate key update”












5















With MySQLdb package in python, I want to insert records with checking some unique keys. The method I used is executemany. The arguments are sql sentence and a tuple. But when I executed it, it raised an error which said "not all argument converted".
The codes are as following:



dData = [[u'Daniel', u'00-50-56-C0-00-12', u'Daniel']]
sql = "INSERT INTO app_network_white_black_list (biz_id, shop_id, type, mac_phone, remarks, create_time) "
"VALUES ({bsid}, {shop_id}, {type}, %s, %s, NOW()) "
"ON DUPLICATE KEY UPDATE type={type}, remarks=%s, create_time=NOW()".format(bsid=bsid, shop_id=shop_id, type=dType)
cur.executemany(sql, tuple(dData))


Someone said this is a bug. But they didn't give me a path to jump over it. Please provide a method if this is a bug.










share|improve this question




















  • 1





    possible duplicate of python executemany with "on duplicate key update"?

    – Air
    Oct 16 '14 at 18:09
















5















With MySQLdb package in python, I want to insert records with checking some unique keys. The method I used is executemany. The arguments are sql sentence and a tuple. But when I executed it, it raised an error which said "not all argument converted".
The codes are as following:



dData = [[u'Daniel', u'00-50-56-C0-00-12', u'Daniel']]
sql = "INSERT INTO app_network_white_black_list (biz_id, shop_id, type, mac_phone, remarks, create_time) "
"VALUES ({bsid}, {shop_id}, {type}, %s, %s, NOW()) "
"ON DUPLICATE KEY UPDATE type={type}, remarks=%s, create_time=NOW()".format(bsid=bsid, shop_id=shop_id, type=dType)
cur.executemany(sql, tuple(dData))


Someone said this is a bug. But they didn't give me a path to jump over it. Please provide a method if this is a bug.










share|improve this question




















  • 1





    possible duplicate of python executemany with "on duplicate key update"?

    – Air
    Oct 16 '14 at 18:09














5












5








5


4






With MySQLdb package in python, I want to insert records with checking some unique keys. The method I used is executemany. The arguments are sql sentence and a tuple. But when I executed it, it raised an error which said "not all argument converted".
The codes are as following:



dData = [[u'Daniel', u'00-50-56-C0-00-12', u'Daniel']]
sql = "INSERT INTO app_network_white_black_list (biz_id, shop_id, type, mac_phone, remarks, create_time) "
"VALUES ({bsid}, {shop_id}, {type}, %s, %s, NOW()) "
"ON DUPLICATE KEY UPDATE type={type}, remarks=%s, create_time=NOW()".format(bsid=bsid, shop_id=shop_id, type=dType)
cur.executemany(sql, tuple(dData))


Someone said this is a bug. But they didn't give me a path to jump over it. Please provide a method if this is a bug.










share|improve this question
















With MySQLdb package in python, I want to insert records with checking some unique keys. The method I used is executemany. The arguments are sql sentence and a tuple. But when I executed it, it raised an error which said "not all argument converted".
The codes are as following:



dData = [[u'Daniel', u'00-50-56-C0-00-12', u'Daniel']]
sql = "INSERT INTO app_network_white_black_list (biz_id, shop_id, type, mac_phone, remarks, create_time) "
"VALUES ({bsid}, {shop_id}, {type}, %s, %s, NOW()) "
"ON DUPLICATE KEY UPDATE type={type}, remarks=%s, create_time=NOW()".format(bsid=bsid, shop_id=shop_id, type=dType)
cur.executemany(sql, tuple(dData))


Someone said this is a bug. But they didn't give me a path to jump over it. Please provide a method if this is a bug.







python mysql-python






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Oct 13 '14 at 10:18







Hualiang Li

















asked Oct 13 '14 at 9:48









Hualiang LiHualiang Li

3014




3014








  • 1





    possible duplicate of python executemany with "on duplicate key update"?

    – Air
    Oct 16 '14 at 18:09














  • 1





    possible duplicate of python executemany with "on duplicate key update"?

    – Air
    Oct 16 '14 at 18:09








1




1





possible duplicate of python executemany with "on duplicate key update"?

– Air
Oct 16 '14 at 18:09





possible duplicate of python executemany with "on duplicate key update"?

– Air
Oct 16 '14 at 18:09












2 Answers
2






active

oldest

votes


















24





+100









What's going wrong



After checking the link in your comment below and doing some more research and testing, I was able to reproduce the error with MySQLdb versions 1.2.4b4 and 1.2.5. As explained in unubtu's answer, this has to do with the limitations of a regular expression that appears in cursors.py. The exact regular expression is slightly different in each release, probably because people keep finding cases it doesn't handle and adjusting the expression instead of looking for a better approach entirely.



What the regular expression does is try to match the VALUES ( ... ) clause of the INSERT statement and identify the beginning and end of the tuple expression it contains. If the match succeeds, executemany tries to convert the single-row insert statement template into a multiple-row insert statement so that it runs faster. I.e., instead of executing this for every row you want to insert:



INSERT INTO table
(foo, bar, ...)
VALUES
(%s, %s, ...);


It tries to rewrite the statement so that it only has to execute once:



INSERT INTO table
(foo, bar, ...)
VALUES
(1, 2, ...),
(3, 4, ...),
(5, 6, ...),
...;


The problem you're running into is that executemany assumes you only have parameter placeholders in the tuple immediately after VALUES. When you also have placeholders later on, it takes this:



INSERT INTO table
(foo, bar, ...)
VALUES
(%s, %s, ...)
ON DUPLICATE KEY UPDATE baz=%s;


And tries to rewrite it like this:



INSERT INTO table
(foo, bar, ...)
VALUES
(1, 2, ...),
(3, 4, ...),
(5, 6, ...),
...
ON DUPLICATE KEY UPDATE baz=%s;


The problem here is that MySQLdb is trying to do string formatting at the same time that it's rewriting the query. Only the VALUES ( ... ) clause needs to be rewritten, so MySQLdb tries to put all your parameters into the matching group (%s, %s, ...), not realizing that some parameters need to go into the UPDATE clause instead.



If you only send parameters for the VALUES clause to executemany, you'll avoid the TypeError but run into a different problem. Notice that the rewritten INSERT ... ON DUPLICATE UPDATE query has numeric literals in the VALUES clause, but there's still a %s placeholder in the UPDATE clause. That's going to throw a syntax error when it reaches the MySQL server.



When I first tested your sample code, I was using MySQLdb 1.2.3c1 and couldn't reproduce your problem. Amusingly, the reason that particular version of the package avoids these problems is that the regular expression is broken and doesn't match the statement at all. Since it doesn't match, executemany doesn't attempt to rewrite the query, and instead just loops through your parameters calling execute repeatedly.



What to do about it



First of all, don't go back and install 1.2.3c1 to make this work. You want to be using updated code where possible.



You could move to another package, as unubtu suggests in the linked Q&A, but that would involve some amount of adjustment and possibly changes to other code.



What I would recommend instead is to rewrite your query in a way that is more straightforward and takes advantage of the VALUES() function in your UPDATE clause. This function allows you to refer back to the values that you would have inserted in the absence of a duplicate key violation, by column name (examples are in the MySQL docs).



With that in mind, here's one way to do it:



dData = [[u'Daniel', u'00-50-56-C0-00-12', u'Daniel']]  # exact input you gave

sql = """
INSERT INTO app_network_white_black_list
(biz_id, shop_id, type, mac_phone, remarks, create_time)
VALUES
(%s, %s, %s, %s, %s, NOW())
ON DUPLICATE KEY UPDATE
type=VALUES(type), remarks=VALUES(remarks), create_time=VALUES(create_time);
""" # keep parameters in one part of the statement

# generator expression takes care of the repeated values
cur.executemany(sql, ((bsid, shop_id, dType, mac, rem) for mac, rem in dData))


This approach should work because there are no parameters in the UPDATE clause, meaning MySQLdb will be able to successfully convert the single-line insert template with parameters into a multi-line insert statement with literal values.



Some things to note:




  • You don't have to supply a tuple to executemany; any iterable is fine.

  • Multiline strings make for much more readable SQL statements in your Python code than implicitly concatenated strings; when you separate the statement from the string delimiters, it's easy to quickly grab the statement and copy it into a client application for testing.

  • If you're going to parameterize part of your query, why not parameterize all of your query? Even if only part of it is user input, it's more readable and maintainable to handle all your input values the same way.

  • That said, I didn't parameterize NOW(). My preferred approach here would be to use CURRENT_TIMESTAMP as the column default and take advantage of DEFAULT in the statement. Others might prefer to generate this value in the application and supply it as a parameter. If you're not worried about version compatibility, it's probably fine as-is.

  • If you can't avoid having parameter placeholders in the UPDATE clause – e.g., because the UPDATE value(s) can't be hard-coded in the statement or derived from the VALUES tuple – you'll have to iterate over execute instead of using executemany.






share|improve this answer


























  • Your answer is quite helpful. And I will check my source code whether there are bugs. But in another question, they figure out that this is caused by the regular express bug in MySQLdb. The following link may be more clear:link

    – Hualiang Li
    Oct 15 '14 at 8:26











  • @HualiangLi Thanks for the link, that's very helpful. You're right, actually, that it's related to the regex; see my updated answer for more details. (The solution is essentially the same.)

    – Air
    Oct 15 '14 at 18:28











  • Brilliant answer! Thanks for the solution!

    – Codewithcheese
    May 19 '15 at 13:23











  • great explanation, I'v searched all day long

    – shenyan
    Mar 24 '17 at 5:48



















-1














You have three elements in dData but only two %s placeholders for them to go into.






share|improve this answer



















  • 1





    The 3rd line in sql has another %s.

    – Hualiang Li
    Oct 13 '14 at 9:52














Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f26337065%2fmysqldb-returns-not-all-arguments-converted-with-on-duplicate-key-update%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























2 Answers
2






active

oldest

votes








2 Answers
2






active

oldest

votes









active

oldest

votes






active

oldest

votes









24





+100









What's going wrong



After checking the link in your comment below and doing some more research and testing, I was able to reproduce the error with MySQLdb versions 1.2.4b4 and 1.2.5. As explained in unubtu's answer, this has to do with the limitations of a regular expression that appears in cursors.py. The exact regular expression is slightly different in each release, probably because people keep finding cases it doesn't handle and adjusting the expression instead of looking for a better approach entirely.



What the regular expression does is try to match the VALUES ( ... ) clause of the INSERT statement and identify the beginning and end of the tuple expression it contains. If the match succeeds, executemany tries to convert the single-row insert statement template into a multiple-row insert statement so that it runs faster. I.e., instead of executing this for every row you want to insert:



INSERT INTO table
(foo, bar, ...)
VALUES
(%s, %s, ...);


It tries to rewrite the statement so that it only has to execute once:



INSERT INTO table
(foo, bar, ...)
VALUES
(1, 2, ...),
(3, 4, ...),
(5, 6, ...),
...;


The problem you're running into is that executemany assumes you only have parameter placeholders in the tuple immediately after VALUES. When you also have placeholders later on, it takes this:



INSERT INTO table
(foo, bar, ...)
VALUES
(%s, %s, ...)
ON DUPLICATE KEY UPDATE baz=%s;


And tries to rewrite it like this:



INSERT INTO table
(foo, bar, ...)
VALUES
(1, 2, ...),
(3, 4, ...),
(5, 6, ...),
...
ON DUPLICATE KEY UPDATE baz=%s;


The problem here is that MySQLdb is trying to do string formatting at the same time that it's rewriting the query. Only the VALUES ( ... ) clause needs to be rewritten, so MySQLdb tries to put all your parameters into the matching group (%s, %s, ...), not realizing that some parameters need to go into the UPDATE clause instead.



If you only send parameters for the VALUES clause to executemany, you'll avoid the TypeError but run into a different problem. Notice that the rewritten INSERT ... ON DUPLICATE UPDATE query has numeric literals in the VALUES clause, but there's still a %s placeholder in the UPDATE clause. That's going to throw a syntax error when it reaches the MySQL server.



When I first tested your sample code, I was using MySQLdb 1.2.3c1 and couldn't reproduce your problem. Amusingly, the reason that particular version of the package avoids these problems is that the regular expression is broken and doesn't match the statement at all. Since it doesn't match, executemany doesn't attempt to rewrite the query, and instead just loops through your parameters calling execute repeatedly.



What to do about it



First of all, don't go back and install 1.2.3c1 to make this work. You want to be using updated code where possible.



You could move to another package, as unubtu suggests in the linked Q&A, but that would involve some amount of adjustment and possibly changes to other code.



What I would recommend instead is to rewrite your query in a way that is more straightforward and takes advantage of the VALUES() function in your UPDATE clause. This function allows you to refer back to the values that you would have inserted in the absence of a duplicate key violation, by column name (examples are in the MySQL docs).



With that in mind, here's one way to do it:



dData = [[u'Daniel', u'00-50-56-C0-00-12', u'Daniel']]  # exact input you gave

sql = """
INSERT INTO app_network_white_black_list
(biz_id, shop_id, type, mac_phone, remarks, create_time)
VALUES
(%s, %s, %s, %s, %s, NOW())
ON DUPLICATE KEY UPDATE
type=VALUES(type), remarks=VALUES(remarks), create_time=VALUES(create_time);
""" # keep parameters in one part of the statement

# generator expression takes care of the repeated values
cur.executemany(sql, ((bsid, shop_id, dType, mac, rem) for mac, rem in dData))


This approach should work because there are no parameters in the UPDATE clause, meaning MySQLdb will be able to successfully convert the single-line insert template with parameters into a multi-line insert statement with literal values.



Some things to note:




  • You don't have to supply a tuple to executemany; any iterable is fine.

  • Multiline strings make for much more readable SQL statements in your Python code than implicitly concatenated strings; when you separate the statement from the string delimiters, it's easy to quickly grab the statement and copy it into a client application for testing.

  • If you're going to parameterize part of your query, why not parameterize all of your query? Even if only part of it is user input, it's more readable and maintainable to handle all your input values the same way.

  • That said, I didn't parameterize NOW(). My preferred approach here would be to use CURRENT_TIMESTAMP as the column default and take advantage of DEFAULT in the statement. Others might prefer to generate this value in the application and supply it as a parameter. If you're not worried about version compatibility, it's probably fine as-is.

  • If you can't avoid having parameter placeholders in the UPDATE clause – e.g., because the UPDATE value(s) can't be hard-coded in the statement or derived from the VALUES tuple – you'll have to iterate over execute instead of using executemany.






share|improve this answer


























  • Your answer is quite helpful. And I will check my source code whether there are bugs. But in another question, they figure out that this is caused by the regular express bug in MySQLdb. The following link may be more clear:link

    – Hualiang Li
    Oct 15 '14 at 8:26











  • @HualiangLi Thanks for the link, that's very helpful. You're right, actually, that it's related to the regex; see my updated answer for more details. (The solution is essentially the same.)

    – Air
    Oct 15 '14 at 18:28











  • Brilliant answer! Thanks for the solution!

    – Codewithcheese
    May 19 '15 at 13:23











  • great explanation, I'v searched all day long

    – shenyan
    Mar 24 '17 at 5:48
















24





+100









What's going wrong



After checking the link in your comment below and doing some more research and testing, I was able to reproduce the error with MySQLdb versions 1.2.4b4 and 1.2.5. As explained in unubtu's answer, this has to do with the limitations of a regular expression that appears in cursors.py. The exact regular expression is slightly different in each release, probably because people keep finding cases it doesn't handle and adjusting the expression instead of looking for a better approach entirely.



What the regular expression does is try to match the VALUES ( ... ) clause of the INSERT statement and identify the beginning and end of the tuple expression it contains. If the match succeeds, executemany tries to convert the single-row insert statement template into a multiple-row insert statement so that it runs faster. I.e., instead of executing this for every row you want to insert:



INSERT INTO table
(foo, bar, ...)
VALUES
(%s, %s, ...);


It tries to rewrite the statement so that it only has to execute once:



INSERT INTO table
(foo, bar, ...)
VALUES
(1, 2, ...),
(3, 4, ...),
(5, 6, ...),
...;


The problem you're running into is that executemany assumes you only have parameter placeholders in the tuple immediately after VALUES. When you also have placeholders later on, it takes this:



INSERT INTO table
(foo, bar, ...)
VALUES
(%s, %s, ...)
ON DUPLICATE KEY UPDATE baz=%s;


And tries to rewrite it like this:



INSERT INTO table
(foo, bar, ...)
VALUES
(1, 2, ...),
(3, 4, ...),
(5, 6, ...),
...
ON DUPLICATE KEY UPDATE baz=%s;


The problem here is that MySQLdb is trying to do string formatting at the same time that it's rewriting the query. Only the VALUES ( ... ) clause needs to be rewritten, so MySQLdb tries to put all your parameters into the matching group (%s, %s, ...), not realizing that some parameters need to go into the UPDATE clause instead.



If you only send parameters for the VALUES clause to executemany, you'll avoid the TypeError but run into a different problem. Notice that the rewritten INSERT ... ON DUPLICATE UPDATE query has numeric literals in the VALUES clause, but there's still a %s placeholder in the UPDATE clause. That's going to throw a syntax error when it reaches the MySQL server.



When I first tested your sample code, I was using MySQLdb 1.2.3c1 and couldn't reproduce your problem. Amusingly, the reason that particular version of the package avoids these problems is that the regular expression is broken and doesn't match the statement at all. Since it doesn't match, executemany doesn't attempt to rewrite the query, and instead just loops through your parameters calling execute repeatedly.



What to do about it



First of all, don't go back and install 1.2.3c1 to make this work. You want to be using updated code where possible.



You could move to another package, as unubtu suggests in the linked Q&A, but that would involve some amount of adjustment and possibly changes to other code.



What I would recommend instead is to rewrite your query in a way that is more straightforward and takes advantage of the VALUES() function in your UPDATE clause. This function allows you to refer back to the values that you would have inserted in the absence of a duplicate key violation, by column name (examples are in the MySQL docs).



With that in mind, here's one way to do it:



dData = [[u'Daniel', u'00-50-56-C0-00-12', u'Daniel']]  # exact input you gave

sql = """
INSERT INTO app_network_white_black_list
(biz_id, shop_id, type, mac_phone, remarks, create_time)
VALUES
(%s, %s, %s, %s, %s, NOW())
ON DUPLICATE KEY UPDATE
type=VALUES(type), remarks=VALUES(remarks), create_time=VALUES(create_time);
""" # keep parameters in one part of the statement

# generator expression takes care of the repeated values
cur.executemany(sql, ((bsid, shop_id, dType, mac, rem) for mac, rem in dData))


This approach should work because there are no parameters in the UPDATE clause, meaning MySQLdb will be able to successfully convert the single-line insert template with parameters into a multi-line insert statement with literal values.



Some things to note:




  • You don't have to supply a tuple to executemany; any iterable is fine.

  • Multiline strings make for much more readable SQL statements in your Python code than implicitly concatenated strings; when you separate the statement from the string delimiters, it's easy to quickly grab the statement and copy it into a client application for testing.

  • If you're going to parameterize part of your query, why not parameterize all of your query? Even if only part of it is user input, it's more readable and maintainable to handle all your input values the same way.

  • That said, I didn't parameterize NOW(). My preferred approach here would be to use CURRENT_TIMESTAMP as the column default and take advantage of DEFAULT in the statement. Others might prefer to generate this value in the application and supply it as a parameter. If you're not worried about version compatibility, it's probably fine as-is.

  • If you can't avoid having parameter placeholders in the UPDATE clause – e.g., because the UPDATE value(s) can't be hard-coded in the statement or derived from the VALUES tuple – you'll have to iterate over execute instead of using executemany.






share|improve this answer


























  • Your answer is quite helpful. And I will check my source code whether there are bugs. But in another question, they figure out that this is caused by the regular express bug in MySQLdb. The following link may be more clear:link

    – Hualiang Li
    Oct 15 '14 at 8:26











  • @HualiangLi Thanks for the link, that's very helpful. You're right, actually, that it's related to the regex; see my updated answer for more details. (The solution is essentially the same.)

    – Air
    Oct 15 '14 at 18:28











  • Brilliant answer! Thanks for the solution!

    – Codewithcheese
    May 19 '15 at 13:23











  • great explanation, I'v searched all day long

    – shenyan
    Mar 24 '17 at 5:48














24





+100







24





+100



24




+100





What's going wrong



After checking the link in your comment below and doing some more research and testing, I was able to reproduce the error with MySQLdb versions 1.2.4b4 and 1.2.5. As explained in unubtu's answer, this has to do with the limitations of a regular expression that appears in cursors.py. The exact regular expression is slightly different in each release, probably because people keep finding cases it doesn't handle and adjusting the expression instead of looking for a better approach entirely.



What the regular expression does is try to match the VALUES ( ... ) clause of the INSERT statement and identify the beginning and end of the tuple expression it contains. If the match succeeds, executemany tries to convert the single-row insert statement template into a multiple-row insert statement so that it runs faster. I.e., instead of executing this for every row you want to insert:



INSERT INTO table
(foo, bar, ...)
VALUES
(%s, %s, ...);


It tries to rewrite the statement so that it only has to execute once:



INSERT INTO table
(foo, bar, ...)
VALUES
(1, 2, ...),
(3, 4, ...),
(5, 6, ...),
...;


The problem you're running into is that executemany assumes you only have parameter placeholders in the tuple immediately after VALUES. When you also have placeholders later on, it takes this:



INSERT INTO table
(foo, bar, ...)
VALUES
(%s, %s, ...)
ON DUPLICATE KEY UPDATE baz=%s;


And tries to rewrite it like this:



INSERT INTO table
(foo, bar, ...)
VALUES
(1, 2, ...),
(3, 4, ...),
(5, 6, ...),
...
ON DUPLICATE KEY UPDATE baz=%s;


The problem here is that MySQLdb is trying to do string formatting at the same time that it's rewriting the query. Only the VALUES ( ... ) clause needs to be rewritten, so MySQLdb tries to put all your parameters into the matching group (%s, %s, ...), not realizing that some parameters need to go into the UPDATE clause instead.



If you only send parameters for the VALUES clause to executemany, you'll avoid the TypeError but run into a different problem. Notice that the rewritten INSERT ... ON DUPLICATE UPDATE query has numeric literals in the VALUES clause, but there's still a %s placeholder in the UPDATE clause. That's going to throw a syntax error when it reaches the MySQL server.



When I first tested your sample code, I was using MySQLdb 1.2.3c1 and couldn't reproduce your problem. Amusingly, the reason that particular version of the package avoids these problems is that the regular expression is broken and doesn't match the statement at all. Since it doesn't match, executemany doesn't attempt to rewrite the query, and instead just loops through your parameters calling execute repeatedly.



What to do about it



First of all, don't go back and install 1.2.3c1 to make this work. You want to be using updated code where possible.



You could move to another package, as unubtu suggests in the linked Q&A, but that would involve some amount of adjustment and possibly changes to other code.



What I would recommend instead is to rewrite your query in a way that is more straightforward and takes advantage of the VALUES() function in your UPDATE clause. This function allows you to refer back to the values that you would have inserted in the absence of a duplicate key violation, by column name (examples are in the MySQL docs).



With that in mind, here's one way to do it:



dData = [[u'Daniel', u'00-50-56-C0-00-12', u'Daniel']]  # exact input you gave

sql = """
INSERT INTO app_network_white_black_list
(biz_id, shop_id, type, mac_phone, remarks, create_time)
VALUES
(%s, %s, %s, %s, %s, NOW())
ON DUPLICATE KEY UPDATE
type=VALUES(type), remarks=VALUES(remarks), create_time=VALUES(create_time);
""" # keep parameters in one part of the statement

# generator expression takes care of the repeated values
cur.executemany(sql, ((bsid, shop_id, dType, mac, rem) for mac, rem in dData))


This approach should work because there are no parameters in the UPDATE clause, meaning MySQLdb will be able to successfully convert the single-line insert template with parameters into a multi-line insert statement with literal values.



Some things to note:




  • You don't have to supply a tuple to executemany; any iterable is fine.

  • Multiline strings make for much more readable SQL statements in your Python code than implicitly concatenated strings; when you separate the statement from the string delimiters, it's easy to quickly grab the statement and copy it into a client application for testing.

  • If you're going to parameterize part of your query, why not parameterize all of your query? Even if only part of it is user input, it's more readable and maintainable to handle all your input values the same way.

  • That said, I didn't parameterize NOW(). My preferred approach here would be to use CURRENT_TIMESTAMP as the column default and take advantage of DEFAULT in the statement. Others might prefer to generate this value in the application and supply it as a parameter. If you're not worried about version compatibility, it's probably fine as-is.

  • If you can't avoid having parameter placeholders in the UPDATE clause – e.g., because the UPDATE value(s) can't be hard-coded in the statement or derived from the VALUES tuple – you'll have to iterate over execute instead of using executemany.






share|improve this answer















What's going wrong



After checking the link in your comment below and doing some more research and testing, I was able to reproduce the error with MySQLdb versions 1.2.4b4 and 1.2.5. As explained in unubtu's answer, this has to do with the limitations of a regular expression that appears in cursors.py. The exact regular expression is slightly different in each release, probably because people keep finding cases it doesn't handle and adjusting the expression instead of looking for a better approach entirely.



What the regular expression does is try to match the VALUES ( ... ) clause of the INSERT statement and identify the beginning and end of the tuple expression it contains. If the match succeeds, executemany tries to convert the single-row insert statement template into a multiple-row insert statement so that it runs faster. I.e., instead of executing this for every row you want to insert:



INSERT INTO table
(foo, bar, ...)
VALUES
(%s, %s, ...);


It tries to rewrite the statement so that it only has to execute once:



INSERT INTO table
(foo, bar, ...)
VALUES
(1, 2, ...),
(3, 4, ...),
(5, 6, ...),
...;


The problem you're running into is that executemany assumes you only have parameter placeholders in the tuple immediately after VALUES. When you also have placeholders later on, it takes this:



INSERT INTO table
(foo, bar, ...)
VALUES
(%s, %s, ...)
ON DUPLICATE KEY UPDATE baz=%s;


And tries to rewrite it like this:



INSERT INTO table
(foo, bar, ...)
VALUES
(1, 2, ...),
(3, 4, ...),
(5, 6, ...),
...
ON DUPLICATE KEY UPDATE baz=%s;


The problem here is that MySQLdb is trying to do string formatting at the same time that it's rewriting the query. Only the VALUES ( ... ) clause needs to be rewritten, so MySQLdb tries to put all your parameters into the matching group (%s, %s, ...), not realizing that some parameters need to go into the UPDATE clause instead.



If you only send parameters for the VALUES clause to executemany, you'll avoid the TypeError but run into a different problem. Notice that the rewritten INSERT ... ON DUPLICATE UPDATE query has numeric literals in the VALUES clause, but there's still a %s placeholder in the UPDATE clause. That's going to throw a syntax error when it reaches the MySQL server.



When I first tested your sample code, I was using MySQLdb 1.2.3c1 and couldn't reproduce your problem. Amusingly, the reason that particular version of the package avoids these problems is that the regular expression is broken and doesn't match the statement at all. Since it doesn't match, executemany doesn't attempt to rewrite the query, and instead just loops through your parameters calling execute repeatedly.



What to do about it



First of all, don't go back and install 1.2.3c1 to make this work. You want to be using updated code where possible.



You could move to another package, as unubtu suggests in the linked Q&A, but that would involve some amount of adjustment and possibly changes to other code.



What I would recommend instead is to rewrite your query in a way that is more straightforward and takes advantage of the VALUES() function in your UPDATE clause. This function allows you to refer back to the values that you would have inserted in the absence of a duplicate key violation, by column name (examples are in the MySQL docs).



With that in mind, here's one way to do it:



dData = [[u'Daniel', u'00-50-56-C0-00-12', u'Daniel']]  # exact input you gave

sql = """
INSERT INTO app_network_white_black_list
(biz_id, shop_id, type, mac_phone, remarks, create_time)
VALUES
(%s, %s, %s, %s, %s, NOW())
ON DUPLICATE KEY UPDATE
type=VALUES(type), remarks=VALUES(remarks), create_time=VALUES(create_time);
""" # keep parameters in one part of the statement

# generator expression takes care of the repeated values
cur.executemany(sql, ((bsid, shop_id, dType, mac, rem) for mac, rem in dData))


This approach should work because there are no parameters in the UPDATE clause, meaning MySQLdb will be able to successfully convert the single-line insert template with parameters into a multi-line insert statement with literal values.



Some things to note:




  • You don't have to supply a tuple to executemany; any iterable is fine.

  • Multiline strings make for much more readable SQL statements in your Python code than implicitly concatenated strings; when you separate the statement from the string delimiters, it's easy to quickly grab the statement and copy it into a client application for testing.

  • If you're going to parameterize part of your query, why not parameterize all of your query? Even if only part of it is user input, it's more readable and maintainable to handle all your input values the same way.

  • That said, I didn't parameterize NOW(). My preferred approach here would be to use CURRENT_TIMESTAMP as the column default and take advantage of DEFAULT in the statement. Others might prefer to generate this value in the application and supply it as a parameter. If you're not worried about version compatibility, it's probably fine as-is.

  • If you can't avoid having parameter placeholders in the UPDATE clause – e.g., because the UPDATE value(s) can't be hard-coded in the statement or derived from the VALUES tuple – you'll have to iterate over execute instead of using executemany.







share|improve this answer














share|improve this answer



share|improve this answer








edited May 23 '17 at 12:17









Community

11




11










answered Oct 14 '14 at 23:25









AirAir

5,39923768




5,39923768













  • Your answer is quite helpful. And I will check my source code whether there are bugs. But in another question, they figure out that this is caused by the regular express bug in MySQLdb. The following link may be more clear:link

    – Hualiang Li
    Oct 15 '14 at 8:26











  • @HualiangLi Thanks for the link, that's very helpful. You're right, actually, that it's related to the regex; see my updated answer for more details. (The solution is essentially the same.)

    – Air
    Oct 15 '14 at 18:28











  • Brilliant answer! Thanks for the solution!

    – Codewithcheese
    May 19 '15 at 13:23











  • great explanation, I'v searched all day long

    – shenyan
    Mar 24 '17 at 5:48



















  • Your answer is quite helpful. And I will check my source code whether there are bugs. But in another question, they figure out that this is caused by the regular express bug in MySQLdb. The following link may be more clear:link

    – Hualiang Li
    Oct 15 '14 at 8:26











  • @HualiangLi Thanks for the link, that's very helpful. You're right, actually, that it's related to the regex; see my updated answer for more details. (The solution is essentially the same.)

    – Air
    Oct 15 '14 at 18:28











  • Brilliant answer! Thanks for the solution!

    – Codewithcheese
    May 19 '15 at 13:23











  • great explanation, I'v searched all day long

    – shenyan
    Mar 24 '17 at 5:48

















Your answer is quite helpful. And I will check my source code whether there are bugs. But in another question, they figure out that this is caused by the regular express bug in MySQLdb. The following link may be more clear:link

– Hualiang Li
Oct 15 '14 at 8:26





Your answer is quite helpful. And I will check my source code whether there are bugs. But in another question, they figure out that this is caused by the regular express bug in MySQLdb. The following link may be more clear:link

– Hualiang Li
Oct 15 '14 at 8:26













@HualiangLi Thanks for the link, that's very helpful. You're right, actually, that it's related to the regex; see my updated answer for more details. (The solution is essentially the same.)

– Air
Oct 15 '14 at 18:28





@HualiangLi Thanks for the link, that's very helpful. You're right, actually, that it's related to the regex; see my updated answer for more details. (The solution is essentially the same.)

– Air
Oct 15 '14 at 18:28













Brilliant answer! Thanks for the solution!

– Codewithcheese
May 19 '15 at 13:23





Brilliant answer! Thanks for the solution!

– Codewithcheese
May 19 '15 at 13:23













great explanation, I'v searched all day long

– shenyan
Mar 24 '17 at 5:48





great explanation, I'v searched all day long

– shenyan
Mar 24 '17 at 5:48













-1














You have three elements in dData but only two %s placeholders for them to go into.






share|improve this answer



















  • 1





    The 3rd line in sql has another %s.

    – Hualiang Li
    Oct 13 '14 at 9:52


















-1














You have three elements in dData but only two %s placeholders for them to go into.






share|improve this answer



















  • 1





    The 3rd line in sql has another %s.

    – Hualiang Li
    Oct 13 '14 at 9:52
















-1












-1








-1







You have three elements in dData but only two %s placeholders for them to go into.






share|improve this answer













You have three elements in dData but only two %s placeholders for them to go into.







share|improve this answer












share|improve this answer



share|improve this answer










answered Oct 13 '14 at 9:51









Daniel RosemanDaniel Roseman

459k42595654




459k42595654








  • 1





    The 3rd line in sql has another %s.

    – Hualiang Li
    Oct 13 '14 at 9:52
















  • 1





    The 3rd line in sql has another %s.

    – Hualiang Li
    Oct 13 '14 at 9:52










1




1





The 3rd line in sql has another %s.

– Hualiang Li
Oct 13 '14 at 9:52







The 3rd line in sql has another %s.

– Hualiang Li
Oct 13 '14 at 9:52




















draft saved

draft discarded




















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f26337065%2fmysqldb-returns-not-all-arguments-converted-with-on-duplicate-key-update%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

MongoDB - Not Authorized To Execute Command

in spring boot 2.1 many test slices are not allowed anymore due to multiple @BootstrapWith

Npm cannot find a required file even through it is in the searched directory