Mixing
Hive query optimization settings- when not to use?
Trying to improve the hive query speed based on the techniques. Below config changes increases speed and want to use these settings for all the queries i execute. But i wanted some input on if these settings will impact inversely if used across all queries.
set hive.vectorized.execution.enabled = true;
set hive.vectorized.execution.reduce.enabled = true;
Vectorized query execution improves performance of operations like scans,
aggregations, filters and joins, by performing them in
batches of 1024 rows at once instead of single row each time.
Introduced in Hive 0.13, this feature significantly improves
query execution time.
set hive.cbo.enable=true;
set hive.compute.query.using.stats=true;
set hive.stats.fetch.column.stats=true;
set hive.stats.fetch.partition.stats=true;
analyze table tweets compute statistics for columns;
Enable cost based optimization(cbo)
set hive.execution.engine=tez;
use tez engine
hive
asked Nov 21 '18 at 9:42
sjdsjd
3331520
add a comment |
Trying to improve the hive query speed based on the techniques. Below config changes increases speed and want to use these settings for all the queries i execute. But i wanted some input on if these settings will impact inversely if used across all queries.
set hive.vectorized.execution.enabled = true;
set hive.vectorized.execution.reduce.enabled = true;
Vectorized query execution improves performance of operations like scans,
aggregations, filters and joins, by performing them in
batches of 1024 rows at once instead of single row each time.
Introduced in Hive 0.13, this feature significantly improves
query execution time.
set hive.cbo.enable=true;
set hive.compute.query.using.stats=true;
set hive.stats.fetch.column.stats=true;
set hive.stats.fetch.partition.stats=true;
analyze table tweets compute statistics for columns;
Enable cost based optimization(cbo)
set hive.execution.engine=tez;
use tez engine
hive
asked Nov 21 '18 at 9:42
sjdsjd
3331520
2
Column statistics can be expensive to fetch. And you do not need it for all tables. Everything else is good to keep for all queries
– leftjoin
Nov 21 '18 at 12:09
@leftjoin Thanks !!. yes analyze table was taking some time to complete.
– sjd
Nov 22 '18 at 7:03
Please have a look at this answer also: stackoverflow.com/a/40783621/2700344
– leftjoin
Nov 22 '18 at 7:16
add a comment |
Trying to improve the hive query speed based on the techniques. Below config changes increases speed and want to use these settings for all the queries i execute. But i wanted some input on if these settings will impact inversely if used across all queries.
set hive.vectorized.execution.enabled = true;
set hive.vectorized.execution.reduce.enabled = true;
Vectorized query execution improves performance of operations like scans,
aggregations, filters and joins, by performing them in
batches of 1024 rows at once instead of single row each time.
Introduced in Hive 0.13, this feature significantly improves
query execution time.
set hive.cbo.enable=true;
set hive.compute.query.using.stats=true;
set hive.stats.fetch.column.stats=true;
set hive.stats.fetch.partition.stats=true;
analyze table tweets compute statistics for columns;
Enable cost based optimization(cbo)
set hive.execution.engine=tez;
use tez engine
hive
asked Nov 21 '18 at 9:42
sjdsjd
3331520
Trying to improve the hive query speed based on the techniques. Below config changes increases speed and want to use these settings for all the queries i execute. But i wanted some input on if these settings will impact inversely if used across all queries.
set hive.vectorized.execution.enabled = true;
set hive.vectorized.execution.reduce.enabled = true;
Vectorized query execution improves performance of operations like scans,
aggregations, filters and joins, by performing them in
batches of 1024 rows at once instead of single row each time.
Introduced in Hive 0.13, this feature significantly improves
query execution time.
set hive.cbo.enable=true;
set hive.compute.query.using.stats=true;
set hive.stats.fetch.column.stats=true;
set hive.stats.fetch.partition.stats=true;
analyze table tweets compute statistics for columns;
Enable cost based optimization(cbo)
set hive.execution.engine=tez;
use tez engine
hive
hive
asked Nov 21 '18 at 9:42
sjdsjd
3331520
asked Nov 21 '18 at 9:42
sjdsjd
3331520
asked Nov 21 '18 at 9:42
sjdsjd
3331520
asked Nov 21 '18 at 9:42
sjdsjd
3331520
asked Nov 21 '18 at 9:42
sjdsjd
3331520
3331520
2
Column statistics can be expensive to fetch. And you do not need it for all tables. Everything else is good to keep for all queries
– leftjoin
Nov 21 '18 at 12:09
@leftjoin Thanks !!. yes analyze table was taking some time to complete.
– sjd
Nov 22 '18 at 7:03
Please have a look at this answer also: stackoverflow.com/a/40783621/2700344
– leftjoin
Nov 22 '18 at 7:16
add a comment |
2
Column statistics can be expensive to fetch. And you do not need it for all tables. Everything else is good to keep for all queries
– leftjoin
Nov 21 '18 at 12:09
@leftjoin Thanks !!. yes analyze table was taking some time to complete.
– sjd
Nov 22 '18 at 7:03
Please have a look at this answer also: stackoverflow.com/a/40783621/2700344
– leftjoin
Nov 22 '18 at 7:16
2
2
Column statistics can be expensive to fetch. And you do not need it for all tables. Everything else is good to keep for all queries
– leftjoin
Nov 21 '18 at 12:09
Column statistics can be expensive to fetch. And you do not need it for all tables. Everything else is good to keep for all queries
– leftjoin
Nov 21 '18 at 12:09
@leftjoin Thanks !!. yes analyze table was taking some time to complete.
– sjd
Nov 22 '18 at 7:03
@leftjoin Thanks !!. yes analyze table was taking some time to complete.
– sjd
Nov 22 '18 at 7:03
Please have a look at this answer also: stackoverflow.com/a/40783621/2700344
– leftjoin
Nov 22 '18 at 7:16
Please have a look at this answer also: stackoverflow.com/a/40783621/2700344
– leftjoin
Nov 22 '18 at 7:16
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53409157%2fhive-query-optimization-settings-when-not-to-use%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53409157%2fhive-query-optimization-settings-when-not-to-use%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
2
Column statistics can be expensive to fetch. And you do not need it for all tables. Everything else is good to keep for all queries
– leftjoin
Nov 21 '18 at 12:09
@leftjoin Thanks !!. yes analyze table was taking some time to complete.
– sjd
Nov 22 '18 at 7:03
Please have a look at this answer also: stackoverflow.com/a/40783621/2700344
– leftjoin
Nov 22 '18 at 7:16