Mixing
Pull the data on a daily basis
I have data in my redshift cluster, What is the best way to pull the data on daily basis from redshift and create a new new table YY in redshift basis of few sql queries.
Like we have a table XX in redshift and i want to create a table in redshift from pull the top 10 rows from table XX
Create table YY as Select top 10 * from XX
amazon-redshift aws-glue
asked Jan 2 at 10:49
AtulAtul
204
add a comment |
I have data in my redshift cluster, What is the best way to pull the data on daily basis from redshift and create a new new table YY in redshift basis of few sql queries.
Like we have a table XX in redshift and i want to create a table in redshift from pull the top 10 rows from table XX
Create table YY as Select top 10 * from XX
amazon-redshift aws-glue
asked Jan 2 at 10:49
AtulAtul
204
1
there is no in built scheduler in Redshift, you can automate using a small linux server on ec2 using crontab, or airflow if your requirement is complex.
– Jon Scott
Jan 2 at 11:08
Hey Jon, Thank you for your reply. Can we do this through Lambda?
– Atul
Jan 8 at 13:33
lambda is OK except for the maximum duration (15 mins), and it is a bit more difficult to catch issues.
– Jon Scott
Jan 8 at 13:39
Thank you. I am unable to link Lambda function with Redshift. Is there any video or documents available where i can reach out and see what are functionality they use to connect lambda function with redshift?
– Atul
Jan 8 at 13:52
add a comment |
I have data in my redshift cluster, What is the best way to pull the data on daily basis from redshift and create a new new table YY in redshift basis of few sql queries.
Like we have a table XX in redshift and i want to create a table in redshift from pull the top 10 rows from table XX
Create table YY as Select top 10 * from XX
amazon-redshift aws-glue
asked Jan 2 at 10:49
AtulAtul
204
I have data in my redshift cluster, What is the best way to pull the data on daily basis from redshift and create a new new table YY in redshift basis of few sql queries.
Like we have a table XX in redshift and i want to create a table in redshift from pull the top 10 rows from table XX
Create table YY as Select top 10 * from XX
amazon-redshift aws-glue
amazon-redshift aws-glue
asked Jan 2 at 10:49
AtulAtul
204
asked Jan 2 at 10:49
AtulAtul
204
asked Jan 2 at 10:49
AtulAtul
204
asked Jan 2 at 10:49
AtulAtul
204
asked Jan 2 at 10:49
AtulAtul
204
204
1
there is no in built scheduler in Redshift, you can automate using a small linux server on ec2 using crontab, or airflow if your requirement is complex.
– Jon Scott
Jan 2 at 11:08
Hey Jon, Thank you for your reply. Can we do this through Lambda?
– Atul
Jan 8 at 13:33
lambda is OK except for the maximum duration (15 mins), and it is a bit more difficult to catch issues.
– Jon Scott
Jan 8 at 13:39
Thank you. I am unable to link Lambda function with Redshift. Is there any video or documents available where i can reach out and see what are functionality they use to connect lambda function with redshift?
– Atul
Jan 8 at 13:52
add a comment |
1
there is no in built scheduler in Redshift, you can automate using a small linux server on ec2 using crontab, or airflow if your requirement is complex.
– Jon Scott
Jan 2 at 11:08
Hey Jon, Thank you for your reply. Can we do this through Lambda?
– Atul
Jan 8 at 13:33
lambda is OK except for the maximum duration (15 mins), and it is a bit more difficult to catch issues.
– Jon Scott
Jan 8 at 13:39
Thank you. I am unable to link Lambda function with Redshift. Is there any video or documents available where i can reach out and see what are functionality they use to connect lambda function with redshift?
– Atul
Jan 8 at 13:52
1
1
there is no in built scheduler in Redshift, you can automate using a small linux server on ec2 using crontab, or airflow if your requirement is complex.
– Jon Scott
Jan 2 at 11:08
there is no in built scheduler in Redshift, you can automate using a small linux server on ec2 using crontab, or airflow if your requirement is complex.
– Jon Scott
Jan 2 at 11:08
Hey Jon, Thank you for your reply. Can we do this through Lambda?
– Atul
Jan 8 at 13:33
Hey Jon, Thank you for your reply. Can we do this through Lambda?
– Atul
Jan 8 at 13:33
lambda is OK except for the maximum duration (15 mins), and it is a bit more difficult to catch issues.
– Jon Scott
Jan 8 at 13:39
lambda is OK except for the maximum duration (15 mins), and it is a bit more difficult to catch issues.
– Jon Scott
Jan 8 at 13:39
Thank you. I am unable to link Lambda function with Redshift. Is there any video or documents available where i can reach out and see what are functionality they use to connect lambda function with redshift?
– Atul
Jan 8 at 13:52
Thank you. I am unable to link Lambda function with Redshift. Is there any video or documents available where i can reach out and see what are functionality they use to connect lambda function with redshift?
– Atul
Jan 8 at 13:52
add a comment |
1 Answer
1
active
oldest
votes
Using AWS-Glue you could schedule the Job and then write the the scripts code to do specific things. AWS-glue code could be triggered on following 3 type of events, in your case I think #1 is applicable.
- A trigger that is based on a cron schedule.
- A trigger that is event-based; for example, the successful completion of another job can start an AWS Glue job.
- A trigger that starts a job on demand.
For your case in my opinion, this should be more applicable.
I hope this should give you some pointers.
answered Jan 2 at 19:31
Red BoyRed Boy
2,26621124
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54004985%2fpull-the-data-on-a-daily-basis%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Using AWS-Glue you could schedule the Job and then write the the scripts code to do specific things. AWS-glue code could be triggered on following 3 type of events, in your case I think #1 is applicable.
- A trigger that is based on a cron schedule.
- A trigger that is event-based; for example, the successful completion of another job can start an AWS Glue job.
- A trigger that starts a job on demand.
For your case in my opinion, this should be more applicable.
I hope this should give you some pointers.
answered Jan 2 at 19:31
Red BoyRed Boy
2,26621124
add a comment |
Using AWS-Glue you could schedule the Job and then write the the scripts code to do specific things. AWS-glue code could be triggered on following 3 type of events, in your case I think #1 is applicable.
- A trigger that is based on a cron schedule.
- A trigger that is event-based; for example, the successful completion of another job can start an AWS Glue job.
- A trigger that starts a job on demand.
For your case in my opinion, this should be more applicable.
I hope this should give you some pointers.
answered Jan 2 at 19:31
Red BoyRed Boy
2,26621124
add a comment |
Using AWS-Glue you could schedule the Job and then write the the scripts code to do specific things. AWS-glue code could be triggered on following 3 type of events, in your case I think #1 is applicable.
- A trigger that is based on a cron schedule.
- A trigger that is event-based; for example, the successful completion of another job can start an AWS Glue job.
- A trigger that starts a job on demand.
For your case in my opinion, this should be more applicable.
I hope this should give you some pointers.
answered Jan 2 at 19:31
Red BoyRed Boy
2,26621124
Using AWS-Glue you could schedule the Job and then write the the scripts code to do specific things. AWS-glue code could be triggered on following 3 type of events, in your case I think #1 is applicable.
- A trigger that is based on a cron schedule.
- A trigger that is event-based; for example, the successful completion of another job can start an AWS Glue job.
- A trigger that starts a job on demand.
For your case in my opinion, this should be more applicable.
I hope this should give you some pointers.
answered Jan 2 at 19:31
Red BoyRed Boy
2,26621124
answered Jan 2 at 19:31
Red BoyRed Boy
2,26621124
answered Jan 2 at 19:31
Red BoyRed Boy
2,26621124
answered Jan 2 at 19:31
Red BoyRed Boy
2,26621124
2,26621124
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54004985%2fpull-the-data-on-a-daily-basis%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
there is no in built scheduler in Redshift, you can automate using a small linux server on ec2 using crontab, or airflow if your requirement is complex.
– Jon Scott
Jan 2 at 11:08
Hey Jon, Thank you for your reply. Can we do this through Lambda?
– Atul
Jan 8 at 13:33
lambda is OK except for the maximum duration (15 mins), and it is a bit more difficult to catch issues.
– Jon Scott
Jan 8 at 13:39
Thank you. I am unable to link Lambda function with Redshift. Is there any video or documents available where i can reach out and see what are functionality they use to connect lambda function with redshift?
– Atul
Jan 8 at 13:52