Mixing
Airflow | DAG | Copy dags written by Dev team to airflow server continuously
How can I continuously copy Dags written by Development team to airflow(/home/ec2-user/airflow/dags) server. I am using Ansible currently to install Airflow.
Current approach which I am using:-
I am asking dev team to push all the dags to Bit Bucket and I have written a ansible script which picks up the dags and put in Airflow server. This is currently handled by Jenkins job which when triggered will copy to server.
amazon-web-services airflow amazon-emr directed-acyclic-graphs
asked Jan 2 at 12:07
KallyKally
338726
add a comment |
How can I continuously copy Dags written by Development team to airflow(/home/ec2-user/airflow/dags) server. I am using Ansible currently to install Airflow.
Current approach which I am using:-
I am asking dev team to push all the dags to Bit Bucket and I have written a ansible script which picks up the dags and put in Airflow server. This is currently handled by Jenkins job which when triggered will copy to server.
amazon-web-services airflow amazon-emr directed-acyclic-graphs
asked Jan 2 at 12:07
KallyKally
338726
Why there is anansibletag? Ansible could not do anything 'continuously', it's not a server.
– George Shuklin
Jan 2 at 14:17
@GeorgeShuklin ok.. I thought there might be any solution via Ansible. So tagged it.
– Kally
Jan 2 at 14:35
add a comment |
How can I continuously copy Dags written by Development team to airflow(/home/ec2-user/airflow/dags) server. I am using Ansible currently to install Airflow.
Current approach which I am using:-
I am asking dev team to push all the dags to Bit Bucket and I have written a ansible script which picks up the dags and put in Airflow server. This is currently handled by Jenkins job which when triggered will copy to server.
amazon-web-services airflow amazon-emr directed-acyclic-graphs
asked Jan 2 at 12:07
KallyKally
338726
How can I continuously copy Dags written by Development team to airflow(/home/ec2-user/airflow/dags) server. I am using Ansible currently to install Airflow.
Current approach which I am using:-
I am asking dev team to push all the dags to Bit Bucket and I have written a ansible script which picks up the dags and put in Airflow server. This is currently handled by Jenkins job which when triggered will copy to server.
amazon-web-services airflow amazon-emr directed-acyclic-graphs
amazon-web-services airflow amazon-emr directed-acyclic-graphs
asked Jan 2 at 12:07
KallyKally
338726
asked Jan 2 at 12:07
KallyKally
338726
edited Jan 3 at 11:37
Kally
asked Jan 2 at 12:07
KallyKally
338726
asked Jan 2 at 12:07
KallyKally
338726
asked Jan 2 at 12:07
KallyKally
338726
338726
Why there is anansibletag? Ansible could not do anything 'continuously', it's not a server.
– George Shuklin
Jan 2 at 14:17
@GeorgeShuklin ok.. I thought there might be any solution via Ansible. So tagged it.
– Kally
Jan 2 at 14:35
add a comment |
Why there is anansibletag? Ansible could not do anything 'continuously', it's not a server.
– George Shuklin
Jan 2 at 14:17
@GeorgeShuklin ok.. I thought there might be any solution via Ansible. So tagged it.
– Kally
Jan 2 at 14:35
Why there is an
ansible tag? Ansible could not do anything 'continuously', it's not a server.– George Shuklin
Jan 2 at 14:17
Why there is an
ansible tag? Ansible could not do anything 'continuously', it's not a server.– George Shuklin
Jan 2 at 14:17
@GeorgeShuklin ok.. I thought there might be any solution via Ansible. So tagged it.
– Kally
Jan 2 at 14:35
@GeorgeShuklin ok.. I thought there might be any solution via Ansible. So tagged it.
– Kally
Jan 2 at 14:35
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54006079%2fairflow-dag-copy-dags-written-by-dev-team-to-airflow-server-continuously%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54006079%2fairflow-dag-copy-dags-written-by-dev-team-to-airflow-server-continuously%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Why there is an
ansibletag? Ansible could not do anything 'continuously', it's not a server.– George Shuklin
Jan 2 at 14:17
@GeorgeShuklin ok.. I thought there might be any solution via Ansible. So tagged it.
– Kally
Jan 2 at 14:35