Airflow orchestration best practices











up vote
0
down vote

favorite












I started to use Airflow to schedule jobs in our company, and I am wondering about its best practices.



Is it recommended to put all my tasks in one DAG ? If not, what is the right middle between one Dag and multiple Dags?



Our scheduled DAG's execute collects, transforms, exports and some other computing programs. So we will continuously have new tasks to add.






airflow orchestration







share|improve this question


























    up vote
    0
    down vote

    favorite












    I started to use Airflow to schedule jobs in our company, and I am wondering about its best practices.



    Is it recommended to put all my tasks in one DAG ? If not, what is the right middle between one Dag and multiple Dags?



    Our scheduled DAG's execute collects, transforms, exports and some other computing programs. So we will continuously have new tasks to add.






    airflow orchestration







    share|improve this question
























      up vote
      0
      down vote

      favorite









      up vote
      0
      down vote

      favorite











      I started to use Airflow to schedule jobs in our company, and I am wondering about its best practices.



      Is it recommended to put all my tasks in one DAG ? If not, what is the right middle between one Dag and multiple Dags?



      Our scheduled DAG's execute collects, transforms, exports and some other computing programs. So we will continuously have new tasks to add.






      airflow orchestration







      share|improve this question













      I started to use Airflow to schedule jobs in our company, and I am wondering about its best practices.



      Is it recommended to put all my tasks in one DAG ? If not, what is the right middle between one Dag and multiple Dags?



      Our scheduled DAG's execute collects, transforms, exports and some other computing programs. So we will continuously have new tasks to add.






      airflow orchestration




      airflow orchestration






      share|improve this question













      share|improve this question











      share|improve this question




      share|improve this question










      asked 2 days ago









      Ismail Addou

      28117




      28117
























          1 Answer
          1






          active

          oldest

          votes

















          up vote
          0
          down vote



          accepted










          Generally, one python file consists of a single DAG with multiple task. This is because it is the logical grouping of the tasks.



          If you have multiple DAG that have dependencies you can use TriggerDagRunOperator at the end of DAG1. This would trigger DAG2 (separate DAG file) if all tasks in DAG1 succeeds.



          An example of this is:



          DAG1: https://github.com/apache/incubator-airflow/blob/master/airflow/example_dags/example_trigger_controller_dag.py



          DAG2: https://github.com/apache/incubator-airflow/blob/master/airflow/example_dags/example_trigger_target_dag.py






          share|improve this answer





















            Your Answer






            StackExchange.ifUsing("editor", function () {
            StackExchange.using("externalEditor", function () {
            StackExchange.using("snippets", function () {
            StackExchange.snippets.init();
            });
            });
            }, "code-snippets");

            StackExchange.ready(function() {
            var channelOptions = {
            tags: "".split(" "),
            id: "1"
            };
            initTagRenderer("".split(" "), "".split(" "), channelOptions);

            StackExchange.using("externalEditor", function() {
            // Have to fire editor after snippets, if snippets enabled
            if (StackExchange.settings.snippets.snippetsEnabled) {
            StackExchange.using("snippets", function() {
            createEditor();
            });
            }
            else {
            createEditor();
            }
            });

            function createEditor() {
            StackExchange.prepareEditor({
            heartbeatType: 'answer',
            convertImagesToLinks: true,
            noModals: true,
            showLowRepImageUploadWarning: true,
            reputationToPostImages: 10,
            bindNavPrevention: true,
            postfix: "",
            imageUploader: {
            brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
            contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
            allowUrls: true
            },
            onDemand: true,
            discardSelector: ".discard-answer"
            ,immediatelyShowMarkdownHelp:true
            });


            }
            });














             

            draft saved


            draft discarded


















            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53365900%2fairflow-orchestration-best-practices%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown

























            1 Answer
            1






            active

            oldest

            votes








            1 Answer
            1






            active

            oldest

            votes









            active

            oldest

            votes






            active

            oldest

            votes








            up vote
            0
            down vote



            accepted










            Generally, one python file consists of a single DAG with multiple task. This is because it is the logical grouping of the tasks.



            If you have multiple DAG that have dependencies you can use TriggerDagRunOperator at the end of DAG1. This would trigger DAG2 (separate DAG file) if all tasks in DAG1 succeeds.



            An example of this is:



            DAG1: https://github.com/apache/incubator-airflow/blob/master/airflow/example_dags/example_trigger_controller_dag.py



            DAG2: https://github.com/apache/incubator-airflow/blob/master/airflow/example_dags/example_trigger_target_dag.py






            share|improve this answer

























              up vote
              0
              down vote



              accepted










              Generally, one python file consists of a single DAG with multiple task. This is because it is the logical grouping of the tasks.



              If you have multiple DAG that have dependencies you can use TriggerDagRunOperator at the end of DAG1. This would trigger DAG2 (separate DAG file) if all tasks in DAG1 succeeds.



              An example of this is:



              DAG1: https://github.com/apache/incubator-airflow/blob/master/airflow/example_dags/example_trigger_controller_dag.py



              DAG2: https://github.com/apache/incubator-airflow/blob/master/airflow/example_dags/example_trigger_target_dag.py






              share|improve this answer























                up vote
                0
                down vote



                accepted







                up vote
                0
                down vote



                accepted






                Generally, one python file consists of a single DAG with multiple task. This is because it is the logical grouping of the tasks.



                If you have multiple DAG that have dependencies you can use TriggerDagRunOperator at the end of DAG1. This would trigger DAG2 (separate DAG file) if all tasks in DAG1 succeeds.



                An example of this is:



                DAG1: https://github.com/apache/incubator-airflow/blob/master/airflow/example_dags/example_trigger_controller_dag.py



                DAG2: https://github.com/apache/incubator-airflow/blob/master/airflow/example_dags/example_trigger_target_dag.py






                share|improve this answer












                Generally, one python file consists of a single DAG with multiple task. This is because it is the logical grouping of the tasks.



                If you have multiple DAG that have dependencies you can use TriggerDagRunOperator at the end of DAG1. This would trigger DAG2 (separate DAG file) if all tasks in DAG1 succeeds.



                An example of this is:



                DAG1: https://github.com/apache/incubator-airflow/blob/master/airflow/example_dags/example_trigger_controller_dag.py



                DAG2: https://github.com/apache/incubator-airflow/blob/master/airflow/example_dags/example_trigger_target_dag.py







                share|improve this answer












                share|improve this answer



                share|improve this answer










                answered 2 days ago









                kaxil

                1,853526




                1,853526






























                     

                    draft saved


                    draft discarded



















































                     


                    draft saved


                    draft discarded














                    StackExchange.ready(
                    function () {
                    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53365900%2fairflow-orchestration-best-practices%23new-answer', 'question_page');
                    }
                    );

                    Post as a guest















                    Required, but never shown





















































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown

































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown







                    Popular posts from this blog

                    android studio warns about leanback feature tag usage required on manifest while using Unity exported app?

                    Image
                    .everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box; } 0 This is a simple project showing cubes on Unity, exported to android project and imported on Android studio. On maifest file it showed following warning: Expecting <uses-feature android:name="android.software.leanback" android:required="false" /> tag I have no information about it. And also on default emptyview android project does not use/mention this feature. I know it is required by TV's and such but why its not added by default? Why is this? And Should I concern about it? android unity3d
                    Read more

                    SQL update select statement

                    Image
                    0 I'm trying to update an entry in my database (postgresql). I'm having a problem with updating the query: INSERT INTO prices (price, product_id) SELECT product_price, commn_id FROM products_temp as prod WHERE NOT EXISTS (SELECT * FROM prices od WHERE prod.commn_id = od.product_id) OR prod.product_price != ( SELECT price FROM prices as p WHERE p.product_id = prod.commn_id order by p.end desc LIMIT 1 ) Query works fine when i delete: OR prod.product_price != ( SELECT price FROM prices as p WHERE p.product_id = prod.commn_id order by p.end desc LIMIT 1 ) So it seems to me that it's looping through this operation. My question is, how can I fix it? sql postgresql
                    Read more

                    'app-layout' is not a known element: how to share Component with different Modules

                    Image
                    0 My application was working fine until I decided to go with lazy loading: So, my shared component looks like this: import { Component, Renderer2 } from '@angular/core'; export interface FormModel { captcha?: string; } @Component({ selector: 'app-layout', templateUrl: './app-layout.component.html', styleUrls: ['./app-layout.css'] }) export class FullLayoutComponent { } app-layout.component.ts I am using this component in my Another component say school-home.component.html like this: <app-layout> </app-layout> and it was working fine until I created a Module school-home.module.ts like this: import { NgModule } from '@angular/core'; import { CommonModule } from '@angular/common'; import { SchoolhomeRoutingModule
                    Read more