Is there any way to find the amount of time taken by any random task in DAG?

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}

I have a requirement of finding the amount of time by any particular task in DAG.

For example:

SampleTask=DummyOperator(

    task_id='SampleTask',

    dag=dag)

Now I want to know how the time taken by the above task based on its task_id, and in the same way for other tasks as well.

edited Jan 3 at 7:40

asked Jan 3 at 7:25

Ashish Kumar

302114

Im not familiar with subject, but would decorators be a solution?

– NEGR KITAEC
Jan 3 at 7:42

Of course @medvedev1088's answer tells you how to do it programmatically, but for a quick visual, Airflow Web UI's Gantt View comes very handy

– y2k-shubham
Feb 13 at 10:37

add a comment |

I have a requirement of finding the amount of time by any particular task in DAG.

For example:

SampleTask=DummyOperator(

    task_id='SampleTask',

    dag=dag)

Now I want to know how the time taken by the above task based on its task_id, and in the same way for other tasks as well.

edited Jan 3 at 7:40

asked Jan 3 at 7:25

Ashish Kumar

302114

Im not familiar with subject, but would decorators be a solution?

– NEGR KITAEC
Jan 3 at 7:42

Of course @medvedev1088's answer tells you how to do it programmatically, but for a quick visual, Airflow Web UI's Gantt View comes very handy

– y2k-shubham
Feb 13 at 10:37

add a comment |

I have a requirement of finding the amount of time by any particular task in DAG.

For example:

SampleTask=DummyOperator(

    task_id='SampleTask',

    dag=dag)

Now I want to know how the time taken by the above task based on its task_id, and in the same way for other tasks as well.

edited Jan 3 at 7:40

asked Jan 3 at 7:25

Ashish Kumar

302114

I have a requirement of finding the amount of time by any particular task in DAG.

For example:

SampleTask=DummyOperator(

    task_id='SampleTask',

    dag=dag)

Now I want to know how the time taken by the above task based on its task_id, and in the same way for other tasks as well.

python google-cloud-platform airflow directed-acyclic-graphs

edited Jan 3 at 7:40

asked Jan 3 at 7:25

Ashish Kumar

302114

edited Jan 3 at 7:40

asked Jan 3 at 7:25

Ashish Kumar

302114

edited Jan 3 at 7:40

asked Jan 3 at 7:25

Ashish Kumar

302114

asked Jan 3 at 7:25

Ashish Kumar

302114

asked Jan 3 at 7:25

Ashish Kumar

302114

Im not familiar with subject, but would decorators be a solution?

– NEGR KITAEC
Jan 3 at 7:42

Of course @medvedev1088's answer tells you how to do it programmatically, but for a quick visual, Airflow Web UI's Gantt View comes very handy

– y2k-shubham
Feb 13 at 10:37

add a comment |

Im not familiar with subject, but would decorators be a solution?

– NEGR KITAEC
Jan 3 at 7:42

Of course @medvedev1088's answer tells you how to do it programmatically, but for a quick visual, Airflow Web UI's Gantt View comes very handy

– y2k-shubham
Feb 13 at 10:37

Im not familiar with subject, but would decorators be a solution?

– NEGR KITAEC
Jan 3 at 7:42

Of course @medvedev1088's answer tells you how to do it programmatically, but for a quick visual, Airflow Web UI's Gantt View comes very handy

– y2k-shubham
Feb 13 at 10:37

add a comment |

1 Answer
1

active

oldest

votes

If you go to Data Profiling > Ad Hoc Query from the Airflow UI, and select the airflow_db database, you can issue this query:

select execution_date, duration from task_instance where task_id = 'SampleTask'

The duration column holds the number of seconds the task ran.

answered Jan 3 at 8:51

medvedev1088

2,6151335

yeah that works @medvedev1088, can you tell me how to use this inside a current DAG? My requirement is to store the value of the duration into a variable inside a DAG. Also this gives me the duration of all the number of times it has run in the past, I want only the duration of present task after completion.

– Ashish Kumar
Jan 3 at 10:00

For this use case I think you can wrap you operator executable into a function that measures start and end time. But it will be a different duration than what's stored in the duration column in the database.

– medvedev1088
Jan 3 at 10:06

@nedvedev1088 is there any way/operator to query airflow_db from DAG?

– Ashish Kumar
Jan 3 at 11:07

I’m not sure but even if there is it won’t help because duration is set when task is finished, but you need to access it while the task is running if I understood you correctly

– medvedev1088
Jan 3 at 11:09

1

There should be a connection called airflow_db. You can use MySqlHook or PostgresHook with conn_id='airflow_db' to access the database.

– medvedev1088
Jan 3 at 11:46

|
show 3 more comments

Your Answer

StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});

}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54018003%2fis-there-any-way-to-find-the-amount-of-time-taken-by-any-random-task-in-dag%23new-answer', 'question_page');
}
);

Post as a guest

Name

Required, but never shown

1 Answer
1

active

oldest

votes

1 Answer
1

active

oldest

votes

If you go to Data Profiling > Ad Hoc Query from the Airflow UI, and select the airflow_db database, you can issue this query:

select execution_date, duration from task_instance where task_id = 'SampleTask'

The duration column holds the number of seconds the task ran.

answered Jan 3 at 8:51

medvedev1088

2,6151335

yeah that works @medvedev1088, can you tell me how to use this inside a current DAG? My requirement is to store the value of the duration into a variable inside a DAG. Also this gives me the duration of all the number of times it has run in the past, I want only the duration of present task after completion.

– Ashish Kumar
Jan 3 at 10:00

For this use case I think you can wrap you operator executable into a function that measures start and end time. But it will be a different duration than what's stored in the duration column in the database.

– medvedev1088
Jan 3 at 10:06

@nedvedev1088 is there any way/operator to query airflow_db from DAG?

– Ashish Kumar
Jan 3 at 11:07

I’m not sure but even if there is it won’t help because duration is set when task is finished, but you need to access it while the task is running if I understood you correctly

– medvedev1088
Jan 3 at 11:09

1

There should be a connection called airflow_db. You can use MySqlHook or PostgresHook with conn_id='airflow_db' to access the database.

– medvedev1088
Jan 3 at 11:46

|
show 3 more comments

If you go to Data Profiling > Ad Hoc Query from the Airflow UI, and select the airflow_db database, you can issue this query:

select execution_date, duration from task_instance where task_id = 'SampleTask'

The duration column holds the number of seconds the task ran.

answered Jan 3 at 8:51

medvedev1088

2,6151335

yeah that works @medvedev1088, can you tell me how to use this inside a current DAG? My requirement is to store the value of the duration into a variable inside a DAG. Also this gives me the duration of all the number of times it has run in the past, I want only the duration of present task after completion.

– Ashish Kumar
Jan 3 at 10:00

For this use case I think you can wrap you operator executable into a function that measures start and end time. But it will be a different duration than what's stored in the duration column in the database.

– medvedev1088
Jan 3 at 10:06

@nedvedev1088 is there any way/operator to query airflow_db from DAG?

– Ashish Kumar
Jan 3 at 11:07

I’m not sure but even if there is it won’t help because duration is set when task is finished, but you need to access it while the task is running if I understood you correctly

– medvedev1088
Jan 3 at 11:09

1

There should be a connection called airflow_db. You can use MySqlHook or PostgresHook with conn_id='airflow_db' to access the database.

– medvedev1088
Jan 3 at 11:46

|
show 3 more comments

If you go to Data Profiling > Ad Hoc Query from the Airflow UI, and select the airflow_db database, you can issue this query:

select execution_date, duration from task_instance where task_id = 'SampleTask'

The duration column holds the number of seconds the task ran.

answered Jan 3 at 8:51

medvedev1088

2,6151335

If you go to Data Profiling > Ad Hoc Query from the Airflow UI, and select the airflow_db database, you can issue this query:

select execution_date, duration from task_instance where task_id = 'SampleTask'

The duration column holds the number of seconds the task ran.

answered Jan 3 at 8:51

medvedev1088

2,6151335

answered Jan 3 at 8:51

medvedev1088

2,6151335

answered Jan 3 at 8:51

medvedev1088

2,6151335

answered Jan 3 at 8:51

medvedev1088

2,6151335

yeah that works @medvedev1088, can you tell me how to use this inside a current DAG? My requirement is to store the value of the duration into a variable inside a DAG. Also this gives me the duration of all the number of times it has run in the past, I want only the duration of present task after completion.

– Ashish Kumar
Jan 3 at 10:00

For this use case I think you can wrap you operator executable into a function that measures start and end time. But it will be a different duration than what's stored in the duration column in the database.

– medvedev1088
Jan 3 at 10:06

@nedvedev1088 is there any way/operator to query airflow_db from DAG?

– Ashish Kumar
Jan 3 at 11:07

I’m not sure but even if there is it won’t help because duration is set when task is finished, but you need to access it while the task is running if I understood you correctly

– medvedev1088
Jan 3 at 11:09

1

There should be a connection called airflow_db. You can use MySqlHook or PostgresHook with conn_id='airflow_db' to access the database.

– medvedev1088
Jan 3 at 11:46

|
show 3 more comments

yeah that works @medvedev1088, can you tell me how to use this inside a current DAG? My requirement is to store the value of the duration into a variable inside a DAG. Also this gives me the duration of all the number of times it has run in the past, I want only the duration of present task after completion.

– Ashish Kumar
Jan 3 at 10:00

For this use case I think you can wrap you operator executable into a function that measures start and end time. But it will be a different duration than what's stored in the duration column in the database.

– medvedev1088
Jan 3 at 10:06

@nedvedev1088 is there any way/operator to query airflow_db from DAG?

– Ashish Kumar
Jan 3 at 11:07

I’m not sure but even if there is it won’t help because duration is set when task is finished, but you need to access it while the task is running if I understood you correctly

– medvedev1088
Jan 3 at 11:09

1

There should be a connection called airflow_db. You can use MySqlHook or PostgresHook with conn_id='airflow_db' to access the database.

– medvedev1088
Jan 3 at 11:46

yeah that works @medvedev1088, can you tell me how to use this inside a current DAG? My requirement is to store the value of the duration into a variable inside a DAG. Also this gives me the duration of all the number of times it has run in the past, I want only the duration of present task after completion.

– Ashish Kumar
Jan 3 at 10:00

For this use case I think you can wrap you operator executable into a function that measures start and end time. But it will be a different duration than what's stored in the duration column in the database.

– medvedev1088
Jan 3 at 10:06

@nedvedev1088 is there any way/operator to query airflow_db from DAG?

– Ashish Kumar
Jan 3 at 11:07

I’m not sure but even if there is it won’t help because duration is set when task is finished, but you need to access it while the task is running if I understood you correctly

– medvedev1088
Jan 3 at 11:09

There should be a connection called airflow_db. You can use MySqlHook or PostgresHook with conn_id='airflow_db' to access the database.

– medvedev1088
Jan 3 at 11:46

|
show 3 more comments

draft saved

draft discarded

Thanks for contributing an answer to Stack Overflow!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

Search This Blog

Ufyukyu