Graphite - Gather metrics only from active service instances

Lets say my spring microservice processes data. Every time a successful processing event occurs, for metrics, I update the micrometer counter. This is registered to a Graphite Registry.

registry = new GraphiteMeterRegistry(new GraphiteConfiguration(), Clock.SYSTEM, HierarchicalNameMapper.DEFAULT);

Counter counter = Counter.builder("process").tag("status","success").register(registry);

So far, it sounds good. But what if I have to create and deploy multiple instances of my service?

How do I get the aggregated count of all successful events from all the instances?

To illustrate my case further, I log the counter.count() value on each increment. Here is what i see ->

<Instance 1> <time> <package-name> Count :122

<Instance 2> <time> <package-name> Count :53

So when I run the graphite query on graphana -

process.status.success.count

I tend to get the random count from either of these instances.

What I need is a query like -

process.service-instance.status.success.count

so that I can run a summarize() function in the end.

Update
Now I'm able to source data from all instances by getting the service instance ID. But that presents a new problem - Since I restart my services time and again, and my service-id changes every time, how do I source data from ONLY ACTIVE services?

Since process.*.status.success.count represents aggregate count of ALL services - dead or alive

edited Jan 9 at 11:42

asked Jan 2 at 10:58

Sameer

378

You can remove netflix-eureka from tags. Question is not related to eureka.

– narendra-choudhary
Jan 5 at 17:39

You can remove spring-cloud as well. There's nothing in this question specific to spring-cloud.

– narendra-choudhary
Jan 5 at 17:54

removed tags as suggested

– Sameer
Jan 9 at 11:45

add a comment |

Lets say my spring microservice processes data. Every time a successful processing event occurs, for metrics, I update the micrometer counter. This is registered to a Graphite Registry.

registry = new GraphiteMeterRegistry(new GraphiteConfiguration(), Clock.SYSTEM, HierarchicalNameMapper.DEFAULT);

Counter counter = Counter.builder("process").tag("status","success").register(registry);

So far, it sounds good. But what if I have to create and deploy multiple instances of my service?

How do I get the aggregated count of all successful events from all the instances?

To illustrate my case further, I log the counter.count() value on each increment. Here is what i see ->

<Instance 1> <time> <package-name> Count :122

<Instance 2> <time> <package-name> Count :53

So when I run the graphite query on graphana -

process.status.success.count

I tend to get the random count from either of these instances.

What I need is a query like -

process.service-instance.status.success.count

so that I can run a summarize() function in the end.

Since process.*.status.success.count represents aggregate count of ALL services - dead or alive

edited Jan 9 at 11:42

asked Jan 2 at 10:58

Sameer

378

You can remove netflix-eureka from tags. Question is not related to eureka.

– narendra-choudhary
Jan 5 at 17:39

You can remove spring-cloud as well. There's nothing in this question specific to spring-cloud.

– narendra-choudhary
Jan 5 at 17:54

removed tags as suggested

– Sameer
Jan 9 at 11:45

add a comment |

Lets say my spring microservice processes data. Every time a successful processing event occurs, for metrics, I update the micrometer counter. This is registered to a Graphite Registry.

registry = new GraphiteMeterRegistry(new GraphiteConfiguration(), Clock.SYSTEM, HierarchicalNameMapper.DEFAULT);

Counter counter = Counter.builder("process").tag("status","success").register(registry);

So far, it sounds good. But what if I have to create and deploy multiple instances of my service?

How do I get the aggregated count of all successful events from all the instances?

To illustrate my case further, I log the counter.count() value on each increment. Here is what i see ->

<Instance 1> <time> <package-name> Count :122

<Instance 2> <time> <package-name> Count :53

So when I run the graphite query on graphana -

process.status.success.count

I tend to get the random count from either of these instances.

What I need is a query like -

process.service-instance.status.success.count

so that I can run a summarize() function in the end.

Since process.*.status.success.count represents aggregate count of ALL services - dead or alive

edited Jan 9 at 11:42

asked Jan 2 at 10:58

Sameer

378

Lets say my spring microservice processes data. Every time a successful processing event occurs, for metrics, I update the micrometer counter. This is registered to a Graphite Registry.

registry = new GraphiteMeterRegistry(new GraphiteConfiguration(), Clock.SYSTEM, HierarchicalNameMapper.DEFAULT);

Counter counter = Counter.builder("process").tag("status","success").register(registry);

So far, it sounds good. But what if I have to create and deploy multiple instances of my service?

How do I get the aggregated count of all successful events from all the instances?

To illustrate my case further, I log the counter.count() value on each increment. Here is what i see ->

<Instance 1> <time> <package-name> Count :122

<Instance 2> <time> <package-name> Count :53

So when I run the graphite query on graphana -

process.status.success.count

I tend to get the random count from either of these instances.

What I need is a query like -

process.service-instance.status.success.count

so that I can run a summarize() function in the end.

Since process.*.status.success.count represents aggregate count of ALL services - dead or alive

grafana graphite micrometer

edited Jan 9 at 11:42

asked Jan 2 at 10:58

Sameer

378

edited Jan 9 at 11:42

asked Jan 2 at 10:58

Sameer

378

edited Jan 9 at 11:42

asked Jan 2 at 10:58

Sameer

378

asked Jan 2 at 10:58

Sameer

378

asked Jan 2 at 10:58

Sameer

378

You can remove netflix-eureka from tags. Question is not related to eureka.

– narendra-choudhary
Jan 5 at 17:39

You can remove spring-cloud as well. There's nothing in this question specific to spring-cloud.

– narendra-choudhary
Jan 5 at 17:54

removed tags as suggested

– Sameer
Jan 9 at 11:45

add a comment |

You can remove netflix-eureka from tags. Question is not related to eureka.

– narendra-choudhary
Jan 5 at 17:39

You can remove spring-cloud as well. There's nothing in this question specific to spring-cloud.

– narendra-choudhary
Jan 5 at 17:54

removed tags as suggested

– Sameer
Jan 9 at 11:45

You can remove netflix-eureka from tags. Question is not related to eureka.

– narendra-choudhary
Jan 5 at 17:39

You can remove spring-cloud as well. There's nothing in this question specific to spring-cloud.

– narendra-choudhary
Jan 5 at 17:54

removed tags as suggested

– Sameer
Jan 9 at 11:45

add a comment |

1 Answer
1

active

oldest

votes

Never use instance ids for aggregation. When instances restart, instance ids will change. (Use instance-id for logging/debugging/record-keeping purpose only.)

Use service-id for aggregation.

For micrometer, you can add service-name in common tags.

registry.config().commonTags("service", "xyz-service");

Common tags are defined at registry level and every metric associated with that registry will have common tags added to it.

And, for dead or alive situation: The metric was pushed when the instance was alive. So if you want to know how many times some step ran, you'll need to consider that count.

To source data from active instances, use time-filter. That will return data pushed by instances that were alive in that duration (Why? Because dead instances do not push metrics).

edited Jan 5 at 17:53

answered Jan 5 at 17:45

narendra-choudhary

2,27221833

thanks for the reply. I get the part with common tags and have already implemented it. However, how do i aggregate it service-id wise? Let's say I register a gauge to count events occuring in an entire day. I reset this gauge every midnight. I have 4 such service instances. How can I sum up the total events?

– Sameer
Jan 9 at 11:42

The WHERE clause ?

– narendra-choudhary
Jan 16 at 14:19

Something like SELECT SUM("value") FROM "metric" WHERE "serviceId" = "service-name"?

– narendra-choudhary
Jan 16 at 14:21

add a comment |

Your Answer

StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});

}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54005107%2fgraphite-gather-metrics-only-from-active-service-instances%23new-answer', 'question_page');
}
);

Post as a guest

Name

Required, but never shown

1 Answer
1

active

oldest

votes

1 Answer
1

active

oldest

votes

Never use instance ids for aggregation. When instances restart, instance ids will change. (Use instance-id for logging/debugging/record-keeping purpose only.)

Use service-id for aggregation.

For micrometer, you can add service-name in common tags.

registry.config().commonTags("service", "xyz-service");

Common tags are defined at registry level and every metric associated with that registry will have common tags added to it.

And, for dead or alive situation: The metric was pushed when the instance was alive. So if you want to know how many times some step ran, you'll need to consider that count.

To source data from active instances, use time-filter. That will return data pushed by instances that were alive in that duration (Why? Because dead instances do not push metrics).

edited Jan 5 at 17:53

answered Jan 5 at 17:45

narendra-choudhary

2,27221833

thanks for the reply. I get the part with common tags and have already implemented it. However, how do i aggregate it service-id wise? Let's say I register a gauge to count events occuring in an entire day. I reset this gauge every midnight. I have 4 such service instances. How can I sum up the total events?

– Sameer
Jan 9 at 11:42

The WHERE clause ?

– narendra-choudhary
Jan 16 at 14:19

Something like SELECT SUM("value") FROM "metric" WHERE "serviceId" = "service-name"?

– narendra-choudhary
Jan 16 at 14:21

add a comment |

Never use instance ids for aggregation. When instances restart, instance ids will change. (Use instance-id for logging/debugging/record-keeping purpose only.)

Use service-id for aggregation.

For micrometer, you can add service-name in common tags.

registry.config().commonTags("service", "xyz-service");

Common tags are defined at registry level and every metric associated with that registry will have common tags added to it.

And, for dead or alive situation: The metric was pushed when the instance was alive. So if you want to know how many times some step ran, you'll need to consider that count.

To source data from active instances, use time-filter. That will return data pushed by instances that were alive in that duration (Why? Because dead instances do not push metrics).

edited Jan 5 at 17:53

answered Jan 5 at 17:45

narendra-choudhary

2,27221833

thanks for the reply. I get the part with common tags and have already implemented it. However, how do i aggregate it service-id wise? Let's say I register a gauge to count events occuring in an entire day. I reset this gauge every midnight. I have 4 such service instances. How can I sum up the total events?

– Sameer
Jan 9 at 11:42

The WHERE clause ?

– narendra-choudhary
Jan 16 at 14:19

Something like SELECT SUM("value") FROM "metric" WHERE "serviceId" = "service-name"?

– narendra-choudhary
Jan 16 at 14:21

add a comment |

Never use instance ids for aggregation. When instances restart, instance ids will change. (Use instance-id for logging/debugging/record-keeping purpose only.)

Use service-id for aggregation.

For micrometer, you can add service-name in common tags.

registry.config().commonTags("service", "xyz-service");

Common tags are defined at registry level and every metric associated with that registry will have common tags added to it.

And, for dead or alive situation: The metric was pushed when the instance was alive. So if you want to know how many times some step ran, you'll need to consider that count.

To source data from active instances, use time-filter. That will return data pushed by instances that were alive in that duration (Why? Because dead instances do not push metrics).

edited Jan 5 at 17:53

answered Jan 5 at 17:45

narendra-choudhary

2,27221833

Never use instance ids for aggregation. When instances restart, instance ids will change. (Use instance-id for logging/debugging/record-keeping purpose only.)

Use service-id for aggregation.

For micrometer, you can add service-name in common tags.

registry.config().commonTags("service", "xyz-service");

Common tags are defined at registry level and every metric associated with that registry will have common tags added to it.

And, for dead or alive situation: The metric was pushed when the instance was alive. So if you want to know how many times some step ran, you'll need to consider that count.

To source data from active instances, use time-filter. That will return data pushed by instances that were alive in that duration (Why? Because dead instances do not push metrics).

edited Jan 5 at 17:53

answered Jan 5 at 17:45

narendra-choudhary

2,27221833

edited Jan 5 at 17:53

answered Jan 5 at 17:45

narendra-choudhary

2,27221833

answered Jan 5 at 17:45

narendra-choudhary

2,27221833

answered Jan 5 at 17:45

narendra-choudhary

2,27221833

thanks for the reply. I get the part with common tags and have already implemented it. However, how do i aggregate it service-id wise? Let's say I register a gauge to count events occuring in an entire day. I reset this gauge every midnight. I have 4 such service instances. How can I sum up the total events?

– Sameer
Jan 9 at 11:42

The WHERE clause ?

– narendra-choudhary
Jan 16 at 14:19

Something like SELECT SUM("value") FROM "metric" WHERE "serviceId" = "service-name"?

– narendra-choudhary
Jan 16 at 14:21

add a comment |

thanks for the reply. I get the part with common tags and have already implemented it. However, how do i aggregate it service-id wise? Let's say I register a gauge to count events occuring in an entire day. I reset this gauge every midnight. I have 4 such service instances. How can I sum up the total events?

– Sameer
Jan 9 at 11:42

The WHERE clause ?

– narendra-choudhary
Jan 16 at 14:19

Something like SELECT SUM("value") FROM "metric" WHERE "serviceId" = "service-name"?

– narendra-choudhary
Jan 16 at 14:21

thanks for the reply. I get the part with common tags and have already implemented it. However, how do i aggregate it service-id wise? Let's say I register a gauge to count events occuring in an entire day. I reset this gauge every midnight. I have 4 such service instances. How can I sum up the total events?

– Sameer
Jan 9 at 11:42

The WHERE clause ?

– narendra-choudhary
Jan 16 at 14:19

Something like SELECT SUM("value") FROM "metric" WHERE "serviceId" = "service-name"?

– narendra-choudhary
Jan 16 at 14:21

add a comment |

draft saved

draft discarded

Thanks for contributing an answer to Stack Overflow!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

Search This Blog

Ufyukyu