Mixing
Azure Stream Analytics to Azure Cosmos DB integration throughput problems in writing data
-1
I have Edge Device Which have 20 Devices sending data to IoTHub. I am using Azure Stream Analytics to push the data to Cosmos DB. In my cosmos DB i have made partition Key as ( deviceId ) so that while writing data they will be writing to different partition at same second.
I see as data is is going from Edge so all messages are going to same Device to cloud partition in Azure IoTHub . As result Stream Analytics is also using only one unit to write and cosmos DB also using one unit to push .
In cosmos DB I removed all the indexes and using partition Key
azure-cosmosdb azure-iot-hub azure-stream-analytics azure-cosmosdb-sqlapi asked Nov 20 '18 at 17:04
Amjath KhanAmjath Khan
488
add a comment |
-1
I have Edge Device Which have 20 Devices sending data to IoTHub. I am using Azure Stream Analytics to push the data to Cosmos DB. In my cosmos DB i have made partition Key as ( deviceId ) so that while writing data they will be writing to different partition at same second.
I see as data is is going from Edge so all messages are going to same Device to cloud partition in Azure IoTHub . As result Stream Analytics is also using only one unit to write and cosmos DB also using one unit to push .
In cosmos DB I removed all the indexes and using partition Key
azure-cosmosdb azure-iot-hub azure-stream-analytics azure-cosmosdb-sqlapi asked Nov 20 '18 at 17:04
Amjath KhanAmjath Khan
488
Unfortunately, there is no specific technical / programming / data detail in your question; rather, it's several paragraphs of rants and frustrations (and this is the completely wrong place for that), and that just makes your question harder to understand. Cosmos DB has, for example, partitioning (and your allocated RU gets divided up across partitions), so depending on how you partitioned your data, you could be writing to a single partition and getting throttled (as just one idea). Please edit your question to 1) get rid of your ranting and 2) provide more exact details.
– David Makogon
Nov 20 '18 at 18:50
And when you add details: Probably a good idea to show your Stream Analytics query, demonstrating how you are both reading from IoT Hub and writing to Cosmos DB. And also, include your partition key, and content frequency (e.g. is it really only one document per second?)
– David Makogon
Nov 20 '18 at 18:51
@DavidMakogon I have edited my Question .
– Amjath Khan
Nov 21 '18 at 8:53
add a comment |
-1
-1
-1
I have Edge Device Which have 20 Devices sending data to IoTHub. I am using Azure Stream Analytics to push the data to Cosmos DB. In my cosmos DB i have made partition Key as ( deviceId ) so that while writing data they will be writing to different partition at same second.
I see as data is is going from Edge so all messages are going to same Device to cloud partition in Azure IoTHub . As result Stream Analytics is also using only one unit to write and cosmos DB also using one unit to push .
In cosmos DB I removed all the indexes and using partition Key
azure-cosmosdb azure-iot-hub azure-stream-analytics azure-cosmosdb-sqlapi asked Nov 20 '18 at 17:04
Amjath KhanAmjath Khan
488
I have Edge Device Which have 20 Devices sending data to IoTHub. I am using Azure Stream Analytics to push the data to Cosmos DB. In my cosmos DB i have made partition Key as ( deviceId ) so that while writing data they will be writing to different partition at same second.
I see as data is is going from Edge so all messages are going to same Device to cloud partition in Azure IoTHub . As result Stream Analytics is also using only one unit to write and cosmos DB also using one unit to push .
In cosmos DB I removed all the indexes and using partition Key
azure-cosmosdb azure-iot-hub azure-stream-analytics azure-cosmosdb-sqlapi azure-cosmosdb azure-iot-hub azure-stream-analytics azure-cosmosdb-sqlapi asked Nov 20 '18 at 17:04
Amjath KhanAmjath Khan
488
asked Nov 20 '18 at 17:04
Amjath KhanAmjath Khan
488
edited Nov 21 '18 at 8:52
Amjath Khan
asked Nov 20 '18 at 17:04
Amjath KhanAmjath Khan
488
asked Nov 20 '18 at 17:04
Amjath KhanAmjath Khan
488
asked Nov 20 '18 at 17:04
Amjath KhanAmjath Khan
488
488
Unfortunately, there is no specific technical / programming / data detail in your question; rather, it's several paragraphs of rants and frustrations (and this is the completely wrong place for that), and that just makes your question harder to understand. Cosmos DB has, for example, partitioning (and your allocated RU gets divided up across partitions), so depending on how you partitioned your data, you could be writing to a single partition and getting throttled (as just one idea). Please edit your question to 1) get rid of your ranting and 2) provide more exact details.
– David Makogon
Nov 20 '18 at 18:50
And when you add details: Probably a good idea to show your Stream Analytics query, demonstrating how you are both reading from IoT Hub and writing to Cosmos DB. And also, include your partition key, and content frequency (e.g. is it really only one document per second?)
– David Makogon
Nov 20 '18 at 18:51
@DavidMakogon I have edited my Question .
– Amjath Khan
Nov 21 '18 at 8:53
add a comment |
Unfortunately, there is no specific technical / programming / data detail in your question; rather, it's several paragraphs of rants and frustrations (and this is the completely wrong place for that), and that just makes your question harder to understand. Cosmos DB has, for example, partitioning (and your allocated RU gets divided up across partitions), so depending on how you partitioned your data, you could be writing to a single partition and getting throttled (as just one idea). Please edit your question to 1) get rid of your ranting and 2) provide more exact details.
– David Makogon
Nov 20 '18 at 18:50
And when you add details: Probably a good idea to show your Stream Analytics query, demonstrating how you are both reading from IoT Hub and writing to Cosmos DB. And also, include your partition key, and content frequency (e.g. is it really only one document per second?)
– David Makogon
Nov 20 '18 at 18:51
@DavidMakogon I have edited my Question .
– Amjath Khan
Nov 21 '18 at 8:53
Unfortunately, there is no specific technical / programming / data detail in your question; rather, it's several paragraphs of rants and frustrations (and this is the completely wrong place for that), and that just makes your question harder to understand. Cosmos DB has, for example, partitioning (and your allocated RU gets divided up across partitions), so depending on how you partitioned your data, you could be writing to a single partition and getting throttled (as just one idea). Please edit your question to 1) get rid of your ranting and 2) provide more exact details.
– David Makogon
Nov 20 '18 at 18:50
Unfortunately, there is no specific technical / programming / data detail in your question; rather, it's several paragraphs of rants and frustrations (and this is the completely wrong place for that), and that just makes your question harder to understand. Cosmos DB has, for example, partitioning (and your allocated RU gets divided up across partitions), so depending on how you partitioned your data, you could be writing to a single partition and getting throttled (as just one idea). Please edit your question to 1) get rid of your ranting and 2) provide more exact details.
– David Makogon
Nov 20 '18 at 18:50
And when you add details: Probably a good idea to show your Stream Analytics query, demonstrating how you are both reading from IoT Hub and writing to Cosmos DB. And also, include your partition key, and content frequency (e.g. is it really only one document per second?)
– David Makogon
Nov 20 '18 at 18:51
And when you add details: Probably a good idea to show your Stream Analytics query, demonstrating how you are both reading from IoT Hub and writing to Cosmos DB. And also, include your partition key, and content frequency (e.g. is it really only one document per second?)
– David Makogon
Nov 20 '18 at 18:51
@DavidMakogon I have edited my Question .
– Amjath Khan
Nov 21 '18 at 8:53
@DavidMakogon I have edited my Question .
– Amjath Khan
Nov 21 '18 at 8:53
add a comment |
1 Answer
1
active
oldest
votes
-1
Responded on msdn as it had more details.
answered Nov 22 '18 at 13:48
ChetanChetan
792
This isn't an answer.
– David Makogon
Nov 22 '18 at 18:17
@chetan in azure stream analytics I can add partition by partitionId which I have already tried, but partition by DeviceId, is not allowed, as I tried that 15 days back, I am not sure now. And as all my data is coming to same device to cloud partition Azure stream analytics will also take same number of parallel streams. Is there possibility that it is false alarm from cosmos db and my stream analytics getting stopped due to that. I have 4kb of data like that I have 20 devices data from edge it is going to Iot hub. Means 80 kb of data per second. And each device has 150 sensors data
– Amjath Khan
Nov 23 '18 at 14:19
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53398009%2fazure-stream-analytics-to-azure-cosmos-db-integration-throughput-problems-in-wri%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
-1
Responded on msdn as it had more details.
answered Nov 22 '18 at 13:48
ChetanChetan
792
This isn't an answer.
– David Makogon
Nov 22 '18 at 18:17
@chetan in azure stream analytics I can add partition by partitionId which I have already tried, but partition by DeviceId, is not allowed, as I tried that 15 days back, I am not sure now. And as all my data is coming to same device to cloud partition Azure stream analytics will also take same number of parallel streams. Is there possibility that it is false alarm from cosmos db and my stream analytics getting stopped due to that. I have 4kb of data like that I have 20 devices data from edge it is going to Iot hub. Means 80 kb of data per second. And each device has 150 sensors data
– Amjath Khan
Nov 23 '18 at 14:19
add a comment |
-1
Responded on msdn as it had more details.
answered Nov 22 '18 at 13:48
ChetanChetan
792
This isn't an answer.
– David Makogon
Nov 22 '18 at 18:17
@chetan in azure stream analytics I can add partition by partitionId which I have already tried, but partition by DeviceId, is not allowed, as I tried that 15 days back, I am not sure now. And as all my data is coming to same device to cloud partition Azure stream analytics will also take same number of parallel streams. Is there possibility that it is false alarm from cosmos db and my stream analytics getting stopped due to that. I have 4kb of data like that I have 20 devices data from edge it is going to Iot hub. Means 80 kb of data per second. And each device has 150 sensors data
– Amjath Khan
Nov 23 '18 at 14:19
add a comment |
-1
-1
-1
Responded on msdn as it had more details.
answered Nov 22 '18 at 13:48
ChetanChetan
792
Responded on msdn as it had more details.
answered Nov 22 '18 at 13:48
ChetanChetan
792
answered Nov 22 '18 at 13:48
ChetanChetan
792
answered Nov 22 '18 at 13:48
ChetanChetan
792
answered Nov 22 '18 at 13:48
ChetanChetan
792
792
This isn't an answer.
– David Makogon
Nov 22 '18 at 18:17
@chetan in azure stream analytics I can add partition by partitionId which I have already tried, but partition by DeviceId, is not allowed, as I tried that 15 days back, I am not sure now. And as all my data is coming to same device to cloud partition Azure stream analytics will also take same number of parallel streams. Is there possibility that it is false alarm from cosmos db and my stream analytics getting stopped due to that. I have 4kb of data like that I have 20 devices data from edge it is going to Iot hub. Means 80 kb of data per second. And each device has 150 sensors data
– Amjath Khan
Nov 23 '18 at 14:19
add a comment |
This isn't an answer.
– David Makogon
Nov 22 '18 at 18:17
@chetan in azure stream analytics I can add partition by partitionId which I have already tried, but partition by DeviceId, is not allowed, as I tried that 15 days back, I am not sure now. And as all my data is coming to same device to cloud partition Azure stream analytics will also take same number of parallel streams. Is there possibility that it is false alarm from cosmos db and my stream analytics getting stopped due to that. I have 4kb of data like that I have 20 devices data from edge it is going to Iot hub. Means 80 kb of data per second. And each device has 150 sensors data
– Amjath Khan
Nov 23 '18 at 14:19
This isn't an answer.
– David Makogon
Nov 22 '18 at 18:17
This isn't an answer.
– David Makogon
Nov 22 '18 at 18:17
@chetan in azure stream analytics I can add partition by partitionId which I have already tried, but partition by DeviceId, is not allowed, as I tried that 15 days back, I am not sure now. And as all my data is coming to same device to cloud partition Azure stream analytics will also take same number of parallel streams. Is there possibility that it is false alarm from cosmos db and my stream analytics getting stopped due to that. I have 4kb of data like that I have 20 devices data from edge it is going to Iot hub. Means 80 kb of data per second. And each device has 150 sensors data
– Amjath Khan
Nov 23 '18 at 14:19
@chetan in azure stream analytics I can add partition by partitionId which I have already tried, but partition by DeviceId, is not allowed, as I tried that 15 days back, I am not sure now. And as all my data is coming to same device to cloud partition Azure stream analytics will also take same number of parallel streams. Is there possibility that it is false alarm from cosmos db and my stream analytics getting stopped due to that. I have 4kb of data like that I have 20 devices data from edge it is going to Iot hub. Means 80 kb of data per second. And each device has 150 sensors data
– Amjath Khan
Nov 23 '18 at 14:19
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53398009%2fazure-stream-analytics-to-azure-cosmos-db-integration-throughput-problems-in-wri%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Unfortunately, there is no specific technical / programming / data detail in your question; rather, it's several paragraphs of rants and frustrations (and this is the completely wrong place for that), and that just makes your question harder to understand. Cosmos DB has, for example, partitioning (and your allocated RU gets divided up across partitions), so depending on how you partitioned your data, you could be writing to a single partition and getting throttled (as just one idea). Please edit your question to 1) get rid of your ranting and 2) provide more exact details.
– David Makogon
Nov 20 '18 at 18:50
And when you add details: Probably a good idea to show your Stream Analytics query, demonstrating how you are both reading from IoT Hub and writing to Cosmos DB. And also, include your partition key, and content frequency (e.g. is it really only one document per second?)
– David Makogon
Nov 20 '18 at 18:51
@DavidMakogon I have edited my Question .
– Amjath Khan
Nov 21 '18 at 8:53