Audit logging of HTTP calls in Azure using C#











up vote
2
down vote

favorite
1












For our application we need to log all modifications of any ressource reaching back one year.
The application is a ASP.Net Core Application exposing simple REST endpoints with CRUD operations.
We have a couple of constraints




  • Every ressource has its own HTTP endpoint.

  • We need to log the state of the ressource after each modification.

  • We need to easily retrieve a log for all modifications using the ressource type (which corresponds to the endpoint http://example.com/order, http://example.com/products, ...) and its ID

  • Each http call may modify multiple ressources, as each endpoint accepts a List

  • The application itself runs inside a Docker container in Azure

  • We need to log up to 200k modifications per hour and up to 500k per day


We are already using Application Insights for monitoring, but as data may be sampled, this does not seem to fit.



Is there an Azure service that meets our requirements?






c# azure logging







share|improve this question






















  • did you explore the ELK stack for this ? Logstash to ingest, Elastic search to query , kibana to visualize
    – Aravind
    2 days ago










  • You could use Azure Event Hub to ingest, azure data lake or blob for storage and use azure data lake analytics to query.
    – Peter Bons
    2 days ago















up vote
2
down vote

favorite
1












For our application we need to log all modifications of any ressource reaching back one year.
The application is a ASP.Net Core Application exposing simple REST endpoints with CRUD operations.
We have a couple of constraints




  • Every ressource has its own HTTP endpoint.

  • We need to log the state of the ressource after each modification.

  • We need to easily retrieve a log for all modifications using the ressource type (which corresponds to the endpoint http://example.com/order, http://example.com/products, ...) and its ID

  • Each http call may modify multiple ressources, as each endpoint accepts a List

  • The application itself runs inside a Docker container in Azure

  • We need to log up to 200k modifications per hour and up to 500k per day


We are already using Application Insights for monitoring, but as data may be sampled, this does not seem to fit.



Is there an Azure service that meets our requirements?






c# azure logging







share|improve this question






















  • did you explore the ELK stack for this ? Logstash to ingest, Elastic search to query , kibana to visualize
    – Aravind
    2 days ago










  • You could use Azure Event Hub to ingest, azure data lake or blob for storage and use azure data lake analytics to query.
    – Peter Bons
    2 days ago













up vote
2
down vote

favorite
1









up vote
2
down vote

favorite
1






1





For our application we need to log all modifications of any ressource reaching back one year.
The application is a ASP.Net Core Application exposing simple REST endpoints with CRUD operations.
We have a couple of constraints




  • Every ressource has its own HTTP endpoint.

  • We need to log the state of the ressource after each modification.

  • We need to easily retrieve a log for all modifications using the ressource type (which corresponds to the endpoint http://example.com/order, http://example.com/products, ...) and its ID

  • Each http call may modify multiple ressources, as each endpoint accepts a List

  • The application itself runs inside a Docker container in Azure

  • We need to log up to 200k modifications per hour and up to 500k per day


We are already using Application Insights for monitoring, but as data may be sampled, this does not seem to fit.



Is there an Azure service that meets our requirements?






c# azure logging







share|improve this question













For our application we need to log all modifications of any ressource reaching back one year.
The application is a ASP.Net Core Application exposing simple REST endpoints with CRUD operations.
We have a couple of constraints




  • Every ressource has its own HTTP endpoint.

  • We need to log the state of the ressource after each modification.

  • We need to easily retrieve a log for all modifications using the ressource type (which corresponds to the endpoint http://example.com/order, http://example.com/products, ...) and its ID

  • Each http call may modify multiple ressources, as each endpoint accepts a List

  • The application itself runs inside a Docker container in Azure

  • We need to log up to 200k modifications per hour and up to 500k per day


We are already using Application Insights for monitoring, but as data may be sampled, this does not seem to fit.



Is there an Azure service that meets our requirements?






c# azure logging




c# azure logging






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked 2 days ago









wertzui

2,07111831




2,07111831












  • did you explore the ELK stack for this ? Logstash to ingest, Elastic search to query , kibana to visualize
    – Aravind
    2 days ago










  • You could use Azure Event Hub to ingest, azure data lake or blob for storage and use azure data lake analytics to query.
    – Peter Bons
    2 days ago


















  • did you explore the ELK stack for this ? Logstash to ingest, Elastic search to query , kibana to visualize
    – Aravind
    2 days ago










  • You could use Azure Event Hub to ingest, azure data lake or blob for storage and use azure data lake analytics to query.
    – Peter Bons
    2 days ago
















did you explore the ELK stack for this ? Logstash to ingest, Elastic search to query , kibana to visualize
– Aravind
2 days ago




did you explore the ELK stack for this ? Logstash to ingest, Elastic search to query , kibana to visualize
– Aravind
2 days ago












You could use Azure Event Hub to ingest, azure data lake or blob for storage and use azure data lake analytics to query.
– Peter Bons
2 days ago




You could use Azure Event Hub to ingest, azure data lake or blob for storage and use azure data lake analytics to query.
– Peter Bons
2 days ago

















active

oldest

votes











Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














 

draft saved


draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53373611%2faudit-logging-of-http-calls-in-azure-using-c-sharp%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown






























active

oldest

votes













active

oldest

votes









active

oldest

votes






active

oldest

votes
















 

draft saved


draft discarded



















































 


draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53373611%2faudit-logging-of-http-calls-in-azure-using-c-sharp%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

MongoDB - Not Authorized To Execute Command

Image
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box; } 0 I have successfully enabled authorization on MongoDB and I have created an account on the admin database and then I created an account for my database called test. The following connection string to connect to my test database works successfully: mongo --host 192.168.17.52 --port 27017 -u user1 -p password --authenticationDatabase test Only problem I have now is, I cannot execute commands such as: show dbs. I get the following error when I try to do so: "errmsg" : "not authorized on admin to execute command { listDatabases: 1.0, lsid: { id: UUID("a1d5bc0d-bc58-485e-b232-270758a89455") }, $db: "admin...
Read more

in spring boot 2.1 many test slices are not allowed anymore due to multiple @BootstrapWith

Image
1 I tried to upgrade a yummy sandwich made of two test slices (@JsonTest and @JdbcTest in my case, crunchy test code in between) adding spring boot 2.1 flavour to it. But it seems it was not much of a success. I cannot annotate my tests with many @...Test since they are now each bringing their own XxxTestContextBootstrapper. It used to work when they all used same SpringBootTestContextBootstrapper. @RunWith(SpringRunner.class) @JdbcTest @JsonTest public class Test { @Test public void test() { System.out.printn("Hello, World !"); } } The error I get from BootstrapUtils is illegalStateException : Configuration error: found multiple declarations of @BootstrapWith for test class I understand I might be doing something wrong here but is there an easy way I could load both Json and Jdbc contex...
Read more

How to fix TextFormField cause rebuild widget in Flutter

Image
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box; } 0 I've got the same issue at #9280. I tried all solutions from that topic with/without GlobalKey for Scaffold and Form but the problem wasn't fixed. I tried to build a authentication form with login and signup. But the keyboard doesn't show when I tap the TextFormField at the first time. Then I try to input some letter from physical keyboard, the widget is rebuilt. Like this: login screen Nothing to be log when I run flutter run --verbose, so I logged manually. Every tapping on field, the widget called dispose -> init -> build. [√] Flutter (Channel beta, v1.0.0, on Microsoft Windows [Version 10.0.17763.195], ...
Read more