Mixing
How should I modify logstash.conf to get the field I want?
I use the ELK + Filebeat , all version is 6.4.3 , the OS is windows 10
I add custom field in filebeat.yml , the key name is log_type , the value of log_type is nginx-access
The picture show part of filebeat.yml.
The content of logstash.conf is :
input {
beats {
host => "0.0.0.0"
port => "5544"
}
}
filter {
mutate {
rename => { "[host][name]" => "host" }
}
if [fields][log_type] == "nginx-access" {
grok {
match => { "message" => ["%{IPORHOST:[nginx][access][remote_ip]} - %{DATA:[nginx][access][user_name]} [%{HTTPDATE:[nginx][access][time]}] "%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http_version]}" %{NUMBER:[nginx][access][response_code]} %{NUMBER:[nginx][access][body_sent][bytes]} "%{DATA:[nginx][access][referrer]}" "%{DATA:[nginx][access][agent]}" "%{DATA:[nginx][access][x_forwarded_for]}" %{NUMBER:[nginx][access][request_time]}"] }
}
mutate {
copy => { "[nginx][access][request_time]" => "[nginx][access][requesttime]" }
}
mutate {
convert => {
"[nginx][access][requesttime]" => "float"
}
}
}
}
output {
stdout {
codec => rubydebug { metadata => true }
}
elasticsearch {
hosts => ["localhost:9200"]
}
}
When I use the command :
logstash.bat -f logstash.conf
The output is :
Question 1：
The field in the red box above is "requesttime" and "request_time" , what I want the field is nginx.access.requesttime and nginx.access.request_time，not requesttime and request_time 。 How should I modify logstash.conf to achieve my goal?
Question 2:
When I use the above logstash.conf , the field of the kibana management interface is only "request_time" field .
The picture show this :
If I want the "nginx.access.requesttime" field to also appear in the fields of the Kibana management interface, how should I modify the logstash.conf ?
nginx logstash-configuration
asked Nov 19 '18 at 13:30
Andrew
467
add a comment |
I use the ELK + Filebeat , all version is 6.4.3 , the OS is windows 10
I add custom field in filebeat.yml , the key name is log_type , the value of log_type is nginx-access
The picture show part of filebeat.yml.
The content of logstash.conf is :
input {
beats {
host => "0.0.0.0"
port => "5544"
}
}
filter {
mutate {
rename => { "[host][name]" => "host" }
}
if [fields][log_type] == "nginx-access" {
grok {
match => { "message" => ["%{IPORHOST:[nginx][access][remote_ip]} - %{DATA:[nginx][access][user_name]} [%{HTTPDATE:[nginx][access][time]}] "%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http_version]}" %{NUMBER:[nginx][access][response_code]} %{NUMBER:[nginx][access][body_sent][bytes]} "%{DATA:[nginx][access][referrer]}" "%{DATA:[nginx][access][agent]}" "%{DATA:[nginx][access][x_forwarded_for]}" %{NUMBER:[nginx][access][request_time]}"] }
}
mutate {
copy => { "[nginx][access][request_time]" => "[nginx][access][requesttime]" }
}
mutate {
convert => {
"[nginx][access][requesttime]" => "float"
}
}
}
}
output {
stdout {
codec => rubydebug { metadata => true }
}
elasticsearch {
hosts => ["localhost:9200"]
}
}
When I use the command :
logstash.bat -f logstash.conf
The output is :
Question 1：
The field in the red box above is "requesttime" and "request_time" , what I want the field is nginx.access.requesttime and nginx.access.request_time，not requesttime and request_time 。 How should I modify logstash.conf to achieve my goal?
Question 2:
When I use the above logstash.conf , the field of the kibana management interface is only "request_time" field .
The picture show this :
If I want the "nginx.access.requesttime" field to also appear in the fields of the Kibana management interface, how should I modify the logstash.conf ?
nginx logstash-configuration
asked Nov 19 '18 at 13:30
Andrew
467
add a comment |
I use the ELK + Filebeat , all version is 6.4.3 , the OS is windows 10
I add custom field in filebeat.yml , the key name is log_type , the value of log_type is nginx-access
The picture show part of filebeat.yml.
The content of logstash.conf is :
input {
beats {
host => "0.0.0.0"
port => "5544"
}
}
filter {
mutate {
rename => { "[host][name]" => "host" }
}
if [fields][log_type] == "nginx-access" {
grok {
match => { "message" => ["%{IPORHOST:[nginx][access][remote_ip]} - %{DATA:[nginx][access][user_name]} [%{HTTPDATE:[nginx][access][time]}] "%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http_version]}" %{NUMBER:[nginx][access][response_code]} %{NUMBER:[nginx][access][body_sent][bytes]} "%{DATA:[nginx][access][referrer]}" "%{DATA:[nginx][access][agent]}" "%{DATA:[nginx][access][x_forwarded_for]}" %{NUMBER:[nginx][access][request_time]}"] }
}
mutate {
copy => { "[nginx][access][request_time]" => "[nginx][access][requesttime]" }
}
mutate {
convert => {
"[nginx][access][requesttime]" => "float"
}
}
}
}
output {
stdout {
codec => rubydebug { metadata => true }
}
elasticsearch {
hosts => ["localhost:9200"]
}
}
When I use the command :
logstash.bat -f logstash.conf
The output is :
Question 1：
The field in the red box above is "requesttime" and "request_time" , what I want the field is nginx.access.requesttime and nginx.access.request_time，not requesttime and request_time 。 How should I modify logstash.conf to achieve my goal?
Question 2:
When I use the above logstash.conf , the field of the kibana management interface is only "request_time" field .
The picture show this :
If I want the "nginx.access.requesttime" field to also appear in the fields of the Kibana management interface, how should I modify the logstash.conf ?
nginx logstash-configuration
asked Nov 19 '18 at 13:30
Andrew
467
I use the ELK + Filebeat , all version is 6.4.3 , the OS is windows 10
I add custom field in filebeat.yml , the key name is log_type , the value of log_type is nginx-access
The picture show part of filebeat.yml.
The content of logstash.conf is :
input {
beats {
host => "0.0.0.0"
port => "5544"
}
}
filter {
mutate {
rename => { "[host][name]" => "host" }
}
if [fields][log_type] == "nginx-access" {
grok {
match => { "message" => ["%{IPORHOST:[nginx][access][remote_ip]} - %{DATA:[nginx][access][user_name]} [%{HTTPDATE:[nginx][access][time]}] "%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http_version]}" %{NUMBER:[nginx][access][response_code]} %{NUMBER:[nginx][access][body_sent][bytes]} "%{DATA:[nginx][access][referrer]}" "%{DATA:[nginx][access][agent]}" "%{DATA:[nginx][access][x_forwarded_for]}" %{NUMBER:[nginx][access][request_time]}"] }
}
mutate {
copy => { "[nginx][access][request_time]" => "[nginx][access][requesttime]" }
}
mutate {
convert => {
"[nginx][access][requesttime]" => "float"
}
}
}
}
output {
stdout {
codec => rubydebug { metadata => true }
}
elasticsearch {
hosts => ["localhost:9200"]
}
}
When I use the command :
logstash.bat -f logstash.conf
The output is :
Question 1：
The field in the red box above is "requesttime" and "request_time" , what I want the field is nginx.access.requesttime and nginx.access.request_time，not requesttime and request_time 。 How should I modify logstash.conf to achieve my goal?
Question 2:
When I use the above logstash.conf , the field of the kibana management interface is only "request_time" field .
The picture show this :
If I want the "nginx.access.requesttime" field to also appear in the fields of the Kibana management interface, how should I modify the logstash.conf ?
nginx logstash-configuration
nginx logstash-configuration
asked Nov 19 '18 at 13:30
Andrew
467
asked Nov 19 '18 at 13:30
Andrew
467
asked Nov 19 '18 at 13:30
Andrew
467
asked Nov 19 '18 at 13:30
Andrew
467
asked Nov 19 '18 at 13:30
Andrew
467
467
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Question 1:
I believe what you are looking for is
mutate {
copy => { "[nginx][access][request_time]" => "nginx.access.requesttime" }
}
Question 2:
Whethere something is a keyword is determined by the template field mapping in Elasticsearch. Try the option above and see if the issue resolved.
This issue in elastic forum may help you.
answered Nov 19 '18 at 21:52
Gal S
757213
Thanks very much. Which of the following two convert formats is correct? mutate { convert => { "[nginx][access][requesttime]" => "float" } } mutate { convert => { "nginx.access.requesttime" => "float" } }
– Andrew
Nov 20 '18 at 1:59
Having that the convert is done after the copy you should use the "nginx.access.requesttime" since the event is already named like that. You can see in logstash source code def copy(event)
– Gal S
Nov 20 '18 at 14:58
Thanks very much! @Gal S
– Andrew
Nov 20 '18 at 15:02
I mark the answer as the chosen one.
– Andrew
Nov 21 '18 at 3:05
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53375717%2fhow-should-i-modify-logstash-conf-to-get-the-field-i-want%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Question 1:
I believe what you are looking for is
mutate {
copy => { "[nginx][access][request_time]" => "nginx.access.requesttime" }
}
Question 2:
Whethere something is a keyword is determined by the template field mapping in Elasticsearch. Try the option above and see if the issue resolved.
This issue in elastic forum may help you.
answered Nov 19 '18 at 21:52
Gal S
757213
Thanks very much. Which of the following two convert formats is correct? mutate { convert => { "[nginx][access][requesttime]" => "float" } } mutate { convert => { "nginx.access.requesttime" => "float" } }
– Andrew
Nov 20 '18 at 1:59
Having that the convert is done after the copy you should use the "nginx.access.requesttime" since the event is already named like that. You can see in logstash source code def copy(event)
– Gal S
Nov 20 '18 at 14:58
Thanks very much! @Gal S
– Andrew
Nov 20 '18 at 15:02
I mark the answer as the chosen one.
– Andrew
Nov 21 '18 at 3:05
add a comment |
Question 1:
I believe what you are looking for is
mutate {
copy => { "[nginx][access][request_time]" => "nginx.access.requesttime" }
}
Question 2:
Whethere something is a keyword is determined by the template field mapping in Elasticsearch. Try the option above and see if the issue resolved.
This issue in elastic forum may help you.
answered Nov 19 '18 at 21:52
Gal S
757213
Thanks very much. Which of the following two convert formats is correct? mutate { convert => { "[nginx][access][requesttime]" => "float" } } mutate { convert => { "nginx.access.requesttime" => "float" } }
– Andrew
Nov 20 '18 at 1:59
Having that the convert is done after the copy you should use the "nginx.access.requesttime" since the event is already named like that. You can see in logstash source code def copy(event)
– Gal S
Nov 20 '18 at 14:58
Thanks very much! @Gal S
– Andrew
Nov 20 '18 at 15:02
I mark the answer as the chosen one.
– Andrew
Nov 21 '18 at 3:05
add a comment |
Question 1:
I believe what you are looking for is
mutate {
copy => { "[nginx][access][request_time]" => "nginx.access.requesttime" }
}
Question 2:
Whethere something is a keyword is determined by the template field mapping in Elasticsearch. Try the option above and see if the issue resolved.
This issue in elastic forum may help you.
answered Nov 19 '18 at 21:52
Gal S
757213
Question 1:
I believe what you are looking for is
mutate {
copy => { "[nginx][access][request_time]" => "nginx.access.requesttime" }
}
Question 2:
Whethere something is a keyword is determined by the template field mapping in Elasticsearch. Try the option above and see if the issue resolved.
This issue in elastic forum may help you.
answered Nov 19 '18 at 21:52
Gal S
757213
answered Nov 19 '18 at 21:52
Gal S
757213
answered Nov 19 '18 at 21:52
Gal S
757213
answered Nov 19 '18 at 21:52
Gal S
757213
757213
Thanks very much. Which of the following two convert formats is correct? mutate { convert => { "[nginx][access][requesttime]" => "float" } } mutate { convert => { "nginx.access.requesttime" => "float" } }
– Andrew
Nov 20 '18 at 1:59
Having that the convert is done after the copy you should use the "nginx.access.requesttime" since the event is already named like that. You can see in logstash source code def copy(event)
– Gal S
Nov 20 '18 at 14:58
Thanks very much! @Gal S
– Andrew
Nov 20 '18 at 15:02
I mark the answer as the chosen one.
– Andrew
Nov 21 '18 at 3:05
add a comment |
Thanks very much. Which of the following two convert formats is correct? mutate { convert => { "[nginx][access][requesttime]" => "float" } } mutate { convert => { "nginx.access.requesttime" => "float" } }
– Andrew
Nov 20 '18 at 1:59
Having that the convert is done after the copy you should use the "nginx.access.requesttime" since the event is already named like that. You can see in logstash source code def copy(event)
– Gal S
Nov 20 '18 at 14:58
Thanks very much! @Gal S
– Andrew
Nov 20 '18 at 15:02
I mark the answer as the chosen one.
– Andrew
Nov 21 '18 at 3:05
Thanks very much. Which of the following two convert formats is correct? mutate { convert => { "[nginx][access][requesttime]" => "float" } } mutate { convert => { "nginx.access.requesttime" => "float" } }
– Andrew
Nov 20 '18 at 1:59
Thanks very much. Which of the following two convert formats is correct? mutate { convert => { "[nginx][access][requesttime]" => "float" } } mutate { convert => { "nginx.access.requesttime" => "float" } }
– Andrew
Nov 20 '18 at 1:59
Having that the convert is done after the copy you should use the "nginx.access.requesttime" since the event is already named like that. You can see in logstash source code def copy(event)
– Gal S
Nov 20 '18 at 14:58
Having that the convert is done after the copy you should use the "nginx.access.requesttime" since the event is already named like that. You can see in logstash source code def copy(event)
– Gal S
Nov 20 '18 at 14:58
Thanks very much! @Gal S
– Andrew
Nov 20 '18 at 15:02
Thanks very much! @Gal S
– Andrew
Nov 20 '18 at 15:02
I mark the answer as the chosen one.
– Andrew
Nov 21 '18 at 3:05
I mark the answer as the chosen one.
– Andrew
Nov 21 '18 at 3:05
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53375717%2fhow-should-i-modify-logstash-conf-to-get-the-field-i-want%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
