migrate data from mysql to aws elasticsearch logstash

up vote
0
down vote

favorite

I'm new to ELK and I have never used AWS Elasticsearch before.

I'm trying to migrate data from mysql server to aws elasticsearch using logstash.

However, I got an error saying something like this:

[ERROR][logstash.pipeline        ] Error registering plugin {:pipeline_id=>"main", :plugin=>"<LogStash::OutputDelegator:0x3ad80dbe>", :error=>"undefined method `credentials' for nil:NilClass", :thread=>"#<Thread:0x1cddccdd run>"}

[ERROR][logstash.pipeline        ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<NoMethodError: undefined method `credentials' for nil:NilClass>,

[ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}

[ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Stop/pipeline_id:main, :exception=>"NoMethodError", :message=>"undefined method `unlock' for nil:NilClass"

I installed aws logstash plugin using:

bin/logstash-plugin install logstash-output-amazon_es

And my config file is the following:

input {

  jdbc {

    jdbc_driver_library => "/library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/mysql-connector-java-8.0.13.jar"

    jdbc_driver_class => "com.mysql.jdbc.Driver"

    jdbc_connection_string => "jdbc:mysql:SOMETHING"

    jdbc_user => "USER"

    jdbc_password => "PASSWORD"

    statement => "select * from TABLE_NAME"



      }

    }



output {

  amazon_es {

    hosts => ["MY_AWS_ES_DOMAIN"]

    region => "REGION"

    index => "INDEX_NAME"

    document_type => "TYPE"

    }

  stdout {

    codec => rubydebug

  }

}

Please help me find the way out from this :'((
Thank you so much in advance!

Thanks,
Sam

mysql elasticsearch amazon-s3 plugins logstash

share|improve this question

asked 22 hours ago

Sam Seo

31

New contributor

Sam Seo is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

add a comment | 

up vote
0
down vote

favorite

I'm new to ELK and I have never used AWS Elasticsearch before.

I'm trying to migrate data from mysql server to aws elasticsearch using logstash.

However, I got an error saying something like this:

[ERROR][logstash.pipeline        ] Error registering plugin {:pipeline_id=>"main", :plugin=>"<LogStash::OutputDelegator:0x3ad80dbe>", :error=>"undefined method `credentials' for nil:NilClass", :thread=>"#<Thread:0x1cddccdd run>"}

[ERROR][logstash.pipeline        ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<NoMethodError: undefined method `credentials' for nil:NilClass>,

[ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}

[ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Stop/pipeline_id:main, :exception=>"NoMethodError", :message=>"undefined method `unlock' for nil:NilClass"

I installed aws logstash plugin using:

bin/logstash-plugin install logstash-output-amazon_es

And my config file is the following:

input {

  jdbc {

    jdbc_driver_library => "/library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/mysql-connector-java-8.0.13.jar"

    jdbc_driver_class => "com.mysql.jdbc.Driver"

    jdbc_connection_string => "jdbc:mysql:SOMETHING"

    jdbc_user => "USER"

    jdbc_password => "PASSWORD"

    statement => "select * from TABLE_NAME"



      }

    }



output {

  amazon_es {

    hosts => ["MY_AWS_ES_DOMAIN"]

    region => "REGION"

    index => "INDEX_NAME"

    document_type => "TYPE"

    }

  stdout {

    codec => rubydebug

  }

}

Please help me find the way out from this :'((
Thank you so much in advance!

Thanks,
Sam

mysql elasticsearch amazon-s3 plugins logstash

share|improve this question

asked 22 hours ago

Sam Seo

31

New contributor

Sam Seo is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

add a comment | 

up vote
0
down vote

favorite

up vote
0
down vote

favorite

I'm new to ELK and I have never used AWS Elasticsearch before.

I'm trying to migrate data from mysql server to aws elasticsearch using logstash.

However, I got an error saying something like this:

[ERROR][logstash.pipeline        ] Error registering plugin {:pipeline_id=>"main", :plugin=>"<LogStash::OutputDelegator:0x3ad80dbe>", :error=>"undefined method `credentials' for nil:NilClass", :thread=>"#<Thread:0x1cddccdd run>"}

[ERROR][logstash.pipeline        ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<NoMethodError: undefined method `credentials' for nil:NilClass>,

[ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}

[ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Stop/pipeline_id:main, :exception=>"NoMethodError", :message=>"undefined method `unlock' for nil:NilClass"

I installed aws logstash plugin using:

bin/logstash-plugin install logstash-output-amazon_es

And my config file is the following:

input {

  jdbc {

    jdbc_driver_library => "/library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/mysql-connector-java-8.0.13.jar"

    jdbc_driver_class => "com.mysql.jdbc.Driver"

    jdbc_connection_string => "jdbc:mysql:SOMETHING"

    jdbc_user => "USER"

    jdbc_password => "PASSWORD"

    statement => "select * from TABLE_NAME"



      }

    }



output {

  amazon_es {

    hosts => ["MY_AWS_ES_DOMAIN"]

    region => "REGION"

    index => "INDEX_NAME"

    document_type => "TYPE"

    }

  stdout {

    codec => rubydebug

  }

}

Please help me find the way out from this :'((
Thank you so much in advance!

Thanks,
Sam

mysql elasticsearch amazon-s3 plugins logstash

share|improve this question

asked 22 hours ago

Sam Seo

31

New contributor

Sam Seo is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

I'm new to ELK and I have never used AWS Elasticsearch before.

I'm trying to migrate data from mysql server to aws elasticsearch using logstash.

However, I got an error saying something like this:

[ERROR][logstash.pipeline        ] Error registering plugin {:pipeline_id=>"main", :plugin=>"<LogStash::OutputDelegator:0x3ad80dbe>", :error=>"undefined method `credentials' for nil:NilClass", :thread=>"#<Thread:0x1cddccdd run>"}

[ERROR][logstash.pipeline        ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<NoMethodError: undefined method `credentials' for nil:NilClass>,

[ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}

[ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Stop/pipeline_id:main, :exception=>"NoMethodError", :message=>"undefined method `unlock' for nil:NilClass"

I installed aws logstash plugin using:

bin/logstash-plugin install logstash-output-amazon_es

And my config file is the following:

input {

  jdbc {

    jdbc_driver_library => "/library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/lib/mysql-connector-java-8.0.13.jar"

    jdbc_driver_class => "com.mysql.jdbc.Driver"

    jdbc_connection_string => "jdbc:mysql:SOMETHING"

    jdbc_user => "USER"

    jdbc_password => "PASSWORD"

    statement => "select * from TABLE_NAME"



      }

    }



output {

  amazon_es {

    hosts => ["MY_AWS_ES_DOMAIN"]

    region => "REGION"

    index => "INDEX_NAME"

    document_type => "TYPE"

    }

  stdout {

    codec => rubydebug

  }

}

Please help me find the way out from this :'((
Thank you so much in advance!

Thanks,
Sam

mysql elasticsearch amazon-s3 plugins logstash

mysql elasticsearch amazon-s3 plugins logstash

share|improve this question

asked 22 hours ago

Sam Seo

31

New contributor

Sam Seo is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

share|improve this question

asked 22 hours ago

Sam Seo

31

New contributor

Sam Seo is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

share|improve this question

share|improve this question

asked 22 hours ago

Sam Seo

31

New contributor

Sam Seo is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

asked 22 hours ago

Sam Seo

31

asked 22 hours ago

Sam Seo

31

31

New contributor

Sam Seo is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

New contributor

Sam Seo is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

Sam Seo is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

add a comment | 

add a comment | 

active

oldest

votes

Your Answer

StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});

Sam Seo is a new contributor. Be nice, and check out our Code of Conduct.

 

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53372110%2fmigrate-data-from-mysql-to-aws-elasticsearch-logstash%23new-answer', 'question_page');
}
);

Post as a guest

Name

Email

Required, but never shown

active

oldest

votes

active

oldest

votes

active

oldest

votes

active

oldest

votes

Sam Seo is a new contributor. Be nice, and check out our Code of Conduct.

 

draft saved

draft discarded

Sam Seo is a new contributor. Be nice, and check out our Code of Conduct.

Sam Seo is a new contributor. Be nice, and check out our Code of Conduct.

Sam Seo is a new contributor. Be nice, and check out our Code of Conduct.

 

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53372110%2fmigrate-data-from-mysql-to-aws-elasticsearch-logstash%23new-answer', 'question_page');
}
);

Post as a guest

Name

Email

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

Name

Name

Email

Required, but never shown

Email

Required, but never shown

Email

Required, but never shown

Email

Required, but never shown

Name

Name

Email

Required, but never shown

Email

Required, but never shown

Email

Required, but never shown

Email

Required, but never shown

This page is only for reference, If you need detailed information, please check here