How to run Spark Streaming application with Kafka Direct Stream in IntelliJ IDEA?

I am running the program for Spark streaming using Kafka and getting error. All imports are done and looks resolved without any issue.

I have written few code using IntelliJ IDEA and getting an error while running the program for the first time and I am new to Java but from C# background. So couldn't understand the issue. The zookeeper service is started, along with kafka-server started and also created a topic called topicA. Producer is also ready to stream data but i have problem running code in IntelliJ for listening to the queue

def main(args: Array[String]) {

  val kafkaParams = Map[String, Object](

    "bootstrap.servers" -> "localhost:9092",

    "key.deserializer" -> classOf[StringDeserializer],

    "value.deserializer" -> classOf[StringDeserializer],

    "group.id" -> "0",

    "auto.offset.reset" -> "latest",

    "enable.auto.commit" -> (false: java.lang.Boolean)

  )

  val conf = new SparkConf().setAppName("Simple Streaming Application")

  val ssc = new StreamingContext(conf, Seconds(5))

  val topics = Array("topicA")

  val stream = KafkaUtils.createDirectStream[String, String](

    ssc,

    PreferConsistent,

    Subscribe[String, String](topics, kafkaParams)

  )



  stream.foreachRDD { rdd =>

    // Get the offset ranges in the RDD

    val offsetRanges = rdd.asInstanceOf[HasOffsetRanges].offsetRanges

    for (o <- offsetRanges) {

      println(s"${o.topic} ${o.partition} offsets: ${o.fromOffset} to ${o.untilOffset}")

    }

  }



  ssc.start



  // the above code is printing out topic details every 5 seconds

  // until you stop it.



  ssc.stop(stopSparkContext = false)

}

Exception produced is:

Exception in thread "main" java.lang.VerifyError: class scala.collection.mutable.WrappedArray overrides final method toBuffer.()Lscala/collection/mutable/Buffer;

at java.lang.ClassLoader.defineClass1(Native Method)

at java.lang.ClassLoader.defineClass(ClassLoader.java:763)

at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)

at java.net.URLClassLoader.access$100(URLClassLoader.java:74)

at java.net.URLClassLoader$1.run(URLClassLoader.java:369)

at java.net.URLClassLoader$1.run(URLClassLoader.java:363)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:362)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at org.apache.spark.SparkConf.loadFromSystemProperties(SparkConf.scala:75)

at org.apache.spark.SparkConf.<init>(SparkConf.scala:70)

at org.apache.spark.SparkConf.<init>(SparkConf.scala:57)

at sparkStreamClass$.main(sparkStreamClass.scala:20)

at sparkStreamClass.main(sparkStreamClass.scala)

here is my pom.xml

    <?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0"

         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <modelVersion>4.0.0</modelVersion>



    <groupId>com.learnStreaming</groupId>

    <artifactId>sparkProjectArtifact</artifactId>

    <version>1.0-SNAPSHOT</version>



<dependencies>

    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->

    <dependency>

        <groupId>org.apache.spark</groupId>

        <artifactId>spark-core_2.11</artifactId>

        <version>2.3.1</version>

    </dependency>

    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka-0-10 -->

    <dependency>

        <groupId>org.apache.spark</groupId>

        <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>

        <version>2.3.1</version>

    </dependency>

    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming -->

    <dependency>

        <groupId>org.apache.spark</groupId>

        <artifactId>spark-streaming_2.11</artifactId>

        <version>2.3.1</version>

        <scope>provided</scope>

    </dependency>



</dependencies>

</project>

edited Jan 1 at 18:50

Jacek Laskowski

45.6k18135275

asked Dec 30 '18 at 20:19

durgesh yadav

164

1

It's not possible to debug this as-is. Can you show the whole of the source file that contains the main function? Are you using SBT, Maven, Gradle, etc. to retrieve the necessary libraries, configure the compiler, etc. or have you installed them manually? If the former, can you post your build file (e.g. build.sbt, pom.xml, etc.). Which version of Scala are you using and which version of Spark (the latter must support the former)?

– Mike Allen
Dec 30 '18 at 23:51

One guess would be that you are using wrong version of Scala. Unfortunatelly Scala major releases are not binary compatible. So it would really help if you show us your build configuration including all the dependencies

– SergGr
Dec 31 '18 at 19:52

installed scala version on my system is 2.13 and spark version is 2.3.1

– durgesh yadav
Jan 1 at 16:16

add a comment |

I am running the program for Spark streaming using Kafka and getting error. All imports are done and looks resolved without any issue.

def main(args: Array[String]) {

  val kafkaParams = Map[String, Object](

    "bootstrap.servers" -> "localhost:9092",

    "key.deserializer" -> classOf[StringDeserializer],

    "value.deserializer" -> classOf[StringDeserializer],

    "group.id" -> "0",

    "auto.offset.reset" -> "latest",

    "enable.auto.commit" -> (false: java.lang.Boolean)

  )

  val conf = new SparkConf().setAppName("Simple Streaming Application")

  val ssc = new StreamingContext(conf, Seconds(5))

  val topics = Array("topicA")

  val stream = KafkaUtils.createDirectStream[String, String](

    ssc,

    PreferConsistent,

    Subscribe[String, String](topics, kafkaParams)

  )



  stream.foreachRDD { rdd =>

    // Get the offset ranges in the RDD

    val offsetRanges = rdd.asInstanceOf[HasOffsetRanges].offsetRanges

    for (o <- offsetRanges) {

      println(s"${o.topic} ${o.partition} offsets: ${o.fromOffset} to ${o.untilOffset}")

    }

  }



  ssc.start



  // the above code is printing out topic details every 5 seconds

  // until you stop it.



  ssc.stop(stopSparkContext = false)

}

Exception produced is:

Exception in thread "main" java.lang.VerifyError: class scala.collection.mutable.WrappedArray overrides final method toBuffer.()Lscala/collection/mutable/Buffer;

at java.lang.ClassLoader.defineClass1(Native Method)

at java.lang.ClassLoader.defineClass(ClassLoader.java:763)

at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)

at java.net.URLClassLoader.access$100(URLClassLoader.java:74)

at java.net.URLClassLoader$1.run(URLClassLoader.java:369)

at java.net.URLClassLoader$1.run(URLClassLoader.java:363)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:362)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at org.apache.spark.SparkConf.loadFromSystemProperties(SparkConf.scala:75)

at org.apache.spark.SparkConf.<init>(SparkConf.scala:70)

at org.apache.spark.SparkConf.<init>(SparkConf.scala:57)

at sparkStreamClass$.main(sparkStreamClass.scala:20)

at sparkStreamClass.main(sparkStreamClass.scala)

here is my pom.xml

    <?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0"

         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <modelVersion>4.0.0</modelVersion>



    <groupId>com.learnStreaming</groupId>

    <artifactId>sparkProjectArtifact</artifactId>

    <version>1.0-SNAPSHOT</version>



<dependencies>

    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->

    <dependency>

        <groupId>org.apache.spark</groupId>

        <artifactId>spark-core_2.11</artifactId>

        <version>2.3.1</version>

    </dependency>

    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka-0-10 -->

    <dependency>

        <groupId>org.apache.spark</groupId>

        <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>

        <version>2.3.1</version>

    </dependency>

    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming -->

    <dependency>

        <groupId>org.apache.spark</groupId>

        <artifactId>spark-streaming_2.11</artifactId>

        <version>2.3.1</version>

        <scope>provided</scope>

    </dependency>



</dependencies>

</project>

edited Jan 1 at 18:50

Jacek Laskowski

45.6k18135275

asked Dec 30 '18 at 20:19

durgesh yadav

164

1

It's not possible to debug this as-is. Can you show the whole of the source file that contains the main function? Are you using SBT, Maven, Gradle, etc. to retrieve the necessary libraries, configure the compiler, etc. or have you installed them manually? If the former, can you post your build file (e.g. build.sbt, pom.xml, etc.). Which version of Scala are you using and which version of Spark (the latter must support the former)?

– Mike Allen
Dec 30 '18 at 23:51

One guess would be that you are using wrong version of Scala. Unfortunatelly Scala major releases are not binary compatible. So it would really help if you show us your build configuration including all the dependencies

– SergGr
Dec 31 '18 at 19:52

installed scala version on my system is 2.13 and spark version is 2.3.1

– durgesh yadav
Jan 1 at 16:16

add a comment |

I am running the program for Spark streaming using Kafka and getting error. All imports are done and looks resolved without any issue.

def main(args: Array[String]) {

  val kafkaParams = Map[String, Object](

    "bootstrap.servers" -> "localhost:9092",

    "key.deserializer" -> classOf[StringDeserializer],

    "value.deserializer" -> classOf[StringDeserializer],

    "group.id" -> "0",

    "auto.offset.reset" -> "latest",

    "enable.auto.commit" -> (false: java.lang.Boolean)

  )

  val conf = new SparkConf().setAppName("Simple Streaming Application")

  val ssc = new StreamingContext(conf, Seconds(5))

  val topics = Array("topicA")

  val stream = KafkaUtils.createDirectStream[String, String](

    ssc,

    PreferConsistent,

    Subscribe[String, String](topics, kafkaParams)

  )



  stream.foreachRDD { rdd =>

    // Get the offset ranges in the RDD

    val offsetRanges = rdd.asInstanceOf[HasOffsetRanges].offsetRanges

    for (o <- offsetRanges) {

      println(s"${o.topic} ${o.partition} offsets: ${o.fromOffset} to ${o.untilOffset}")

    }

  }



  ssc.start



  // the above code is printing out topic details every 5 seconds

  // until you stop it.



  ssc.stop(stopSparkContext = false)

}

Exception produced is:

Exception in thread "main" java.lang.VerifyError: class scala.collection.mutable.WrappedArray overrides final method toBuffer.()Lscala/collection/mutable/Buffer;

at java.lang.ClassLoader.defineClass1(Native Method)

at java.lang.ClassLoader.defineClass(ClassLoader.java:763)

at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)

at java.net.URLClassLoader.access$100(URLClassLoader.java:74)

at java.net.URLClassLoader$1.run(URLClassLoader.java:369)

at java.net.URLClassLoader$1.run(URLClassLoader.java:363)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:362)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at org.apache.spark.SparkConf.loadFromSystemProperties(SparkConf.scala:75)

at org.apache.spark.SparkConf.<init>(SparkConf.scala:70)

at org.apache.spark.SparkConf.<init>(SparkConf.scala:57)

at sparkStreamClass$.main(sparkStreamClass.scala:20)

at sparkStreamClass.main(sparkStreamClass.scala)

here is my pom.xml

    <?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0"

         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <modelVersion>4.0.0</modelVersion>



    <groupId>com.learnStreaming</groupId>

    <artifactId>sparkProjectArtifact</artifactId>

    <version>1.0-SNAPSHOT</version>



<dependencies>

    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->

    <dependency>

        <groupId>org.apache.spark</groupId>

        <artifactId>spark-core_2.11</artifactId>

        <version>2.3.1</version>

    </dependency>

    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka-0-10 -->

    <dependency>

        <groupId>org.apache.spark</groupId>

        <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>

        <version>2.3.1</version>

    </dependency>

    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming -->

    <dependency>

        <groupId>org.apache.spark</groupId>

        <artifactId>spark-streaming_2.11</artifactId>

        <version>2.3.1</version>

        <scope>provided</scope>

    </dependency>



</dependencies>

</project>

edited Jan 1 at 18:50

Jacek Laskowski

45.6k18135275

asked Dec 30 '18 at 20:19

durgesh yadav

164

I am running the program for Spark streaming using Kafka and getting error. All imports are done and looks resolved without any issue.

def main(args: Array[String]) {

  val kafkaParams = Map[String, Object](

    "bootstrap.servers" -> "localhost:9092",

    "key.deserializer" -> classOf[StringDeserializer],

    "value.deserializer" -> classOf[StringDeserializer],

    "group.id" -> "0",

    "auto.offset.reset" -> "latest",

    "enable.auto.commit" -> (false: java.lang.Boolean)

  )

  val conf = new SparkConf().setAppName("Simple Streaming Application")

  val ssc = new StreamingContext(conf, Seconds(5))

  val topics = Array("topicA")

  val stream = KafkaUtils.createDirectStream[String, String](

    ssc,

    PreferConsistent,

    Subscribe[String, String](topics, kafkaParams)

  )



  stream.foreachRDD { rdd =>

    // Get the offset ranges in the RDD

    val offsetRanges = rdd.asInstanceOf[HasOffsetRanges].offsetRanges

    for (o <- offsetRanges) {

      println(s"${o.topic} ${o.partition} offsets: ${o.fromOffset} to ${o.untilOffset}")

    }

  }



  ssc.start



  // the above code is printing out topic details every 5 seconds

  // until you stop it.



  ssc.stop(stopSparkContext = false)

}

Exception produced is:

Exception in thread "main" java.lang.VerifyError: class scala.collection.mutable.WrappedArray overrides final method toBuffer.()Lscala/collection/mutable/Buffer;

at java.lang.ClassLoader.defineClass1(Native Method)

at java.lang.ClassLoader.defineClass(ClassLoader.java:763)

at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)

at java.net.URLClassLoader.access$100(URLClassLoader.java:74)

at java.net.URLClassLoader$1.run(URLClassLoader.java:369)

at java.net.URLClassLoader$1.run(URLClassLoader.java:363)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:362)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at org.apache.spark.SparkConf.loadFromSystemProperties(SparkConf.scala:75)

at org.apache.spark.SparkConf.<init>(SparkConf.scala:70)

at org.apache.spark.SparkConf.<init>(SparkConf.scala:57)

at sparkStreamClass$.main(sparkStreamClass.scala:20)

at sparkStreamClass.main(sparkStreamClass.scala)

here is my pom.xml

    <?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0"

         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <modelVersion>4.0.0</modelVersion>



    <groupId>com.learnStreaming</groupId>

    <artifactId>sparkProjectArtifact</artifactId>

    <version>1.0-SNAPSHOT</version>



<dependencies>

    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->

    <dependency>

        <groupId>org.apache.spark</groupId>

        <artifactId>spark-core_2.11</artifactId>

        <version>2.3.1</version>

    </dependency>

    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka-0-10 -->

    <dependency>

        <groupId>org.apache.spark</groupId>

        <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>

        <version>2.3.1</version>

    </dependency>

    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming -->

    <dependency>

        <groupId>org.apache.spark</groupId>

        <artifactId>spark-streaming_2.11</artifactId>

        <version>2.3.1</version>

        <scope>provided</scope>

    </dependency>



</dependencies>

</project>

scala apache-spark apache-kafka spark-streaming

edited Jan 1 at 18:50

Jacek Laskowski

45.6k18135275

asked Dec 30 '18 at 20:19

durgesh yadav

164

edited Jan 1 at 18:50

Jacek Laskowski

45.6k18135275

asked Dec 30 '18 at 20:19

durgesh yadav

164

edited Jan 1 at 18:50

Jacek Laskowski

45.6k18135275

edited Jan 1 at 18:50

Jacek Laskowski

45.6k18135275

edited Jan 1 at 18:50

Jacek Laskowski

45.6k18135275

asked Dec 30 '18 at 20:19

durgesh yadav

164

asked Dec 30 '18 at 20:19

durgesh yadav

164

asked Dec 30 '18 at 20:19

durgesh yadav

164

1

It's not possible to debug this as-is. Can you show the whole of the source file that contains the main function? Are you using SBT, Maven, Gradle, etc. to retrieve the necessary libraries, configure the compiler, etc. or have you installed them manually? If the former, can you post your build file (e.g. build.sbt, pom.xml, etc.). Which version of Scala are you using and which version of Spark (the latter must support the former)?

– Mike Allen
Dec 30 '18 at 23:51

One guess would be that you are using wrong version of Scala. Unfortunatelly Scala major releases are not binary compatible. So it would really help if you show us your build configuration including all the dependencies

– SergGr
Dec 31 '18 at 19:52

installed scala version on my system is 2.13 and spark version is 2.3.1

– durgesh yadav
Jan 1 at 16:16

add a comment |

1

It's not possible to debug this as-is. Can you show the whole of the source file that contains the main function? Are you using SBT, Maven, Gradle, etc. to retrieve the necessary libraries, configure the compiler, etc. or have you installed them manually? If the former, can you post your build file (e.g. build.sbt, pom.xml, etc.). Which version of Scala are you using and which version of Spark (the latter must support the former)?

– Mike Allen
Dec 30 '18 at 23:51

One guess would be that you are using wrong version of Scala. Unfortunatelly Scala major releases are not binary compatible. So it would really help if you show us your build configuration including all the dependencies

– SergGr
Dec 31 '18 at 19:52

installed scala version on my system is 2.13 and spark version is 2.3.1

– durgesh yadav
Jan 1 at 16:16

It's not possible to debug this as-is. Can you show the whole of the source file that contains the main function? Are you using SBT, Maven, Gradle, etc. to retrieve the necessary libraries, configure the compiler, etc. or have you installed them manually? If the former, can you post your build file (e.g. build.sbt, pom.xml, etc.). Which version of Scala are you using and which version of Spark (the latter must support the former)?

– Mike Allen
Dec 30 '18 at 23:51

One guess would be that you are using wrong version of Scala. Unfortunatelly Scala major releases are not binary compatible. So it would really help if you show us your build configuration including all the dependencies

– SergGr
Dec 31 '18 at 19:52

installed scala version on my system is 2.13 and spark version is 2.3.1

– durgesh yadav
Jan 1 at 16:16

add a comment |

1 Answer
1

active

oldest

votes

modified pom.xml and it worked for me !

 <properties>

    <spark.version>2.1.0</spark.version>

</properties>



<dependencies>

    <dependency>

        <groupId>org.apache.spark</groupId>

        <artifactId>spark-core_2.11</artifactId>

        <version>${spark.version}</version>

    </dependency>

    <dependency>

        <groupId>org.apache.spark</groupId>

        <artifactId>spark-streaming_2.11</artifactId>

        <version>${spark.version}</version>

    </dependency>

    <dependency>

        <groupId>org.apache.spark</groupId>

        <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>

        <version>${spark.version}</version>

    </dependency>

</dependencies>

answered Jan 1 at 18:16

durgesh yadav

164

add a comment |

Your Answer

StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});

}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53981121%2fhow-to-run-spark-streaming-application-with-kafka-direct-stream-in-intellij-idea%23new-answer', 'question_page');
}
);

Post as a guest

Name

Required, but never shown

1 Answer
1

active

oldest

votes

1 Answer
1

active

oldest

votes

modified pom.xml and it worked for me !

 <properties>

    <spark.version>2.1.0</spark.version>

</properties>



<dependencies>

    <dependency>

        <groupId>org.apache.spark</groupId>

        <artifactId>spark-core_2.11</artifactId>

        <version>${spark.version}</version>

    </dependency>

    <dependency>

        <groupId>org.apache.spark</groupId>

        <artifactId>spark-streaming_2.11</artifactId>

        <version>${spark.version}</version>

    </dependency>

    <dependency>

        <groupId>org.apache.spark</groupId>

        <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>

        <version>${spark.version}</version>

    </dependency>

</dependencies>

answered Jan 1 at 18:16

durgesh yadav

164

add a comment |

modified pom.xml and it worked for me !

 <properties>

    <spark.version>2.1.0</spark.version>

</properties>



<dependencies>

    <dependency>

        <groupId>org.apache.spark</groupId>

        <artifactId>spark-core_2.11</artifactId>

        <version>${spark.version}</version>

    </dependency>

    <dependency>

        <groupId>org.apache.spark</groupId>

        <artifactId>spark-streaming_2.11</artifactId>

        <version>${spark.version}</version>

    </dependency>

    <dependency>

        <groupId>org.apache.spark</groupId>

        <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>

        <version>${spark.version}</version>

    </dependency>

</dependencies>

answered Jan 1 at 18:16

durgesh yadav

164

add a comment |

modified pom.xml and it worked for me !

 <properties>

    <spark.version>2.1.0</spark.version>

</properties>



<dependencies>

    <dependency>

        <groupId>org.apache.spark</groupId>

        <artifactId>spark-core_2.11</artifactId>

        <version>${spark.version}</version>

    </dependency>

    <dependency>

        <groupId>org.apache.spark</groupId>

        <artifactId>spark-streaming_2.11</artifactId>

        <version>${spark.version}</version>

    </dependency>

    <dependency>

        <groupId>org.apache.spark</groupId>

        <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>

        <version>${spark.version}</version>

    </dependency>

</dependencies>

answered Jan 1 at 18:16

durgesh yadav

164

modified pom.xml and it worked for me !

 <properties>

    <spark.version>2.1.0</spark.version>

</properties>



<dependencies>

    <dependency>

        <groupId>org.apache.spark</groupId>

        <artifactId>spark-core_2.11</artifactId>

        <version>${spark.version}</version>

    </dependency>

    <dependency>

        <groupId>org.apache.spark</groupId>

        <artifactId>spark-streaming_2.11</artifactId>

        <version>${spark.version}</version>

    </dependency>

    <dependency>

        <groupId>org.apache.spark</groupId>

        <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>

        <version>${spark.version}</version>

    </dependency>

</dependencies>

answered Jan 1 at 18:16

durgesh yadav

164

answered Jan 1 at 18:16

durgesh yadav

164

answered Jan 1 at 18:16

durgesh yadav

164

answered Jan 1 at 18:16

durgesh yadav

164

add a comment |

draft saved

draft discarded

Thanks for contributing an answer to Stack Overflow!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

Search This Blog

Ufyukyu