I created a simple SBT project in IntelliJ IDE with the following library dependencies in build.sbt:
import _root_.sbt.Keys._
name := "untitled"
version := "1.0"
scalaVersion := "2.11.7"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.5.1",
"org.apache.spark" %% "spark-sql" % "1.5.1" ,
"org.apache.spark" %% "spark-mllib" % "1.5.1")
The objective is to import Spark and MLLIB of Spark, and then to create a Scala object as explained here.
However, the following error occurs on importing:
SBT project import
[warn] Multiple dependencies with the same organization/name but different versions. To avoid conflict, pick one version: [warn] *
org.scala-lang:scala-compiler:(2.11.0, 2.11.7) [warn] *
org.apache.commons:commons-lang3:(3.3.2, 3.0) [warn] *
jline:jline:(0.9.94, 2.12.1) [warn] *
org.scala-lang.modules:scala-parser-combinators_2.11:(1.0.1, 1.0.4)
[warn] * org.scala-lang.modules:scala-xml_2.11:(1.0.1, 1.0.4) [warn] *
org.slf4j:slf4j-api:(1.7.10, 1.7.2) [warn] [FAILED ]
net.sourceforge.f2j#arpack_combined_all;0.1!arpack_combined_all.jar(src):
(0ms) [warn] ==== local: tried [warn]
C:\Users\Cezar.ivy2\local\net.sourceforge.f2j\arpack_combined_all\0.1\srcs\arpack_combined_all-sources.jar
[warn] ==== public: tried [warn]
https://repo1.maven.org/maven2/net/sourceforge/f2j/arpack_combined_all/0.1/arpack_combined_all-0.1-sources.jar
[warn] [FAILED ] javax.xml.bind#jsr173_api;1.0!jsr173_api.jar(doc):
(0ms) [warn] ==== local: tried [warn]
C:\Users\Cezar.ivy2\local\javax.xml.bind\jsr173_api\1.0\docs\jsr173_api-javadoc.jar
[warn] ==== public: tried [warn]
https://repo1.maven.org/maven2/javax/xml/bind/jsr173_api/1.0/jsr173_api-1.0-javadoc.jar
[warn] [FAILED ] javax.xml.bind#jsr173_api;1.0!jsr173_api.jar(src):
(0ms) [warn] ==== local: tried [warn]
C:\Users\Cezar.ivy2\local\javax.xml.bind\jsr173_api\1.0\srcs\jsr173_api-sources.jar
[warn] ==== public: tried [warn]
https://repo1.maven.org/maven2/javax/xml/bind/jsr173_api/1.0/jsr173_api-1.0-sources.jar
[warn] :::::::::::::::::::::::::::::::::::::::::::::: [warn] :: FAILED
DOWNLOADS :: [warn] :: ^ see resolution messages for details ^ ::
[warn] :::::::::::::::::::::::::::::::::::::::::::::: [warn] ::
net.sourceforge.f2j#arpack_combined_all;0.1!arpack_combined_all.jar(src)
[warn] :: javax.xml.bind#jsr173_api;1.0!jsr173_api.jar(doc) [warn] ::
javax.xml.bind#jsr173_api;1.0!jsr173_api.jar(src) [warn]
::::::::::::::::::::::::::::::::::::::::::::::
Spark will not work with Scala 2.11. It uses Scala 2.10, so you need to use a compatible Scala version (see http://spark.apache.org/docs/latest/).
Alternatively, you can, as #eliasah has also mentioned in a comment, build Spark yourself. Instructions on how to build Spark can be found at http://spark.apache.org/docs/latest/building-spark.html
Related
I created new play-java application by using command from play website
sbt new playframework/play-java-seed.g8
Then added EBean dependencies like below (link):
addSbtPlugin("com.typesafe.sbt" % "sbt-play-ebean" % "4.0.1")
And see below message. How in sbt can I verify what exactly is wrong?
::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: com.typesafe.sbt#sbt-play-ebean;4.0.1: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn] Note: Some unresolved dependencies have extra attributes. Check that these dependencies exist with the requested attributes.
[warn] com.typesafe.sbt:sbt-play-ebean:4.0.1 (scalaVersion=2.12, sbtVersion=1.0)
[warn]
[warn] Note: Unresolved dependencies path:
[warn] com.typesafe.sbt:sbt-play-ebean:4.0.1 (scalaVersion=2.12, sbtVersion=1.0) (/home/user/src/github/test-proj/project/plugins.sbt#L5-6)
[warn] +- default:test-proj-build:0.1-SNAPSHOT (scalaVersion=2.12, sbtVersion=1.0)
[error] sbt.librarymanagement.ResolveException: unresolved dependency: com.typesafe.sbt#sbt-play-ebean;4.0.1: not found
The version 4.0.1 of sbt-play-ebean is not available for Scala 2.12 and Sbt 1.x.
You should upgrade the sbt-play-ebean to 4.1.0 or at least to 4.0.6.
For reference see Bintray repo
I would like to use the scala-xml jar with my project, I have the following configuration for the fat jar.
In a previous project, I was able to use XML locally using the version:
lazy val scalaXml = "org.scala-lang" % "scala-xml" % "2.11.0-M4"
but when I try and use that version with my fat jar, I get the error from my deployed jar( however, my tests still run correctly on my local machine):
scala/Product$class: java.lang.NoClassDefFoundError
java.lang.NoClassDefFoundError: scala/Product$class
at scala.xml.NamespaceBinding.<init>(NamespaceBinding.scala:23)
at scala.xml.TopScope$.<init>(TopScope.scala:17)
at scala.xml.TopScope$.<clinit>(TopScope.scala)
at example.models.soap11.SoapEnvelope$.toXML(SoapEnvelope.scala:9)
at example.AWSLambdaRequestHandler.execute(RequestHandler.scala:73)
scala-xml related
lazy val scalaXml = "org.scala-lang.modules" %% "scala-xml" % "1.0.6"
Error(tl;dr)
org.scala-lang#scala-library;2.12: not found
[warn] :: org.scala-lang#scala-reflect;2.12: not found
[warn] :: org.scala-lang#scala-compiler;2.12: not found
I have gone through this issue where the solution recommended was to do sbt update but that did not resolve my issue. Instead it resulted in the Error message above.
The build.sbt file of the project here seems to use
libraryDependencies += ("org.scala-lang" % "scala-compiler" % scalaVersion.value % "test").exclude("org.scala-lang.modules", s"scala-xml_${scalaVersion.value}")
as a test dependency. There are 2 other versions defined there:
version := "1.1.0-SNAPSHOT",
mimaPreviousVersion := Some("1.0.6"),
I have also tried the following configurations:
lazy val scalaXml = "org.scala-lang.modules" % "scala-xml_2.12" % "1.0.6"
lazy val scalaXml = "org.scala-lang.modules" %% "scala-xml_2.12" % "1.0.6"
lazy val scalaXml = "org.scala-lang" %% "scala-xml" % "1.0.6"
Detailed config and Error message:
build.sbt(Entire file)
import sbt.Keys.{libraryDependencies, _}
import sbt._
val lambdaVersion = "1.2.0"
val awsVersion = "1.11.22"
val json4sVersion = "3.6.0-M2"
val sttpVersion = "1.1.1"
lazy val scalaXml = "org.scala-lang.modules" %% "scala-xml" % "1.0.6"
lazy val sttp = Seq(
"com.softwaremill.sttp" %% "core",
"com.softwaremill.sttp" %% "json4s"
).map(_ % sttpVersion)
lazy val json4s = Seq(
"org.json4s" %% "json4s-native",
"org.json4s" %% "json4s-ext"
).map(_ % json4sVersion)
lazy val awsLambda = Seq(
"com.amazonaws" % "aws-lambda-java-events",
"com.amazonaws" % "aws-lambda-java-core"
).map(_ % lambdaVersion)
lazy val aws = Seq(
"com.amazonaws" % "aws-java-sdk-core",
"com.amazonaws" % "aws-java-sdk-s3"
).map (_ % awsVersion)
javacOptions ++= Seq("-source", "1.8", "-target", "1.8", "-Xlint")
lazy val root = (project in file(".")).
settings(
inThisBuild(List(
organization := "example",
scalaVersion := "2.12",
version := "0.1.0-SNAPSHOT",
retrieveManaged := true
)),
name := "aws-scala-lambda-test",
libraryDependencies += scalaXml,
libraryDependencies ++= json4s,
libraryDependencies ++= awsLambda,
libraryDependencies ++= aws,
libraryDependencies ++= sttp,
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.3" % Test,
libraryDependencies += "org.mockito" % "mockito-all" % "1.10.19" % Test
)
assemblyJarName in assembly := "my-example.jar"
assemblyMergeStrategy in assembly := {
case PathList("META-INF", _ # _*) => MergeStrategy.discard
case _ => MergeStrategy.first
}
Error
$ sbt
> update
[info] Resolving org.scala-lang#scala-reflect;2.12 ...
[info] Resolving org.scala-lang#scala-reflect;2.12 ...
[warn] module not found: org.scala-lang#scala-reflect;2.12
[warn] ==== local: tried
[warn] /Users/localuser/.ivy2/local/org.scala-lang/scala-reflect/2.12/ivys/ivy.xml
[warn] ==== public: tried
[warn] https://repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.12/scala-reflect-2.12.pom
[warn] ==== local-preloaded-ivy: tried
[warn] /Users/localuser/.sbt/preloaded/org.scala-lang/scala-reflect/2.12/ivys/ivy.xml
[warn] ==== local-preloaded: tried
[warn] file:////Users/localuser/.sbt/preloaded/org/scala-lang/scala-reflect/2.12/scala-reflect-2.12.pom
[info] Resolving org.scala-lang.modules#scala-parser-combinators_2.12;1.0.4 ...
[info] Resolving org.mockito#mockito-all;1.10.19 ...
[info] Resolving org.scala-lang#scala-compiler;2.12 ...
[info] Resolving org.scala-lang#scala-compiler;2.12 ...
[warn] module not found: org.scala-lang#scala-compiler;2.12
[warn] ==== local: tried
[warn] /Users/localuser/.ivy2/local/org.scala-lang/scala-compiler/2.12/ivys/ivy.xml
[warn] ==== public: tried
[warn] https://repo1.maven.org/maven2/org/scala-lang/scala-compiler/2.12/scala-compiler-2.12.pom
[warn] ==== local-preloaded-ivy: tried
[warn] /Users/localuser/.sbt/preloaded/org.scala-lang/scala-compiler/2.12/ivys/ivy.xml
[warn] ==== local-preloaded: tried
[warn] file:////Users/localuser/.sbt/preloaded/org/scala-lang/scala-compiler/2.12/scala-compiler-2.12.pom
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: org.scala-lang#scala-library;2.12: not found
[warn] :: org.scala-lang#scala-reflect;2.12: not found
[warn] :: org.scala-lang#scala-compiler;2.12: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
I am doing a Scala project, built with SBT and trying to follow the documentation in:
Elasticsearch website, about their test framework: https://www.elastic.co/guide/en/elasticsearch/reference/current/using-elasticsearch-test-classes.html
There, they talk about using the same versions of ES, which in my case would be ES: 2.2.0 and lucene: 5.4.1
However, I am not able to build my project, due to dependencies issues with the test framework artifact and version.
I have added:
"org.elasticsearch.test" % "framework" % "2.2.0"
But SBT fails with:
Error:Error while importing SBT project:<br/>...<br/><pre>[info]
Resolving commons-io#commons-io;2.4 ... [info] Resolving io.spray#spray-
json_2.12;1.3.2 ... [info] Resolving com.typesafe.akka#akka-http-
testkit_2.12;10.0.1 ... [info] Resolving com.typesafe.akka#akka-stream-
testkit_2.12;2.4.16 ... [info] Resolving com.typesafe.akka#akka-
testkit_2.12;2.4.16 ... [info] Resolving jline#jline;2.14.1 ... [warn]
:::::::::::::::::::::::::::::::::::::::::::::: [warn] ::
UNRESOLVED DEPENDENCIES :: [warn] :::::::::::::::::::::::::::::::::::::::::::::: [warn] ::
org.elasticsearch.test#framework;2.2.0: not found [warn]
:::::::::::::::::::::::::::::::::::::::::::::: [warn] [warn] Note:
Unresolved dependencies path: [warn]
org.elasticsearch.test:framework:2.2.0
(/Users/filipemiranda/datawerks/scala-query-engine/build.sbt#L13-30)
[warn] +- default:scala-query-engine_2.12:1.0 [trace] Stack trace
suppressed: run 'last *:update' for the full output. [trace] Stack trace
suppressed: run 'last *:ssExtractDependencies' for the full output.
[error] (*:update) sbt.ResolveException: unresolved dependency:
org.elasticsearch.test#framework;2.2.0: not found [error]
(*:ssExtractDependencies) sbt.ResolveException: unresolved dependency:
org.elasticsearch.test#framework;2.2.0: not found [error] Total time: 6 s,
completed Jan 21, 2017 8:20:44 PM</pre><br/>See complete log in <a
href="/Users/filipemiranda/Library/Logs/IdeaIC2016.1/sbt.last.log">/Users/
filipemiranda/Library/Logs/IdeaIC2016.1/sbt.last.log</a>
I ave looked at: https://oss.sonatype.org/content/repositories/releases/org/elasticsearch/test/framework/
And I did not find the same version there.
Has anyone faced similar issues, how to solve this?
Here is my entire build.sbt
name := "project-name"
version := "1.0"
lazy val scalaV = "2.12.1"
scalaVersion := scalaV
lazy val akkaHttpV = "10.0.1"
lazy val esVersion = "2.2.0"
lazy val luceneVersion = "5.4.1"
lazy val scalatestV = "3.0.1"
libraryDependencies ++= Seq(
"org.scala-lang" % "scala-compiler" % scalaV,
"org.scalatest" %% "scalatest" % scalatestV,
"com.typesafe.akka" %% "akka-http-core" % akkaHttpV,
"com.typesafe.akka" %% "akka-http" % akkaHttpV,
"com.typesafe.akka" %% "akka-stream" % "2.4.16",
"com.storm-enroute" %% "scalameter" % "0.8.2" % "test",
"io.searchbox" % "jest" % "2.0.3",
"org.elasticsearch" % "elasticsearch" % "2.2.0" withSources,
"com.twitter" %% "util-eval" % "6.40.0",
"org.apache.lucene" % "lucene-test-framework" % "5.4.1",
"org.elasticsearch.test" % "framework" % "2.2.0",
"com.storm-enroute" %% "scalameter" % "0.8.2" % "test",
"com.typesafe.akka" %% "akka-http-testkit" % akkaHttpV % "test"
)
How can I work this around???
Sorry in advance for my bad english..
Today I wanted to implement Deadbolt in my project.
I read the documentation in this webstite, but at the begining I had problems.
At the first instruction it says that I need to add these lines in my built.sbt file.
libraryDependencies ++= Seq(
"be.objectify" %% "deadbolt-scala" % "2.5.1-SNAPSHOT"
)
resolvers += Resolver.sonatypeRepo("snapshots")
I use IntellijIDEA and when I refresh the project I have this error.
SBT 'Simulateur' project refresh failed
Error:Error:Error while importing SBT project:
[info] Resolving com.typesafe.play#play-doc_2.11;1.2.2 ...
[info] Resolving org.pegdown#pegdown;1.4.0 ...
[info] Resolving org.parboiled#parboiled-java;1.1.5 ...
[info] Resolving org.parboiled#parboiled-core;1.1.5 ...
[info] Resolving org.ow2.asm#asm;4.1 ...
[info] Resolving org.ow2.asm#asm-tree;4.1 ...
[info] Resolving org.ow2.asm#asm-analysis;4.1 ...
[info] Resolving org.ow2.asm#asm-util;4.1 ...
[info] Resolving jline#jline;2.12.1 ...
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: be.objectify#deadbolt-scala_2.11;2.5.1-SNAPSHOT: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[trace] Stack trace suppressed: run 'last *:update' for the full output.
[trace] Stack trace suppressed: run 'last :ssExtractDependencies' for the full output.
[error] (:update) sbt.ResolveException: unresolved dependency:be.objectify#deadbolt-scala_2.11;2.5.1-SNAPSHOT: not found
[error] (*:ssExtractDependencies) sbt.ResolveException: unresolved dependency: be.objectify#deadbolt-scala_2.11;2.5.1-SNAPSHOT: not found
[error] Total time: 18 s, completed 20 déc. 2016 22:00:15
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=384M; support was removed in 8.0See complete log in C:\Users\PCDamien.IntelliJIdea2016.2\system\log\sbt.last.log
Now I will show my built.sbt file
name := "Simulateur"
version := "1.0"
lazy val `simulateur` = (project in file(".")).enablePlugins(PlayScala)
scalaVersion := "2.11.7"
libraryDependencies ++= Seq( jdbc , cache , ws , specs2 % Test, "be.objectify" %% "deadbolt-scala" % "2.5.1-SNAPSHOT" )
unmanagedResourceDirectories in Test <+= baseDirectory ( _ /"target/web/public/test" )
resolvers += "scalaz-bintray" at "https://dl.bintray.com/scalaz/releases"
resolvers += Resolver.sonatypeRepo("snapshots")
routesGenerator := StaticRoutesGenerator
Thank you in advance for your help ! :)
Sorry, this is entirely my fault. I released 2.5.1 a couple of days ago and forgot to update the version when I moved the documentation from beta to final.
The correct dependency is
libraryDependencies ++= Seq(
"be.objectify" %% "deadbolt-scala" % "2.5.1"
)
This has been corrected on the web site you reference.
I tried to use TestNG with Apache Ivy, but was unsuccessful. Here is my ivy.xml:
<ivy-module version="2.0">
<info organisation="me" module="myproject"/>
<dependencies>
<dependency org="org.testng" name="testng" rev="5.8" />
</dependencies>
</ivy-module>
This fails to actually download a TestNG jarfile. It seems to be because TestNG has a jarfile for jdk14 and jdk15. Here's the output from ivy:retrieve:
[ivy:retrieve] :: resolving dependencies :: me#myproject;working#jared-mbp17
[ivy:retrieve] confs: [default]
[ivy:retrieve] found org.testng#testng;5.8 in public
[ivy:retrieve] :: resolution report :: resolve 1139ms :: artifacts dl 11ms
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 1 | 1 | 1 | 0 || 1 | 0 |
---------------------------------------------------------------------
[ivy:retrieve]
[ivy:retrieve] :: problems summary ::
[ivy:retrieve] :::: WARNINGS
[ivy:retrieve] [FAILED ] org.testng#testng;5.8!testng.jar: (0ms)
[ivy:retrieve] ==== shared: tried
[ivy:retrieve] /Users/jared/.ivy2/shared/org.testng/testng/5.8/jars/testng.jar
[ivy:retrieve] ==== public: tried
[ivy:retrieve] http://repo1.maven.org/maven2/org/testng/testng/5.8/testng-5.8.jar
[ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:retrieve] :: FAILED DOWNLOADS ::
[ivy:retrieve] :: ^ see resolution messages for details ^ ::
[ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:retrieve] :: org.testng#testng;5.8!testng.jar
[ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:retrieve]
[ivy:retrieve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
In the repository you can see two different jarfiles:
http://repo1.maven.org/maven2/org/testng/testng/5.8/testng-5.8-jdk14.jar
http://repo1.maven.org/maven2/org/testng/testng/5.8/testng-5.8-jdk15.jar
How do I specify either jdk14 or jdk15?
You need to specify the classifier of the artifact you want.
There is a related fix as of 2.1.0-RC1. You can use the element artifact within the dependency element to specify the classifier you want. In this case, the classifier should be jdk14 or jdk15. If you want jdk15 your ivy.xml would then be:
<ivy-module version="2.0"
xmlns:e="http://ant.apache.org/ivy/extra">
<info organisation="me" module="myproject"/>
<dependencies>
<dependency org="org.testng" name="testng" rev="5.8"
transitive="false">
<artifact name="testng" type="jar" ext="jar" e:classifier="jdk15" />
</dependency>
</dependencies>
</ivy-module>
Note the specification of the XML namespace "http://ant.apache.org/ivy/extra" as an attribute of the ivy-module element. Without that the e:classifier won't work.