We Keep Coding

sbt can not find the ebean dependency (Some unresolved dependencies have extra attributes)

I created new play-java application by using command from play website
sbt new playframework/play-java-seed.g8
Then added EBean dependencies like below (link):
addSbtPlugin("com.typesafe.sbt" % "sbt-play-ebean" % "4.0.1")
And see below message. How in sbt can I verify what exactly is wrong?
::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: com.typesafe.sbt#sbt-play-ebean;4.0.1: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn] Note: Some unresolved dependencies have extra attributes. Check that these dependencies exist with the requested attributes.
[warn] com.typesafe.sbt:sbt-play-ebean:4.0.1 (scalaVersion=2.12, sbtVersion=1.0)
[warn]
[warn] Note: Unresolved dependencies path:
[warn] com.typesafe.sbt:sbt-play-ebean:4.0.1 (scalaVersion=2.12, sbtVersion=1.0) (/home/user/src/github/test-proj/project/plugins.sbt#L5-6)
[warn] +- default:test-proj-build:0.1-SNAPSHOT (scalaVersion=2.12, sbtVersion=1.0)
[error] sbt.librarymanagement.ResolveException: unresolved dependency: com.typesafe.sbt#sbt-play-ebean;4.0.1: not found

The version 4.0.1 of sbt-play-ebean is not available for Scala 2.12 and Sbt 1.x.
You should upgrade the sbt-play-ebean to 4.1.0 or at least to 4.0.6.
For reference see Bintray repo

filesystem resolver and transitive dependencies / master configurations

I have setup Ivy with a filesystem resolver as explained in this answer. I.e. placing my jars in a 'lib' directory and configuring a filesystem resolver to pick them up from there (instead of using the default ibiblio resolver).
The file ivysettings.xml is as follows:
<ivysettings>
<caches defaultCacheDir="${ivy.settings.dir}/cache"/>
<settings defaultResolver="local"/>
<resolvers>
<filesystem name="local">
<artifact pattern="${ivy.settings.dir}/lib/[artifact]-[revision].[ext]"/>
</filesystem>
</resolvers>
</ivysettings>
My ivy.xml is also very simple and for test purposes I only define a single configuration:
<?xml version="1.0" encoding="ISO-8859-1"?>
<ivy-module version="2.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="http://ant.apache.org/ivy/schemas/ivy.xsd">
<info organisation="foo" module="foo" status="foo"/>
<configurations>
<conf name="theConf" description="just one configuration"/>
</configurations>
<dependencies>
<dependency org="org.apache.commons" name="commons-lang3" rev="3.1" conf="theConf->default"/>
</dependencies>
</ivy-module>
The above works and the dependency is properly retrieved from the lib directory. However I've noticed that if I change the dependency element in the ivy.xml file to fetch the "master" as opposed to the "default" configuration:
<dependency org="org.apache.commons" name="commons-lang3" rev="3.1" conf="theConf->master"/>
… then it no longer works and I get the following:
resolve: [ivy:resolve] :: Apache Ivy 2.4.0-local-20161112135640 -
20161112135640 :: http://ant.apache.org/ivy/ :: [ivy:resolve] ::
loading settings :: file =
/home/mperdikeas/play/ivy-with-local-filesystem/ivysettings.xml
[ivy:resolve] :: resolving dependencies :: foo#foo;working#mp-t420
[ivy:resolve] confs: [theConf] [ivy:resolve] found
org.apache.commons#commons-lang3;3.1 in local [ivy:resolve] ::
resolution report :: resolve 66ms :: artifacts dl 0ms
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| theConf | 1 | 0 | 0 | 0 || 0 | 0 |
---------------------------------------------------------------------
[ivy:resolve] [ivy:resolve] :: problems summary :: [ivy:resolve] ::::
WARNINGS [ivy:resolve]
:::::::::::::::::::::::::::::::::::::::::::::: [ivy:resolve] ::
UNRESOLVED DEPENDENCIES :: [ivy:resolve]
:::::::::::::::::::::::::::::::::::::::::::::: [ivy:resolve] ::
org.apache.commons#commons-lang3;3.1: configuration not found in
org.apache.commons#commons-lang3;3.1: 'master'. It was required from
foo#foo;working#mp-t420 theConf [ivy:resolve]
:::::::::::::::::::::::::::::::::::::::::::::: [ivy:resolve]
[ivy:resolve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
So even though the module is found it's "master" configuration is not found.
To make this work I either need to change theConf->master to theConf->default or simply remove the conf mapping altogether.
Further reflecting upon this I realized that I also don't understand how a filesystem resolver that's picking up jars laid out flat insider a directory is supposed to distinguish between different configurations and fetch, e.g. only compile time versus transitive dependencies of a module.
At any rate having to always use the "default" configuration causes me problems as I have to modify my ivy.xml file. When I resolve using the ibiblio resolver I typically have in my ivy.xml either:
conf="compile->master"
… or:
conf="runtime->default"
… depending on whether I want the transitive dependencies or not. I would like to keep the same ivy.xml file regardless of the resolver I have configured to use in ivysettings.xml.
My questions are:
is it possible to keep the same ivy.xml file for both the ibiblio and the filesystem resolver and what to do with "master" configurations in such a case?
how are configurations understood by a filesystem resolver that's picking up jars that are laid flat inside a directory?
is it possible to configure a filesystem resolver such that transitive dependencies can also be fetched with a dependency and what should be the file system structure in such a case?
I am simply copying jars inside the directory that the filesystem resolver is looking at. Should I be using some importation mechanism instead (e.g. to create the proper directory structure that might perhaps allow the resolver to also fetch transitive dependencies).

I gave up using a flat directory structure with just a bunch of jars. It is clear to me now that in addition to the jars, declarative files need to be present to properly identify the dependencies of each module.
What I have discovered works is to use the ivy:install task to import the dependencies I need from the ibiblio resolver into my local filesystem resolver. More or less as described in this post.
I've created a build-import-Maven-dependencies-to-local.xml script to automate this process. With some added "cleverness" (which is not crucial and you can dispense with) for aligning all the dependencies visually line after line it looks like the following (the gist is the last target, "install"):
<project name="local repository importation" default="install-deps-locally"
xmlns:contrib="http://net.sf.antcontrib"
xmlns:ivy="antlib:org.apache.ivy.ant">
<taskdef uri="http://net.sf.antcontrib" resource="net/sf/antcontrib/antlib.xml" classpath="${basedir}/tools/ant-contrib.jar" />
<target name="install-deps-locally" description="local at non default location">
<!-- "iw" starts for "Import Wrapper" and "d" stands for "dependency" -->
<antcall target="iw"><param name="d" value="javax.servlet javax.servlet-api 3.1.0" /></antcall>
<antcall target="iw"><param name="d" value="com.google.code.gson gson 2.8.0" /></antcall>
<antcall target="iw"><param name="d" value="junit junit 4.12" /></antcall>
</target>
<target name="iw"> <!-- "iw" stands for "Import Wrapper" and "d" stands for "dependency" -->
<property name="REGULAR_EXPRESSION" value="(\S*)(\s*)(\S*)(\s*)(\S*)"/>
<contrib:propertyregex property="organisation"
input="${d}"
regexp="${REGULAR_EXPRESSION}"
select="\1"
casesensitive="false"/>
<contrib:propertyregex property="module"
input="${d}"
regexp="${REGULAR_EXPRESSION}"
select="\3"
casesensitive="false"/>
<contrib:propertyregex property="revision"
input="${d}"
regexp="${REGULAR_EXPRESSION}"
select="\5"
casesensitive="false"/>
<antcall target="install">
<param name="organisation" value="${organisation}"/>
<param name="module" value="${module}"/>
<param name="revision" value="${revision}"/>
</antcall>
</target>
<target name="install" description="import module from public Maven repository into local repository">
<ivy:settings id="ivysettings-ibiblio-to-local" file="ivysettings-ibiblio-to-local.xml"/>
<ivy:install settingsRef="ivysettings-ibiblio-to-local"
organisation="${organisation}"
module="${module}"
revision="${revision}"
from="public"
to="local"
transitive="true"
overwrite="true"/>
</target>
</project>
Resolvers "public" and "local" are defined in the ivysettings.file and correspond to ibiblio and filesystem respectively.
This creates the correct structure and places ivy-x.y.z.xml files that provide the necessary information. This is a partial example of the directory structure that got created in my system and the files that are present.
$ tree repo/ | head -20
repo/
├── avalon-framework
│   └── avalon-framework
│   ├── ivys
│   │   ├── ivy-4.1.5.xml
│   │   ├── ivy-4.1.5.xml.md5
│   │   └── ivy-4.1.5.xml.sha1
│   └── jars
│   ├── avalon-framework-4.1.5.jar
│   ├── avalon-framework-4.1.5.jar.md5
│   └── avalon-framework-4.1.5.jar.sha1
├── com.google.code.findbugs
│   └── jsr305
│   ├── ivys
│   │   ├── ivy-1.3.9.xml
│   │   ├── ivy-1.3.9.xml.md5
│   │   └── ivy-1.3.9.xml.sha1
│   └── jars
│   ├── jsr305-1.3.9.jar
│   ├── jsr305-1.3.9.jar.md5
Once this is in place the filesystem resolver works like a charm for both transitive (default) and compile-time (master) dependencies.

Error on integrating Spark with Scala project in IntelliJ IDE

I created a simple SBT project in IntelliJ IDE with the following library dependencies in build.sbt:
import _root_.sbt.Keys._
name := "untitled"
version := "1.0"
scalaVersion := "2.11.7"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.5.1",
"org.apache.spark" %% "spark-sql" % "1.5.1" ,
"org.apache.spark" %% "spark-mllib" % "1.5.1")
The objective is to import Spark and MLLIB of Spark, and then to create a Scala object as explained here.
However, the following error occurs on importing:
SBT project import
[warn] Multiple dependencies with the same organization/name but different versions. To avoid conflict, pick one version: [warn] *
org.scala-lang:scala-compiler:(2.11.0, 2.11.7) [warn] *
org.apache.commons:commons-lang3:(3.3.2, 3.0) [warn] *
jline:jline:(0.9.94, 2.12.1) [warn] *
org.scala-lang.modules:scala-parser-combinators_2.11:(1.0.1, 1.0.4)
[warn] * org.scala-lang.modules:scala-xml_2.11:(1.0.1, 1.0.4) [warn] *
org.slf4j:slf4j-api:(1.7.10, 1.7.2) [warn] [FAILED ]
net.sourceforge.f2j#arpack_combined_all;0.1!arpack_combined_all.jar(src):
(0ms) [warn] ==== local: tried [warn]
C:\Users\Cezar.ivy2\local\net.sourceforge.f2j\arpack_combined_all\0.1\srcs\arpack_combined_all-sources.jar
[warn] ==== public: tried [warn]
https://repo1.maven.org/maven2/net/sourceforge/f2j/arpack_combined_all/0.1/arpack_combined_all-0.1-sources.jar
[warn] [FAILED ] javax.xml.bind#jsr173_api;1.0!jsr173_api.jar(doc):
(0ms) [warn] ==== local: tried [warn]
C:\Users\Cezar.ivy2\local\javax.xml.bind\jsr173_api\1.0\docs\jsr173_api-javadoc.jar
[warn] ==== public: tried [warn]
https://repo1.maven.org/maven2/javax/xml/bind/jsr173_api/1.0/jsr173_api-1.0-javadoc.jar
[warn] [FAILED ] javax.xml.bind#jsr173_api;1.0!jsr173_api.jar(src):
(0ms) [warn] ==== local: tried [warn]
C:\Users\Cezar.ivy2\local\javax.xml.bind\jsr173_api\1.0\srcs\jsr173_api-sources.jar
[warn] ==== public: tried [warn]
https://repo1.maven.org/maven2/javax/xml/bind/jsr173_api/1.0/jsr173_api-1.0-sources.jar
[warn] :::::::::::::::::::::::::::::::::::::::::::::: [warn] :: FAILED
DOWNLOADS :: [warn] :: ^ see resolution messages for details ^ ::
[warn] :::::::::::::::::::::::::::::::::::::::::::::: [warn] ::
net.sourceforge.f2j#arpack_combined_all;0.1!arpack_combined_all.jar(src)
[warn] :: javax.xml.bind#jsr173_api;1.0!jsr173_api.jar(doc) [warn] ::
javax.xml.bind#jsr173_api;1.0!jsr173_api.jar(src) [warn]
::::::::::::::::::::::::::::::::::::::::::::::

Spark will not work with Scala 2.11. It uses Scala 2.10, so you need to use a compatible Scala version (see http://spark.apache.org/docs/latest/).
Alternatively, you can, as #eliasah has also mentioned in a comment, build Spark yourself. Instructions on how to build Spark can be found at http://spark.apache.org/docs/latest/building-spark.html

How do I specify net.sf.json-lib#json-lib#2.3 in Ivy or Maven?

I am in the middle of converting over our build system from Ant to Ant with Ivy, and have run into an issue specifying a particular jar we need.
groupId: net.sf.json-lib
artifactId: json-lib
version: 2.3
I specified it in Ivy as:
<dependency org="net.sf.json-lib" artifact="json-lib"
rev="2.3" conf="compile->default"/>
And got the following error:
[ivy:retrieve] ==== public: tried
[ivy:retrieve] http://buildl01.tcprod.local/artifactory/libs-release/net/sf/json-lib/json-lib/2.3/json-lib-2.3.jar
[ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:retrieve] :: FAILED DOWNLOADS ::
[ivy:retrieve] :: ^ see resolution messages for details ^ ::
[ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:retrieve] :: net.sf.json-lib#json-lib;2.3!json-lib.jar
[ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
Notice Ivy tried to download the file json-lib-2.3.jar from the repository.
I did a search on the Central Maven Repository and found that the artifact isn't called json-lib-2.3.jar but either json-lib-2.3-jdk-13.jar or json-lib-2.3-jdk-15.jar.
The problem is the way the name of the artifact is specified. The version number appears in the middle of the jar name. If the artifact was `json-lib-jdk-15-2.4.jar, I could do this:
<dependency org="net.sf.json-lib" artifact="json-lib"
rev="2.3" conf="compile->default">
<artifact name="json-lib-jdk-15"/>
</dependency>
How can I specify this jar for downloading?

The "thing" at the end of the jar name is called the classifier. It is used to target specific platforms or to provide sources or Javadoc.
To get the jar, you need to define the classifier like this:
<ivy-module version='2.0' xmlns:m="http://ant.apache.org/ivy/maven">
...
<dependency org="net.sf.json-lib" artifact="json-lib" rev="2.3"conf="compile->default">
<artifact name="json-lib" type="jar" m:classifier="jdk15"/>
</dependency>
Read this article for more information.

Using a custom repository with Apache Ivy, no resolver found

It seems that Apache ivy downloads artifacts only from http://mvnrepository.com/ and few other places, but all the jars are outdated there.
So I am trying to add custom repository for Ivy. I am using repository Ivy RoundUp : http://code.google.com/p/ivyroundup/
This is my configuration, but getting error:
build.xml:
<target name="update" depends="init-ivy" description="Download project dependencies">
<!-- edited for brevity -->
<ivy:settings file="ivysettings.xml" />
<ivy:retrieve pattern="war/WEB-INF/lib/[artifact]-[revision].[ext]" />
<!-- edited for brevity -->
</target>
ivy.xml :
<ivy-module version="2.0">
<info organisation="org.apache" module="hello-ivy"/>
<dependencies>
<dependency org="org.springframework" name="spring" rev="3.0.6" conf="default->master"/>
</dependencies>
</ivy-module>
ivysettings.xml:
<ivysettings>
<resolvers>
<packager name="roundup" buildRoot="${user.home}/.ivy2/packager/build" resourceCache="${user.home}/.ivy2/packager/cache">
<ivy pattern="http://ivyroundup.googlecode.com/svn/trunk/repo/modules/[organisation]/[module]/[revision]/ivy.xml"/>
<artifact pattern="http://ivyroundup.googlecode.com/svn/trunk/repo/modules/[organisation]/[module]/[revision]/packager.xml"/>
</packager>
</resolvers>
</ivysettings>
error:
Buildfile: C:\Users\Jansu\workspace\HibernateSpring\build.xml
build:
deploywar:
[war] Building war: C:\Users\Jansu\workspace\HibernateSpring\hibernate.war
[copy] Copying 1 file to C:\apache-tomcat-7.0.20\webapps
download-ivy:
[get] Getting: http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.2.0/ivy-2.2.0.jar
[get] To: C:\Users\Jansu\.ant\lib\ivy-2.2.0.jar
[get] Not modified - so not downloaded
init-ivy:
update:
[ivy:retrieve] :: Ivy 2.2.0 - 20100923230623 :: http://ant.apache.org/ivy/ ::
[ivy:retrieve] :: loading settings :: file = C:\Users\Jansu\workspace\HibernateSpring\ivysettings.xml
[ivy:retrieve] :: resolving dependencies :: org.apache#hello-ivy;working#Jansu-PC
[ivy:retrieve] confs: [default]
[ivy:retrieve] :: resolution report :: resolve 110ms :: artifacts dl 0ms
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 1 | 0 | 0 | 0 || 0 | 0 |
---------------------------------------------------------------------
[ivy:retrieve] :: problems summary ::
[ivy:retrieve] :::: WARNINGS
[ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:retrieve] :: UNRESOLVED DEPENDENCIES ::
[ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:retrieve] :: org.springframework#spring;3.0.6: no resolver found for org.springframework#spring: check your configuration
[ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:retrieve]
[ivy:retrieve] :::: ERRORS
[ivy:retrieve] unknown resolver null
[ivy:retrieve] no resolver found for org.springframework#spring: check your configuration
[ivy:retrieve]
[ivy:retrieve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
BUILD FAILED
C:\Users\Jansu\workspace\HibernateSpring\build.xml:177: impossible to resolve dependencies:
resolve failed - see output for details
Total time: 2 seconds
So it seems that it does not find my resolver . I did configure the resolver like the custom repository asked me to: http://code.google.com/p/ivyroundup/wiki/HowToConfigureIvy
And here is list of all artifacts in that repo, so you can see that Spring 3.0.6 exists in there : http://ivyroundup.googlecode.com/svn/trunk/repo/modules.xml
Any suggestions? Feel free to ask more information.
EDIT:
build.properties :
ivy.install.version=2.2.0
ivy.home=${user.home}/.ant
ivy.jar.dir=${ivy.home}/lib
ivy.jar.file=${ivy.jar.dir}/ivy-${ivy.install.version}.jar

Ivyroundup is designed around the packager resolver in ivy. This resolver is incredibly clever, demonstrates the true power of ivy, but the bulk of the world uses Maven repositories to host their software. Fact is soon Maven Central will contain nearly 90% of the world's Java open source components.
Enabling Maven repositories
Thankfully, ivy fully understands Maven repositories, meaning we can use ivy as a client and let very good products like Nexus host the repository. Here's the settings file that enables Maven Central:
<ivysettings>
<settings defaultResolver='central'/>
<resolvers>
<ibiblio name='central' m2compatible='true'/>
</resolvers>
</ivysettings>
I would highly recommend you consider setting up you own local instance of Nexus (Or Artifactory, or Archiva...). You can then cache Maven central artifacts (more efficient), search for software components and upload and host artifacts which cannot be downloaded, due to license restrictions (JDBC jars).
Enabling a local repository manager also uses the ibiblio resolver as follows:
<ivysettings>
<settings defaultResolver='nexus'/>
<resolvers>
<ibiblio name='nexus' m2compatible='true' root='https://nexus.mydomain.com:8081/nexus/content/groups/central/' />
</resolvers>
</ivysettings>
Searching Maven Central (New ivy support features)
You're looking for the Spring 3.0.6 release? It's already in Maven Central:
http://search.maven.org/#search|ga|1|g%3A%22org.springframework%22%20AND%20v%3A%223.0.6.RELEASE%22
The Spring core artifact details are here:
http://search.maven.org/#artifactdetails|org.springframework|spring-core|3.0.6.RELEASE|jar
The search page now conveniently gives you both the Maven and ivy client declaration to copy into your build:
<dependency org="org.springframework" name="spring-core" rev="3.0.6.RELEASE" >
<artifact name="spring-core" type="jar" />
</dependency>

for sake of completeness ...
ivysettings.xml with a chain resolver for
https://maven.org
https://jitpack.io/
<ivysettings>
<settings defaultResolver="default-chain"/>
<resolvers>
<chain name="default-chain">
<!-- default resolver
<ibiblio name="maven.org" m2compatible="true" root="https://repo1.maven.org/maven2"/>
-->
<ibiblio name="maven.org" m2compatible="true"/>
<!-- resolver for github
example use in ivy.xml: https://github.com/omajid/xdg-java
<dependency org="com.github.omajid" name="xdg-java" rev="7c623b86e6b99bef0a72db86318663ad3c0270e5" conf="compile->master"/>
-->
<ibiblio name="jitpack.io" m2compatible="true" root="https://jitpack.io/"/>
</chain>
</resolvers>
</ivysettings>

Your update target should invoke ivy:resolve task after ivy:settings.
<target name="update" depends="init-ivy" description="Download project dependencies">
<ivy:settings file="ivysettings.xml" />
<ivy:resolve conf="default" />
<ivy:retrieve pattern="war/WEB-INF/lib/[artifact]-[revision].[ext]" />
</target>

you must have something to tell whats the default resolver, like,
<conf defaultResolver="default" />