I'm starting a new project in Scala that heavily depends on source files in another Java project. At first I thought about creating a new package/module in that project, but that seemed messy. However, I don't want to duplicate the Java code in my new project. The best solution I came up with is to reference the Java project through an svn external dependancy. Another solution is creating a jar file from the original, I would attach it as a library to the new Scala project. However, this makes keeping up-to-date more inconvenient. Which route is better?
SVN external:
Easy to update
Easy to commit
Inconvenient setup
Jar file as library:
Easy to setup
Old code isn't in active development(stable, bug fixes only)
Multi-step update
Need to open old project to make changes
You have your Scala project, and it depends on parts of your Java project. To me, that sounds like the Java project should be a library, with a version number, which is simply referenced by your Scala project. Or perhaps those parts that are shared across the two projects should be separated into a library. Using build tools like Maven will keep it clear which version is being used. The Java project can then evolve separately, or if it needs to change for the sake of the Scala project, you can bring out a new version and keep using an older one in other contexts if you're afraid of breakage.
The only exception where you go beyond binary dependencies that I can think of is if the Java code itself is actually being processed in some way at compile-time that is specific to the Scala project. Like, say, annotation processing.
Using SVN externals could be a solution. Just make sure you work with branches and snapshots to make sure some update to your Java code on the trunk doesn't suddenly make your Scala project inoperable.
Whether you have mixed scala/java or not is irrelevent.
You probably want to use external dependencies (jars) if both projects are very distinct and have different release cycles.
Otherwise, you can use sbt and have your two projects be sub-projects of the same build (your sbt build file would have the scala project depend on the java project). Or even, if they are really so intertwined, just have one project with both java and scala source files. Sbt handles that just fine.
Maven is an option too.
You can very easily set up a mixed scala/Java project using maven. Have a look at the scala-maven-plugin.
https://github.com/davidB/scala-maven-plugin
Eclipse users may not be too pleased due the poor maven integration.
Related
I have multiple app projects of of roughly this layout:
example app (Java)
Java Wrapper with additional functionality
C++ + Shallow Java Wrapper
2nd example app (flutter)
flutter wrapper
Java Wrapper with additional functionality
C++ + Shallow Java Wrapper
3rd example app
flutter wrapper
Java Wrapper with additional functionality
C++ + Shallow Java Wrapper
All apps share the same main dependency (java Wrapper with additional functionality) and its dependency tree. Now I am developing on each app all the way down to C++ code. They are managed as git submodules in their respective parent project.
As there is a high change rate along the whole process, I want the final example to be built for testing from all sources.
I tried several approaches for tying this together into one gradle build:
1. Preferred (but failing) solution: settings.gradle in each project, each project only includes direct dependencies
Now I want this full tree to be managed in one flutter build. So I add the direct dependencies in each projects settings.gradle, just to learn that gradle only supports one toplevel settings.gradle. So this does not work. The presented solutions in aforementioned question mostly try to emulate support for multiple settings.gradle files.
2. Functioning but Ugly: Add all dependency projects are included in the toplevel settings.gradle
Do I really have to include all subprojects manually in the toplevel settings.gradle, when each of the subprojects knows its dependencies perfectly fine? Furthermore, since there are multiple projects depending on this, do I have to do this manually for each of them?
(And don't even get me startet about gradle not telling me, I have a wrong projectDir because I got a typo in the 100rth level of recursive descend!)
3. Probably Working Solution: Use composite builds
This will trigger the builds but now I have to resolve the build artifacts instead of the projects. So same problem with other artifacts.
4. Probably Working solution: Publish dependency projects to a maven (or other) repository and pull that into the app
I did not try this because I find the idea abhorent: I want to test one small change in the C++ code and now have to push that to a repository and potentially do the same on every project above?
This works for a stable project but not for flexible exploratory development. Sure, I want to publish something at the end but I don't want to publish every little step in between.
This left me wondering: Am I doing something unusual? I mean: is there nobody who has the same requirements that gradle does not seem able to fit:
live updates from all the way down to quick test local changes
no repeating of transitive dependencies on the toplevel
What is the common practice in this case?
After Lukas Körfer's comment I took a closer look at composite builds again and noticed that I had a misconception about them. I did not understand that their dependency resolution will solve the finding of the build artifacts for me.
Now I use the composite builds to tie together the whole build while using
implementation 'my.group:project'
to import the code of the subprojects and
includeBuild '../path/to/subproject/'
to pull them in.
I am new to using github and have been trying to figure out this question by looking at other people's repositories, but I cannot figure it out. When people fork/clone repositories in github to their local computers to develop on the project, is it expected that the cloned project is complete (ie. it has all of the files that it needs to run properly). For example, if I were to use a third-party library in the form of a .jar file, should I include that .jar file in the repository so that my code is ready to run when someone clones it, or is it better to just make a note that you are using such-and-such third-party libraries and the user will need to download those libraries elsewhere before they begin work. I am just trying to figure at the best practices for my code commits.
Thanks!
Basically it is as Chris said.
You should use a build system that has a package manager. This way you specify which dependencies you need and it downloads them automatically. Personally I have worked with maven and ant. So, here is my experience:
Apache Maven:
First word about maven, it is not a package manager. It is a build system. It just includes a package manager, because for java folks downloading the dependencies is part of the build process.
Maven comes with a nice set of defaults. This means you just use the archtype plugin to create a project ("mvn archetype:create" on the cli). Think of an archetype as a template for your project. You can choose what ever archetype suits your needs best. In case you use some framework, there is probably an archetype for it. Otherwise the simple-project archetype will be your choice. Afterwards your code goes to src/main/java, your test cases go to src/test/java and "mvn install" will build everything. Dependencies can be added to the pom in maven's dependency format. http://search.maven.org/ is the place to look for dependencies. If you find it there, you can simply copy the xml snippet to your pom.xml (which has been created by maven's archetype system for you).
In my experience, maven is the fastest way to get a project with dependencies and test execution set up. Also I never experienced that a maven build which worked on my machine failed somewhere else (except for computers which had year-old java versions). The charm is that maven's default lifecycle (or build cycle) covers all your needs. Also there are a lot of plugins for almost everything. However, you have a big problem if you want to do something that is not covered by maven's lifecycle. However, I only ever encountered that in mixed-language projects. As soon as you need anything but java, you're screwed.
Apache Ivy:
I've only ever used it together with Apache Ant. However, Ivy is a package manager, ant provides a build system. Ivy is integrated into ant as a plugin. While maven usually works out of the box, Ant requires you to write your build file manually. This allows for greater flexibility than maven, but comes with the prize of yet another file to write and maintain. Basically Ant files are as complicated as any source code, which means you should comment and document them. Otherwise you will not be able to maintain your build process later on.
Ivy itself is as easy as maven's dependency system. You have an xml file which defines your dependencies. As for maven, you can find the appropriate xml snippets on maven central http://search.maven.org/.
As a summary, I recommend Maven in case you have a simple Java Project. Ant is for cases where you need to do something special in your build.
In my IDEA project a Scala module depends on a Java module. When I try to compile the Scala module, only scalac is triggered. It compiles both Java and Scala sources.
I'd like scalac to compile only the Scala module, because javac is much faster for Java sources (and my Java project is a big one).
How to make IDEA use different compiler for different modules?
My workaround is to (for each dependency to Java module):
Delete module dependency in project configuration
Add dependency to appropriate compile output directory "MyJavaModule/target/classes"
Obviously I'm not happy with that, because every time I reimport Maven project I need to repeat all of this to have fast compilation. I hope somebody knows a better way.
Clarification: I'd like to stress, that tools like SBT or Maven don't solve my problem. It is not about compilation alone. It's about compilation in IDEA, required for things like Scala Worksheet or running unit tests from IDEA. My goal is to have full range of IDEA niceties (syntax highlighting, intelligent auto-completion, auto-imports, etc) with compilation speed of SBT. Now I have to either tolerate long compilation times (due to dependencies to my Java module) or to use bare-bones REPL and testing in SBT.
Randall Schulz has asked the right question in the comment: "Why does it matter which tool does the compilation?"
Up until now I believed that IDEA needs to compile all classes itself if you want to use its nice features (like IDEA's Scala Console or running tests from within it). I was wrong.
In fact, IDEA will pick up classes compiled by any other tool (like the great SBT for instance). You just need to assure that all classes are up-to-date before using any of IDEA's helpful features. The best way to do it is:
launch continuous incremental compilation in the background (for
example by issuing "~ compile" in SBT)
remove "make" step in IDEA's
run configurations
That's all! You can then use all cool features of IDEA (not only syntax highlighting and code completion, but all auto-imports in Scala Console, quickly running selected unit tests) without switching between different windows.
That's the workflow I missed until now! Thanks to everybody for all the comments about the issue.
You should look at using a dependency management suite like Apache Ivy or Apache Maven. Then put your Java source in a separate artifact, and have your Scala project be dependent on the Java project artifact.
If you go the Maven route, there is a Scala plugin.
Probably the simplest way to get compiled Scala and Java files is SBT - Simple Build Tool. Just create a project (+ add dependencies and so on) and compile it. Scala + Java compilation works out of the box. I've switched to SBT from Maven.
If you have a complex POM or if you have another reason not to migrate to SBT, you can try to configure the POM. Just adding (and possibly configuring) the Scala plugin should be enough. I hope it will not break the Java support.
I'm new to Maven, using the m2e plugin for Eclipse. I'm still wrapping my head around Maven, but it seems like whenever I need to import a new library, like java.util.List, now I have to manually go through the hassle of finding the right repository for the jar and adding it to the dependencies in the POM. This seems like a major hassle, especially since some jars can't be found in public repositories, so they have to be uploaded into the local repository.
Am I missing something about Maven in Eclipse? Is there a way to automatically update the POM when Eclipse automatically imports a new library?
I'm trying to understand how using Maven saves time/effort...
You picked a bad example. Portions of the actual Java Library that come with the Java Standard Runtime are there regardless of Maven configuration.
With that in mind, if you wanted to add something external, say Log4j, then you would need to add a project dependency on Log4j. Maven would then take the dependency information and create a "signature" to search for, first in the local cache, and then in the external repositories.
Such a signature might look like
groupId:artifactId:version
or perhaps
groupId:artifactId:version:classifier
This identifies a maven "module" which will then be downloaded and configured into your system. Once in place it adds all of the classes within the module to your configured project.
Maven principally saves time in downloading and organizing JAR files in your build. By defining a "standard" project layout and a "standard" build order, Maven eliminates a lot of the guesswork in the "why isn't my project building" sweepstakes. Also, you can use neat commands like "mvn dependency:tree" to print out a list of all the JARs your project depends on, recursively.
Warning note: If you are using the M2E plugin and Eclipse, you may also run into problems with the plugin itself. The 1.0 version (hosted at eclipse.org) was much less friendly than the previous 0.12 version (hosted at Sonatype). You can get around this to some extent by downloading and installing the "standalone" version of Maven from apache (maven.apache.org) and running Maven from the command line. This is actually much more stable than trying to run Maven inside Eclipse (in my personal experience) and may save you some pain as you try to learn about Maven.
I am working in a small team (3 persons) on several modules (about 10 currently). The compilation, integration and management of build versions is becoming more and more tedious.
I am looking for a good build / integration tool to replace / complete Ant.
Here is the description of our current development environment :
- Several modules depending on each over and on third party JARs
- Some may export JARS, some export WARS, some export standalone, runnable JARS (with Fat-Jar)
- Javadoc for all of them
- We work with eclipse
- Custom Ant script for each module. Many redundant information between the eclipse configuration and Ant scripts. For example, for the standalone Fat-JAR, we have listed all the recursive dependencies, whereas ideally, it could clearly be imported from the eclipse configuration.
- The source code is versioned using SVN
Here is what I would like a perfect integration tool to do for me :
Automatize the releases and versioning of modules. Ideally, the integration tool should detect if a new version is needed. For example, if I want to release a project A that depends on a project B, and if I have made small changes on the project B locally, then the integration tool should first release a new version of B as well and make A based on it.
Integrate strongly with eclipse, so that it could get the dependencies between modules and third party libs from its configuration. BTW, I would like to continue to configure build path with eclipse without updating some other ".xml" stuff. I saw that Gradle can generate eclipse project files from its configuration, but the counterpart would be great.
Enable a "live" and transparent development on local projects. I mean that I often make small changes on the core / common projects while developing the main / "leaf" projects. I would like to have my changes on core projects immediately available to leaf projects without the need of publishing (even locally) the JARs of my core projects.
Store all versions of the releases of my module on an external server. The simplest (shares folder / Webdav) would be the best. A nice web page with list of modules and delivered artifacts would be great too.
I have looked around for many things. From Ant4eclipse (to integrate the Eclipse configuration into my Ant script), to the Maven / Ivy / Gradle tools.
I am a bit confused.
Here is what I have understood so far:
- Maven is a great / big tool, but is somewhat rigid and obliges you to bend to its structure and concepts. It is based on description rather than on scripting. If you go out of the path, you have to develop you own plugins.
- Ivy is less powerful than maven, it handles less stuff but is more flexible.
- Gradle is in-between. It is general purpose. It enables scripting as well as "convention based" configuration. It integrates Ant and extends it.
So at this point I am looking for actual testimonials from real users.
What tools do you use ? How ? Do you have the same needs as me ?
Does it ease your life or get into the way ?
Are there sample some use cases, or workspace skeletons out there that I could use as a starting point to see what these tools are capable of ?
Sorry for the length of this message.
And thanks in advance for you advice.
Kind regards,
Raphael
Automatize the releases and versioning of modules (...)
The concepts of versioning and repository are built-in with Maven and they could fit here.
Maven supports SNAPSHOT dependencies. When using a snapshot, Maven will periodically try to download the latest available snapshot from a repository when you run a build. SNAPSHOT are typically used when a project is under active development.
Maven 2 also supports version ranges (I do not really recommend them but that's another story) which allow for example to configure A to depend on version [4.0,) of B (any version greater than or equal to 4.0). If you build and release a new version of B, A would use it.
Integrate strongly with eclipse
The m2eclipse plugin provides bi-directional synchronization with Eclipse.
Enable a "live" and transparent development on local projects.
The m2eclipse plugin supports "workspace resolution": if project A depend on project B and if project B is in the workspace, you can configure A to depend on B sources and not on B.jar (that's the default mode if I'm not wrong). So a change on B sources would be directly visible, without the need to build B.jar.
Store all versions of the releases of my module on an external server.
As mentioned earlier, this is actually a central concept of Maven (you don't even have the choice) and deploying through file:// or dav:// are both supported.
To sum up, Maven is (probably) not the only candidate but I'm sure it would fit:
Your project isn't that exotic or complex, there is nothing scaring from your description (some refactoring of the structure will probably be required but this shouldn't be a big deal).
Maven also brings a workflow based on best practices.
m2eclipse provides strong integration with the IDE.
But Maven has some learning curve.
CI tools? To me, there's only one: the Hudson CI.
I've setup a software development environment for Java once, with the components:
eclipse IDE
mercurial
bugzilla
maven
Nexus
Hudson CI
and some apache, mysql, php, perl, python, .. for integration.
The hudson was not integrated with eclipse and that was on purpose, because I wanted to build on a separate server. for all the other tools I had a perfect cross integration (like: mylyn on eclipse to talk with bugzilla, m2eclipse for using maven eclipse, a lot of plugins for hudson, ...)
We've been starting to integrate Gradle into our build process, and I can add to the answers posted already that Gradle would also work. Your assumptions are mostly correct, gradle is more off the cuff, but is powerful and allows for scripting and such within the build itself. It seems that most things maven can do, gradle does as well.
Now for your individual points:
Versioning: gradle supports dependency maps, versioning, and if you add in a CI server, you can trigger automated/dependent builds. For example, almost all of our 'deliverables' are .wars, but we have several code libs (.jars) and one executable .jar in development. One configuration is to make the the wars and the "fat-jar" dependent on the shared code libs. Then, when the shared libs are updated, bump the versions on the shared libs, test the consuming projects, then use Hudson's ability to fire dependent projects to redeploy those. There are other ways, but that seems to work best for us, for now.
Integrate strongly with eclipse: You're right, gradle can generate the eclipse files. We tend to only use the eclipseCp (to update .classpath) task once we get going, as only classpath needs changed. It's kind of quirky (grabs your default JRE, so make sure it's right, doesn't add exported="true" if you need it), but gets you 99% of the way there.
Enable a "live" and transparent development on local projects: This is one I'm not sure about. I've only hacked around gradle in this case; by removing the artifact in the consuming project and marked the shared project as such in eclipse, then reverted afterwards.
Store all versions of the releases of my module on an external server: simple and many approaches are supported, similar to Maven.
As far as examples, the docs for gradle are good, as well as the example projects that come with the full zip. They'll get you up and running fairly quickly.
Have a look at Ant Ivy. http://ant.apache.org/ivy/
There are no silver bullets, but in my experience Maven is a great project management tool. Personally, I like to use a comibnation of subversion (for version control), maven (for project/build management) and hudson (for continuous build/integration).
I find the convention brought by maven is really useful for context switching, and great for dependency management. It can be frustrating if jars aren't in the repositories, but you can install them locally and when you're ready you can host your own private repository which mirrors other places. I have had a good experience using sonar.nexus by http://www.sonatype.com/ . They also provide an excellenmt free book to get you started.
It might seem like overkill now, but setting up a good build / test / integrate / release environment now, will pay dividends later. It's is always harder to retro-fit, and it's something you can replicate easily.
Lastly, I happen to prefer Netbeans integration for maven, but that's just me :)
Some of your topics are part of deployment and release management.
You could check out a product like: Xebia DeployIt
(with an personal edition which is free)