We Keep Coding

How to merge source sets while sharing dependencies to each other

I'd like to publish a library with two different API versions where both use the same core code underneath. I tried shading/shadowing but have struggles getting the visibility right (I'd like to hide the core code from the API user). So I wanted to achieve my goals by having different source sets and configurations:
sourceSets {
// the `main` source set acts as the common code base for `api` and `api2`
api {
java {
srcDir 'src/api/java'
// Includes classes from `main`:
compileClasspath += sourceSets.main.output
runtimeClasspath += sourceSets.main.output
}
}
api2 {
java {
srcDir 'src/api2/java'
// Includes classes from `main`:
compileClasspath += sourceSets.main.output
runtimeClasspath += sourceSets.main.output
}
}
}
configurations {
common {
canBeResolved = true
canBeConsumed = false
}
// These art the configurations used both for being consumed with `project(...)` or published:
exposedApi {
canBeResolved = true
canBeConsumed = true
extendsFrom common
}
exposedApi2 {
canBeResolved = true
canBeConsumed = true
extendsFrom common
}
}
task apiJar(type: Jar) {
group = 'build'
from configurations.exposedApi
baseName = 'api'
}
task api2Jar(type: Jar) {
group = 'build'
from configurations.exposedApi2
baseName = 'api2'
}
publishing {
publications {
api(MavenPublication) {
artifact apiJar
artifactId 'mylib-api'
}
api2(MavenPublication) {
artifact api2Jar
artifactId 'mylib-api2'
}
}
}
dependencies {
common sourceSets.main.output
exposedApi sourceSets.api.output
exposedApi2 sourceSets.api2.output
}
If I want to use one of these APIs I can easily use project(path: ':mylib', configuration: 'exposedApi2') or use one of the published Maven artifacts and it works nicely.
But as soon as I change classes in the main source set to internal in order to achieve proper encapsulation of the main code, the API code won't compile anymore:
Cannot access 'SomeClassInMain': it is internal in '' (<-- yes, it really shows nothing in the '')
I also tried to merge the source set into one, so there is technically not really a main source set anymore:
sourceSets {
api {
java {
srcDirs('src/api/java', 'src/main/java')
}
}
api2 {
java {
srcDirs('src/api2/java', 'src/main/java')
}
}
}
That now works all as intended, no compilation errors, calls from the API to main work as expected and the classes in main even have internal visibility. But unfortunately IntelliJ seems to not pick up the fact that classes in main are really part of the same source set. I get an error (Unresolved reference: SomeClassInMain) in the IDE every time I mention a class from the main sources and of course no auto-completion would work, too, making the solution somehow not really practical in the end.
So just to sum up the goal:
it's important that the main sources are accessible to the API
but not to the user using the API (or the Maven publication) – the only thing the user should be facing is the API
If possible, I'd like to not put the API and main code in separate modules and publish them separately for encapsulation reasons
I tried a shading/shadowing (fat/uber JAR) approach but I haven't managed to reduce the visibility to internal in the main sources
I'm new to the topic of these complicated kinds of build configurations so maybe I simply have chosen the wrong approach. Maybe there's a better one which I haven't yet managed to find?
Many, many thanks in advance!

gradle javaexec error "'apiElements' directly is not allowed"- Gradle 5.4.1

I am new to Gradle and trying to migrate an existing system build from ant to Gradle.
As part of this I need to run a java program on every file in a directory. Directory contains xml files and the java code will parse and convert .xml to .java files (and these Java files would be build to generate class and package in final jar) after performing some business specific transformation.
below is a function I wrote in Gradle
private runJavaFile(String dirPath) {
FileTree tree = fileTree(dir: dirPath, include: '**/*.xml')
tree.each {
def xmlfile = it.path
def javaFile = it.path.replaceFirst(".xml", ".java")
javaexec { //// getting error on this line
classpath configurations.all
main = 'XmlToJavaParser'
args = ["$xmlfile", "$javaFile", 'Java']
}
}
}
I am calling this function from a Gradle task by passing the dir path which contains the xml files to be parsed.
While running the task, I am getting below error:
> Resolving configuration 'apiElements' directly is not allowed
Any help would be appreciated.
Let me know if any more information is needed.

In Gradle, a configuration represents a group of artifacts and their dependencies. You typically have several configurations depending on what you want to do. For instance, you could have one where you declare which dependencies are needed for compilation, which are only needed at runtime, or which are needed for running a particular Java application.
In your case, you are saying that the classpath to the XmlToJavaParser class is "all configurations combined" and that doesn't really make sense. You are also not allowed to do that as some configurations from the Java plugin are not resolvable like this, which is why you get an error.
So to fix it, you should declare your own configuration for XmlToJavaParser. You can then declare dependencies for it like you normally do. Example (using the Groovy DSL):
configurations {
xmlJavaParser {
canBeResolved = true
canBeConsumed = false
}
}
dependencies {
xmlJavaParser "org.example:xml-java-parser:1.0" // or whatever you need
}
private runJavaFile(String dirPath) {
// ...
javaexec {
classpath = configurations.xmlJavaParser // The configuration is referenced here
main = 'XmlToJavaParser'
args = ["$xmlfile", "$javaFile", 'Java']
}
}
There are also other ways to go about it. But the main point is to not use configurations.all as a classpath.

How do I create an executable fat JAR with Gradle with implementation dependencies?

I've got a simple project in Gradle 4.6 and would like to make an executable JAR of it. I've tried shadow, gradle-fatjar-plugin, gradle-one-jar, spring-boot-gradle-plugin plugins but neither of them adds my dependencies declared as implementation (I don't have any compile ones). It works with compile e.g. for gradle-one-jar plugin but I would like to have implementation dependencies.

You can use the following code.
jar {
manifest {
attributes(
'Main-Class': 'com.package.YourClass'
)
}
from {
configurations.runtimeClasspath.collect { it.isDirectory() ? it : zipTree(it) }
}
}
Be sure to replace com.package.YourClass with the fully qualified class name containing static void main( String args[] ).
This will pack the runtime dependencies. Check the docs if you need more info.

Based on the accepted answer, I needed to add one more line of code:
task fatJar(type: Jar) {
manifest {
attributes 'Main-Class': 'com.yourpackage.Main'
}
archiveClassifier = "all"
from {
configurations.compile.collect { it.isDirectory() ? it : zipTree(it) }
configurations.runtimeClasspath.collect { it.isDirectory() ? it : zipTree(it) }
}
with jar
}
Without this additional line, it omitted my source files and only added the dependencies:
configurations.compile.collect { it.isDirectory() ? it : zipTree(it) }
For newer gradle (7+), you may see this error:
Execution failed for task ':fatJar'.
> Entry [some entry here] is a duplicate but no duplicate handling strategy has been set. Please
refer to https://docs.gradle.org/7.1/dsl/org.gradle.api.tasks.Copy.html#org.gradle.api.tasks.Copy:duplicatesStrategy
for details.
If this happens add a duplicatesStrategy such as duplicatesStrategy "exclude" to the fatJar task.
And likewise, for Gradle 7+, you have to just remove the configuration.compile.collect line because it is no longer a valid configuration in this version of gradle.

The same task can be achieved using Gradle Kotlin DSL in a similar way:
val jar by tasks.getting(Jar::class) {
manifest {
attributes["Main-Class"] = "com.package.YourClass"
}
from(configurations
.runtime
// .get() // uncomment this on Gradle 6+
// .files
.map { if (it.isDirectory) it else zipTree(it) })
}

previous answers are a little outdated nowadays, see here for something working with gradle-7.4:
How to create a fat JAR with Gradle Kotlin script?
tasks.jar {
manifest.attributes["Main-Class"] = "com.example.MyMainClass"
val dependencies = configurations
.runtimeClasspath
.get()
.map(::zipTree) // OR .map { zipTree(it) }
from(dependencies)
duplicatesStrategy = DuplicatesStrategy.EXCLUDE
}

Here I provide solutions for Kotlin DSL (build.gradle.kts).
Note that the first 3 methods modify the existing Jar task of Gradle.
Method 1: Placing library files beside the result JAR
This method does not need application or any other plugins.
tasks.jar {
manifest.attributes["Main-Class"] = "com.example.MyMainClass"
manifest.attributes["Class-Path"] = configurations
.runtimeClasspath
.get()
.joinToString(separator = " ") { file ->
"libs/${file.name}"
}
}
Note that Java requires us to use relative URLs for the Class-Path attribute. So, we cannot use the absolute path of Gradle dependencies (which is also prone to being changed and not available on other systems). If you want to use absolute paths, maybe this workaround will work.
Create the JAR with the following command:
./gradlew jar
The result JAR will be created in build/libs/ directory by default.
After creating your JAR, copy your library JARs in libs/ sub-directory of where you put your result JAR. Make sure your library JAR files do not contain space in their file name (their file name should match the one specified by ${file.name} variable above in the task).
Method 2: Embedding the libraries in the result JAR (fat or uber JAR)
This method too does not need any Gradle plugin.
tasks.jar {
manifest.attributes["Main-Class"] = "com.example.MyMainClass"
val dependencies = configurations
.runtimeClasspath
.get()
.map(::zipTree) // OR .map { zipTree(it) }
from(dependencies)
duplicatesStrategy = DuplicatesStrategy.EXCLUDE
}
Creating the JAR is exactly the same as the previous method.
Method 3: Using the Shadow plugin (to create a fat or uber JAR)
plugins {
id("com.github.johnrengelman.shadow") version "6.0.0"
}
// Shadow task depends on Jar task, so these configs are reflected for Shadow as well
tasks.jar {
manifest.attributes["Main-Class"] = "org.example.MainKt"
}
Create the JAR with this command:
./gradlew shadowJar
See Shadow documentations for more information about configuring the plugin.
Method 4: Creating a new task (instead of modifying the Jar task)
tasks.create("MyFatJar", Jar::class) {
group = "my tasks" // OR, for example, "build"
description = "Creates a self-contained fat JAR of the application that can be run."
manifest.attributes["Main-Class"] = "com.example.MyMainClass"
duplicatesStrategy = DuplicatesStrategy.EXCLUDE
val dependencies = configurations
.runtimeClasspath
.get()
.map(::zipTree)
from(dependencies)
with(tasks.jar.get())
}
Running the created JAR
java -jar my-artifact.jar
The above solutions were tested with:
Java 17
Gradle 7.1 (which uses Kotlin 1.4.31 for .kts build scripts)
See the official Gradle documentation for creating uber (fat) JARs.
For more information about manifests, see Oracle Java Documentation: Working with Manifest files.
For difference between tasks.create() and tasks.register() see this post.
Note that your resource files will be included in the JAR file automatically (assuming they were placed in /src/main/resources/ directory or any custom directory set as resources root in the build file). To access a resource file in your application, use this code (note the / at the start of names):
Kotlin
val vegetables = MyClass::class.java.getResource("/vegetables.txt").readText()
// Alternative ways:
// val vegetables = object{}.javaClass.getResource("/vegetables.txt").readText()
// val vegetables = MyClass::class.java.getResourceAsStream("/vegetables.txt").reader().readText()
// val vegetables = object{}.javaClass.getResourceAsStream("/vegetables.txt").reader().readText()
Java
var stream = MyClass.class.getResource("/vegetables.txt").openStream();
// OR var stream = MyClass.class.getResourceAsStream("/vegetables.txt");
var reader = new BufferedReader(new InputStreamReader(stream));
var vegetables = reader.lines().collect(Collectors.joining("\n"));

from { configurations.runtimeClasspath.collect { it.isDirectory() ? it : zipTree(it) } }
This line is essential to me.

Kotlin 1.3.72 & JVM plugin, Gradle 6.5.1
Syntax is changing quickly in all these platforms
tasks {
compileKotlin {
kotlinOptions.jvmTarget = "1.8"
}
compileTestKotlin {
kotlinOptions.jvmTarget = "1.8"
}
val main = sourceSets.main.get()
//TODO
register<Jar>("buildFatJar") {
group = "app-backend"
dependsOn(build)
// shouldRunAfter(parent!!.tasks["prepCopyJsBundleToKtor"]) -> This is for incorporating KotlinJS gradle subproject resulting js file.
manifest {
attributes["Main-Class"] = "com.app.app.BackendAppKt"
}
from(configurations.compileClasspath.get().files.map { if (it.isDirectory) it else zipTree(it) })
with(jar.get() as CopySpec)
archiveBaseName.set("${project.name}-fat")
}
}

mainClassName = 'Main'
sourceSets {
main {
java {
srcDirs 'src/main/java', 'src/main/resources'
}
}
}
jar{
manifest {
attributes(
"Main-Class": "$mainClassName",
)
}
from {
configurations.runtimeClasspath.collect { it.isDirectory() ? it : zipTree(it)
}
}
exclude 'META-INF/*.RSA', 'META-INF/*.SF','META-INF/*.DSA'
duplicatesStrategy = DuplicatesStrategy.EXCLUDE
dependsOn ('dependencies')
}

How to make Gradle fail the build if a file dependency is not found?

I have a Gradle build that has some dependencies of the form
compile files('path/to/local/lib.jar')
(the build is being migrated - eventually these will be replaced)
The build failed because one of these paths was incorrectly specified. But it failed due to a compile error - it looked like Gradle silently ignored the missing dependency.
Is there a simple option or switch that will force Gradle to fail the build if any dependency (particularly local file dependencies) cannot be resolved (eg., file missing)?
Edit: to clarify further:
If a dependency cannot be found in the configured repositories, Gradle will fail the build when attempting to resolve them, as expected.
BUT - if a dependency is defined as "compile files ....", and the file specified does not exist at build time, Gradle will IGNORE that error, and attempt compilation anyway. That seems spectacularly wrong-headed and inconsistent default behaviour.
My question is - is there a Gradle option or switch or environment variable or system property that I can set to force Gradle to verify that file dependencies exist? (E.g,, behave in a sane and rational way?)

This is a bit of an old thread, but given that none of the currently proposed solutions actually works, and the solution appears to be trivial (collating two of them), I am leaving it here for future reference.
The point here is that we simply want to ensure that the files do exist, so we can just use the exists() method of the File class:
task ensureDepsExist() {
doLast {
configurations.implementation.canBeResolved(true)
Set<File> impFiles = configurations.implementation.resolve()
impFiles.forEach { f ->
if (!f.exists()) {
ant.fail "${f} could not be found"
}
}
}
}
compileJava.dependsOn ensureDepsExist
The canBeResolved() call is required, or Gradle will complain that configurations dependencies cannot be resolved.

Here's how you can check transitive dependencies using Gradle 7.3 (example: Fail if the project depends on log4j directly or transitively).
Kotlin DSL
configurations {
all {
relsolutionStrategy {
eachDependency {
if (requested.name == "log4j") {
throw RuntimeException("Project depends on log4j")
}
}
}
}
}
Groovy DSL
configurations.all {
resolutionStrategy.eachDependency { DependencyResolveDetails details ->
if (details.requested.name == 'log4j') {
throw new RuntimeException("Project depends on log4j")
}
}
}

You could do something as shown below. It is not a built-in Gradle function but does not require code to check each dependency specifically (it checks all in the compile configuration):
apply plugin: 'java'
dependencies {
compile files('lib/abc.jar')
compile files('lib/def.jar')
}
task checkDependencies() {
doLast {
configurations.compile.each { file ->
assert file.exists()
}
}
}
compileJava.dependsOn checkDependencies

To fail the build you can:
ant.fail('message why it failed')
Then you can craft a condition then fail the build with nice message ;)
I would suggest to create a task that will bring the file to the project first with a condition to check if the file is available etc if not then throw a Gradle exception and fail the build with a message, and execute the task first in the execution phase.
I have no chance to test it now but it could be something like this, correct me if any syntax is wrong - but you should get the idea.
def yourDep = $/\path\to\your\depdendency/$
task bringDeps << {
if (yourDep.exists()){
copy {
from yourDep
into $projectDir/depsOrSmthg
}
} else{
ant.fail('message why it failed')
}
}

task ensureDependenciesExist() {
doLast {
configurations.implementation.canBeResolved(true)
DependencySet deps = configurations.implementation.getDependencies()
Set<File> impFiles = configurations.implementation.resolve()
deps.each { d ->
boolean depWasResolved = impFiles.any { impFile -> impFile.name.find(".*${d.name}.*${d.version}") }
if (!depWasResolved) {
println "${d} was not resolved"
assert depWasResolved
}
}
}
}
compileJava.dependsOn ensureDependenciesExist

Gradle generates Querydsl metadata twice via different annotation processors

I have a gradle build script. I want said script to generate QueryDSL-Metadata. Those metadata should be generated under the build/generated-sources/metamodel Folder.
The problem I am facing at the moment is that the metamodel is not only being generated once, but twice. Along with the desired target it is also being generated in the "default" buld/classes/... resulting in a "duplicate class"-error.
sourceSets {
generated.java.srcDirs=['build/generated-sources/metamodel']
main {
java { srcDir 'src/main/java' }
}
test {
java { srcDir 'src/main/test' }
}
}
configurations { querydslapt }
dependencies {
compile 'org.hibernate:hibernate-entitymanager:5.2.3.Final',
'org.hibernate.javax.persistence:hibernate-jpa-2.1-api:1.0.0.Final-redhat-1',
'com.querydsl:querydsl-jpa:4.1.3',
// ... others, non-hibernate/querydsl ...
querydslapt 'com.querydsl:querydsl-apt:4.1.3'
}
task generateSources(type: JavaCompile, group: 'build', description:'Generates the QueryDSL query types') {
source = sourceSets.main.java
classpath = configurations.compile + configurations.querydslapt
options.compilerArgs = ['-proc:only',
'-processor', 'com.querydsl.apt.hibernate.HibernateAnnotationProcessor']
destinationDir = sourceSets.generated.java.srcDirs.iterator().next()
}
compileJava {
dependsOn generateSources
source generateSources.destinationDir
}
According to the gradle trace, the Problem appears to be that there are two AnnotatioProcessors in the mix. First, the HibernateAnnotationProcessor. Second, a JPAAnnotationProcessor, eventually generating the duplicate class. And I can't figure out why, the build script looks ok-ish. I know, it might be guesswork, but I am grateful for any suggestions. I even cleaned my gradle-cache, just in case. It might not even be a pure build-script related issue, but the behavior persists even if I run the script via console.
Gist, basically exactly what I "should" need
(older) Post regarding this issue

This thread's solution works for me, the idea is to hook the Annotation Processor into the javac, the HibernateAnnotationProcessor can be declared via compilerArgs, roughly like:
dependencies {
compile 'org.hibernate:hibernate-entitymanager:5.2.3.Final',
'org.hibernate.javax.persistence:hibernate-jpa-2.1-api:1.0.0.Final-redhat-1',
'com.querydsl:querydsl-jpa:4.1.4',
'com.querydsl:querydsl-apt:4.1.4',
// other
}
ext {
generatedSourcesDir = file("build/generated-sources/metamodel")
}
sourceSets {
main {
java {
srcDir 'src/main/java'
srcDir generatedSourcesDir
}
}
test {
java { srcDir 'src/main/test' }
}
}
compileJava {
doFirst {
generatedSourcesDir.mkdirs()
}
options.compilerArgs += ['-s', generatedSourcesDir,
'-processor', 'com.querydsl.apt.hibernate.HibernateAnnotationProcessor']
}
But I still wonder why the first approach does not work (runs two annotation processors), so any idea is still highly appreciated.