I am trying to use deeplearning4j on Android device, switching between debugging on emulator and physical device.
For this I have multiple dependencies that are of a huge size and are all necessary. Thankfully, there is an implementation for each platform provided by org.bytedeco.
So targeting Android arm-64 which are the most common I omited other platforms like
// implementation group: 'org.nd4j', name: 'nd4j-native', version: '1.0.0-beta6', classifier: "android-x86_64" //for emulator
.
Wanted dependencies are like:
implementation (group: 'org.deeplearning4j', name: 'deeplearning4j-core', version: '1.0.0-beta7') {
exclude group: 'org.bytedeco', module: 'opencv-platform'
exclude group: 'org.bytedeco', module: 'leptonica-platform'
exclude group: 'org.bytedeco', module: 'hdf5-platform'
}
implementation group: 'org.nd4j', name: 'nd4j-native', version: '1.0.0-beta7'
implementation group: 'org.bytedeco', name: 'openblas', version: '0.3.9-1.5.3'
implementation group: 'org.bytedeco', name: 'opencv', version: '4.3.0-1.5.3'
implementation group: 'org.bytedeco', name: 'leptonica', version: '1.79.0-1.5.3'
implementation group: 'org.nd4j', name: 'nd4j-native', version: '1.0.0-beta7', classifier: "android-arm64"
implementation group: 'org.bytedeco', name: 'openblas', version: '0.3.9-1.5.3', classifier: "android-arm64"
implementation group: 'org.bytedeco', name: 'opencv', version: '4.3.0-1.5.3', classifier: "android-arm64"
implementation group: 'org.bytedeco', name: 'leptonica', version: '1.79.0-1.5.3', classifier: "android-arm64"
The problem is, when I build or clean and build the output Jar is huge, it is more than 800 mb, and if I see the tree of dependencies, I always see other platforms jars in path like:
I know that not all of these are necessary, because at one point in time (and I couldn't reproduce what was present and what I aim for again) I had not all of these dependencies. The exported APK was around 400 mb.
So what's up with the Windows, Linux, dependencies...
Sync project and Gradle clean Gradle build do nothing to these, exporting APK as well. Always an APK of > 800mb
Edit:
Please do not pay attention to versions mismatch between different dependencies and API and implementations. I am switching between versions dl4j beta6, dl4j beta7 and others
Related
I have been developing MS in Micronaut, which I want to run as Graalvm native image on AWS Lambda. I have few dependencies like Jackson ObjectMapper, Apache POI, etc. in the project which does use reflection framework. But, Graalvm doesn't support reflection framework out of the box. So, we need to provide the configuration for the classes which are using reflection framework. To generate such configuration, I am trying to use Graalvm tracing agent, but it doesn't seems to be working.
It generates the config JSON files successfully.
But these config JSONs doesn't contain the classes from 3rd party libs which are added in build.gradle.
I have below parameter in gradle.properties file:
org.gradle.jvmargs=-agentlib:native-image-agent=config-output-dir=<path to dir>
Then, I run the project using gradle's run task from the IntelliJ, and do execute all the methods which actually use these libs (thus all those libs' code gets executed).
Once the execution is stopped, I find few config files generated on the dir path mentioned, but those files contains only classes related to Gradle, Jetbrains and few others. I cannot find any classes related to 3rd party libs which I am using and which are mentioned in build.gradle.
What should be the correct way to generate the reflection configuration JSON file for Micronaut Graalvm native image?
build.gradle:
plugins {
id("com.github.johnrengelman.shadow") version "7.1.2"
id("io.micronaut.application") version "3.5.1"
}
version = "0.1"
group = "com.base.package"
repositories {
mavenCentral()
}
ext {
orgJsonVersion = "20220320"
poiVersion = "3.9"
apacheCommons = "3.12.0"
googleGson = "2.9.1"
springValidation = "2.7.3"
eclipseLink = "3.0.3"
servletApi = "2.5"
velocity = "1.7"
velocityTools = "2.0"
kafkaClient = "2.1.1"
googleGuava = "27.0-jre"
}
dependencies {
annotationProcessor('io.micronaut:micronaut-inject-java')
annotationProcessor("io.micronaut.data:micronaut-data-processor")
annotationProcessor("io.micronaut:micronaut-http-validation")
implementation("io.micronaut:micronaut-http-client")
implementation("io.micronaut:micronaut-jackson-databind")
implementation("jakarta.annotation:jakarta.annotation-api")
runtimeOnly("ch.qos.logback:logback-classic")
// runtimeOnly('org.slf4j:log4j-over-slf4j:2.0.2')
implementation("io.micronaut:micronaut-validation")
implementation("io.micronaut.data:micronaut-data-hibernate-jpa")
implementation("io.micronaut.sql:micronaut-jdbc-hikari")
compileOnly("io.micronaut.sql:micronaut-hibernate-jpa")
runtimeOnly("mysql:mysql-connector-java")
compileOnly("org.graalvm.nativeimage:svm")
// compileOnly group: 'io.micronaut', name: 'micronaut-http-server-netty'
implementation("io.micronaut.aws:micronaut-function-aws")
implementation("io.micronaut.aws:micronaut-function-aws-custom-runtime")
implementation group: 'org.json', name: 'json', version: "${orgJsonVersion}"
implementation group: 'org.apache.poi', name: 'poi', version: "${poiVersion}"
implementation group: 'org.apache.poi', name: 'poi-ooxml', version: "${poiVersion}"
implementation group: 'org.apache.commons', name: 'commons-lang3', version: "${apacheCommons}"
implementation group: 'com.google.code.gson', name: 'gson', version: "${googleGson}"
// implementation group: 'org.springframework.boot', name: 'spring-boot-starter-validation', version: "${springValidation}"
implementation group: 'org.eclipse.persistence', name: 'eclipselink', version: "${eclipseLink}"
// compileOnly group: 'javax.servlet', name: 'servlet-api', version: "${servletApi}"
implementation("org.apache.velocity:velocity:${velocity}") {
exclude group: 'log4j', module: 'log4j'
}
implementation("org.apache.velocity:velocity-tools:${velocityTools}") {
exclude group: 'log4j', module: 'log4j'
}
implementation group: 'org.apache.kafka', name: 'kafka-clients', version: "${kafkaClient}"
implementation group: 'com.google.guava', name: 'guava', version: "${googleGuava}"
// implementation group: 'org.hibernate', name: 'hibernate-core', version: '5.6.9.Final'
implementation group: 'com.fasterxml.jackson.datatype', name: 'jackson-datatype-hibernate5', version: '2.8.3'
implementation group: 'com.fasterxml.jackson.datatype', name: 'jackson-datatype-jsr310', version: '2.8.3'
}
application {
mainClass.set("com.base.package.MainApplication")
}
shadowJar {
mergeServiceFiles()
}
java {
sourceCompatibility = JavaVersion.toVersion("11")
targetCompatibility = JavaVersion.toVersion("11")
}
graalvmNative.toolchainDetection = false
micronaut {
runtime("lambda_provided")
testRuntime("junit5")
processing {
incremental(true)
annotations("com.base.package.*")
}
}
tasks.named("dockerfileNative") {
args(
"-XX:MaximumHeapSizePercent=80",
"-Dio.netty.allocator.numDirectArenas=0",
"-Dio.netty.noPreferDirect=true"
)
}
graalvmNative.binaries.all {
buildArgs.add("--no-server -J-Xmx12g -J-Xms4g -O0")
}
Since this is something you rarely do I usually run the command directly against the fat jar.
Here is how I do it.
Run ./gradlew build. This creates the fat jar in builds/libs
Goto build/libs
Run java -jar -agentlib:native-image-agent=config-output-dir=META-INF/native-image <your jar name>-all.jar
How to avoid redeclaring dependencies in gradle. com.foo.sso.keycloak was built using org.keycloak.keycloak-admin-client and I don't want to redeclare the dependency even if I use packages from keycloak-admin-client. How do I enable this?
MyApplication's build.gradle
implementation group: 'org.keycloak', name: 'keycloak-spring-boot-starter', version: '15.0.0'
implementation group: 'org.keycloak', name: 'keycloak-admin-client', version: '15.0.0'
implementation group: 'com.foo.sso.keycloak', name: 'sso-keycloak-admin', version: '15.0.0'
Use api when defining the dependencies in build.gradle of keycloak-admin-client. This will expose them to the projects that in turn depend on the client.
Ex:
// build.gradle (keycloak-admin-client)
api group: 'org.keycloak', name: 'keycloak-spring-boot-starter', version: '15.0.0'
// build.gradle (com.foo.sso.keycloak)
implementation group: 'org.keycloak', name: 'keycloak-admin-client', version: '15.0.0'
implementation group: 'com.foo.sso.keycloak', name: 'sso-keycloak-admin', version: '15.0.0'
I updated my java project with the latest Java JDK and Tomcat 10. The new libraries use jakarta and you need to rename all your javax. to jakarta...
The libraries included in gradle are:
compileOnly group: 'jakarta.servlet', name: 'jakarta.servlet-api', version: '5.0.0'
compileOnly group: 'jakarta.servlet.jsp', name: 'jakarta.servlet.jsp-api', version: '3.0.0'
compileOnly group: 'jakarta.el', name: 'jakarta.el-api', version: '4.0.0'
compileOnly group: 'jakarta.websocket', name: 'jakarta.websocket-api', version: '2.0.0'
compileOnly group: 'jakarta.security.enterprise', name: 'jakarta.security.enterprise-api', version: '2.0.0'
implementation group: 'jakarta.servlet.jsp.jstl', name: 'jakarta.servlet.jsp.jstl-api', version: '2.0.0'
implementation group: 'org.glassfish.web', name: 'jakarta.servlet.jsp.jstl', version: '2.0.0'
But I found I couldnt use Sitemesh and Htmlcompressor libraries anymore because they use old javax. library. If I include javax. libraries:
compileOnly group: 'javax.servlet', name: 'javax.servlet-api', version: '4.0.1'
I get java.lang.NoClassDefFoundError errors.
Is there a way to have compatibility with the old libraries?
Include both the old and the new libraries? Since there are two different namespaces, they might be able to live together. If it works, it should be enough as a transitory solution before everybody migrates to jakarta.
I'm using selenium for UI testing in java.
I have a module cloned from git hub which is being used in my project.
In spite of having the latest jar dependencies in my build.gradle file(intelliJ), I'm encountering
package org.openqa.selenium.html5 not found error
Only during runtime.
The selenium dependency does not complain at compile time, but during run time I am having errors that say
java:package org.openqa.selenium.logging does not exist
java:package org.openqa.selenium.remote does not exist
java:package org.openqa.selenium.support.ui does not exist
java:package org.openqa.selenium does not exist
I try clearing my gradle cache and re-installing the jars with the latest versions, and yet it's still the same issue.
These are the dependencies I have for selenium in my project:
implementation group: 'org.seleniumhq.selenium', name: 'selenium-java', version: '4.0.0-alpha-2'
implementation group: 'org.seleniumhq.selenium', name: 'selenium-server', version: '4.0.0-alpha-2'
implementation group: 'org.seleniumhq.selenium', name: 'selenium-api', version: '4.0.0-alpha-2'
implementation group: 'org.seleniumhq.selenium', name: 'selenium-support', version: '4.0.0-alpha-2'
implementation group: 'org.seleniumhq.selenium', name: 'selenium-remote-driver', version: '4.0.0-alpha-2'
implementation group: 'org.seleniumhq.selenium', name: 'selenium-chrome-driver', version: '4.0.0-alpha-2'
I also tried with different versions of selenium, but nothing works.
Any solutions that I could try to fix this issue.
Or am I missing something?
I am relatively new to using gradle, and am having trouble understanding dependencies. I have been using the Apache POI library to read from excel for a project, and initially I downloaded the JARs into my project and it worked completely fine. I decided to use gradle dependencies when I put my project on GitLab (from what I understand it is the same as using JARs but better when sharing the project since you don't have to download them?). This worked for most of the imports; however, for some reason my project is no longer able to import CellType. I redownloaded the JARs for the time being and it works again.
A summary of the question(s):
Why is CellType the only thing that stopped working, and why did it stop working? What is different about it?
Edit: here is the dependencies section from my build.gradle
compile 'com.google.apis:google-api-services-sheets:v4-rev499-1.23.0'
testCompile group: 'org.testng', name: 'testng', version: '6.9.10'
compile group: 'org.seleniumhq.selenium', name: 'selenium-java', version: '3.8.1'
// https://mvnrepository.com/artifact/com.google.oauth-client/google-oauth-client-java6
compile group: 'com.google.oauth-client', name: 'google-oauth-client-java6', version: '1.11.0-beta'
// https://mvnrepository.com/artifact/com.google.oauth-client/google-oauth-client-jetty
compile group: 'com.google.oauth-client', name: 'google-oauth-client-jetty', version: '1.11.0-beta'
// https://mvnrepository.com/artifact/org.apache.poi/poi
compile group: 'org.apache.poi', name: 'poi', version: '3.9'
// https://mvnrepository.com/artifact/org.apache.poi/poi-ooxml
compile group: 'org.apache.poi', name: 'poi-ooxml', version: '3.9'
// https://mvnrepository.com/artifact/org.apache.poi/poi-ooxml-schemas
compile group: 'org.apache.poi', name: 'poi-ooxml-schemas', version: '3.17'
// https://mvnrepository.com/artifact/org.apache.commons/commons-collections4
compile group: 'org.apache.commons', name: 'commons-collections4', version: '4.1'
(Sorry for the messiness :/ )
The compile error I was getting was something like "cannot resolve" (the import statement was turning red)
import org.apache.poi.ss.usermodel.CellType;