Environment variables with Spring Boot & Kubernetes
In my OS, the following command retrieves a key managed by K8s:
kubectl exec -it nginx-secret-7665cd8fb8-mj26h cat /mnt/secrets/ScAppCryptKey
As mentioned, the result is something like this:
xPeShVmYq3t6w9z#B#E?H#McQfTjWnZr4u7x%D*F-JaNdRgUkXp2s5v8y#B?E%
Now, I have to find a way to get this result (key) and use it in my Spring Boot Application. I think the most proper way to express my problem is:
During runtime (or when starting my app), I need to execute a command in the OS and get its result.
Is there a way to automatically execute commands in the terminal with Spring? Let me try to explain through code:
// If I had an environment variable, this would be like:
String key = System.getenv("myKey");
// But in my case, I need to execute a command, like this:
String keyFromKubernetes = System.getenv(
"kubectl exec -it nginx-secret-7665cd8fb8-mj26h cat /mnt/secrets/ScAppCryptKey"
);
I have two projects - A and B, they are in separate git repos.
Project A is a gradle+kotlin application with its own dependencies.
Project B is a gradle+kotlin testing suite. It wants to import Project A and start it new process like this:
val command = ArrayList<String>()
command.add(javaBin) // from java.home + path to java bin
command.addAll(jvmArgs)// user-specified args, usually empty
command.add("-cp")
command.add(classpath) // from java.class.path
command.add(className) // projectA.ClassWithMain
command.addAll(args) // user-specified args, usually empty
(source)
When I simply add jar via jitpack.io, the process starts, shows signs of life (prints internal logs) and then almost immediately fails with this error:
Exception in thread "main" java.lang.NoSuchMethodError: org.jetbrains.exposed.sql.Table.text$default(Lorg/jetbrains/exposed/sql/Table;Ljava/lang/String;Ljava/lang/String;ZILjava/lang/Object;)Lorg/jetbrains/exposed/sql/Column;
Note that Project A has this in dependecies:
implementation("org.jetbrains.exposed:exposed-jdbc:$exposedVersion")
What can I do to fix this?
I am trying a simple movie recommendation machine learning program in spark.
Spark version:2.1.1
Java version:java 8
Scala version: Scala code runner version 2.11.7
Env: windows 7
Running these commands to start master and worker slaves
//start master
spark-class org.apache.spark.deploy.master.Master
//start worker
spark-class org.apache.spark.deploy.worker.Worker spark://valid ip:7077
I am trying a very simple movie recommendation code from here: http://blogs.quovantis.com/recommendation-engine-using-apache-spark/
I have updated code to :
SparkConf conf = new SparkConf().setAppName("Collaborative Filtering Example").setMaster("spark://valid ip:7077");
conf.setJars(new String[] {"C:\\Spark2.1.1\\spark-2.1.1-bin-hadoop2.7\\jars\\spark-mllib_2.11-2.1.1.jar"});
I cannot run this thru intelliJ
Running mvn clean install and copying the jar to folder does not work.
The command I used to run on :
bin\spark-submit --verbose –-jars jars\spark-mllib_2.11-2.1.1.jar –-class “com.abc.enterprise.RecommendationEngine” –-master spark://valid ip:7077 C:\Spark2.1.1\spark-2.1.1-bin-hadoop2.7\spark-mllib-example\spark-poc-1.0-SNAPSHOT.jar C:\Spark2.1.1\spark-2.1.1-bin-hadoop2.7\spark-mllib-example\ratings.csv C:\Spark2.1.1\spark-2.1.1-bin-hadoop2.7\spark-mllib-example\movies.csv 10
The error I see is:
C:\Spark2.1.1\spark-2.1.1-bin-hadoop2.7>bin\spark-submit --verbose --class "com.sandc.enterprise.RecommendationEngine" --master spark://10.64.98.101:7077 C:\Spark2.1.1\spark-2.1.1-
bin-hadoop2.7\spark-mllib-example\spark-poc-1.0-SNAPSHOT.jar C:\Spark2.1.1\spark-2.1.1-bin-hadoop2.7\spark-mllib-example\ratings.csv C:\Spark2.1.1\spark-2.1.1-bin-hadoop2.7\spark-m
llib-example\movies.csv 10
Using properties file: C:\Spark2.1.1\spark-2.1.1-bin-hadoop2.7\bin\..\conf\spark-defaults.conf
Adding default property: spark.serializer=org.apache.spark.serializer.KryoSerializer
Adding default property: spark.executor.extraJavaOptions=-XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"
Adding default property: spark.eventLog.enabled=true
Adding default property: spark.driver.memory=5g
Adding default property: spark.master=spark://valid ip:7077
Error: Cannot load main class from JAR file:/C:/Spark2.1.1/spark-2.1.1-bin-hadoop2.7/û-class
Run with --help for usage help or --verbose for debug output
If I give the --jar command, it gives the error:
Error: Cannot load main class from JAR file:/C:/Spark2.1.1/spark-2.1.1-bin-hadoop2.7/û-jars
Any ideas how I can submit this job to spark??
Is your Jar built correctly ?
Also you don't need to add double quotes for --class option value.
I'm developing simple Spring web application that communicates with remote host and I would like to test it locally behind corporate proxy.
I use "Spring Boot" gradle plugin and the question is how can I specify proxy settings for JVM?
I have try several ways to do it:
gradle -Dhttp.proxyHost=X.X.X.X -Dhttp.proxyPort=8080 bootRun
export JAVA_OPTS="-Dhttp.proxyHost=X.X.X.X -Dhttp.proxyPort=8080"
export GRADLE_OPTS="-Dhttp.proxyHost=X.X.X.X -Dhttp.proxyPort=8080"
But it seems like none of them work - "NoRouteToHostException" throws in "network" code.
Also, I have added some extra code to debug JVM start arguments:
RuntimeMXBean runtimeMxBean = ManagementFactory.getRuntimeMXBean();
List<String> arguments = runtimeMxBean.getInputArguments();
for (String arg: arguments) System.out.println(arg);
And only one argument was printed: "-Dfile.encoding=UTF-8".
If I set system property in code:
System.setProperty("http.proxyHost", "X.X.X.X");
System.setProperty("http.proxyPort", "8080");
Everything works just fine!
Original Answer (using Gradle 1.12 and Spring Boot 1.0.x):
The bootRun task of the Spring Boot gradle plugin extends the gradle JavaExec task. See this.
That means that you can configure the plugin to use the proxy by adding:
bootRun {
jvmArgs = "-Dhttp.proxyHost=xxxxxx", "-Dhttp.proxyPort=xxxxxx"
}
to your build file.
Of course you could use the systemProperties instead of jvmArgs
If you want to conditionally add jvmArgs from the command line you can do the following:
bootRun {
if ( project.hasProperty('jvmArgs') ) {
jvmArgs project.jvmArgs.split('\\s+')
}
}
gradle bootRun -PjvmArgs="-Dwhatever1=value1 -Dwhatever2=value2"
Updated Answer:
After trying out my solution above using Spring Boot 1.2.6.RELEASE and Gradle 2.7 I observed that it was not working as some of the comments mention.
However, a few minor tweaks can be made to recover the working state.
The new code is:
bootRun {
jvmArgs = ["-Dhttp.proxyHost=xxxxxx", "-Dhttp.proxyPort=xxxxxx"]
}
for hard-coded arguments, and
bootRun {
if ( project.hasProperty('jvmArgs') ) {
jvmArgs = (project.jvmArgs.split("\\s+") as List)
}
}
for arguments provided from the command line
bootRun {
// support passing -Dsystem.property=value to bootRun task
systemProperties = System.properties
}
This should pass all JVM options to the app started via bootRun.
In gradle build script, define systemProperties for run task.
//to provide the properties while running the application using spring-boot's run task
run {
systemProperties['property name'] = 'value'
}
and gradle run should accept this value.
Or define a project level property as mentioned in
http://forums.gradle.org/gradle/topics/how_can_i_provide_command_line_args_to_application_started_with_gradle_run
#marvin, thanks for your post it was very helpful.
Sharing how I used it:
test {
// support passing -Dsystem.property=value to bootRun task
systemProperties = System.properties
}
I have JUnit tests that I wanted to skip unless a property was used to include such tests. Using JUnit Assume for including the tests conditionally:
//first line of test
assumeThat(Boolean.parseBoolean(System.getProperty("deep.test.run","false"),true)
Doing this with gradle required that the system property provided at the time of running gradle build, shown here,
gradle build -Ddeep.test.run=true
was indeed passed through to the tests.
Hope this helps others trying out this approach for running tests conditionally.
bootRun {
args = ['myProgramArgument1', 'myProgramArgument2']
}
Using jvmArgs may cause JVM start issues. Using args allows you to pass your custom program arguments
It seems to work:
bootRun {
systemProperties "property1": "value1", "property2": "value2"
}
I got into a similar problem, bootRun needed some parameters but I wouldn't feel like modifying bootRun as I want to keep some flexibility and stick to standard bootRun behaviour. My suggestion is to add some custom tasks (let's say bootRunDev, bootRunProxy) that extends bootRun, as described in the following code snippet
task bootRunPxy(type: org.springframework.boot.gradle.run.BootRunTask, dependsOn: 'build') {
group = 'Application'
doFirst() {
main = project.mainClassName
classpath = sourceSets.main.runtimeClasspath
systemProperty 'http.proxyHost', 'xxxxx'
systemProperty 'http.proxyPort', 'yyyyy'
}
}
I don't have an environment to exercise the script but I used this approach to pass profile to spring using the property spring.profiles.active.
Credits should go to Karol Kaliński
It's worth mentioning, here, that some systems that use Gradle and Spring Boot are starting JVM outside of build.gradle, e.g. in a Dockerfile.
It's not pointless to mention this on a thread specifically about bootRun! I wound up here because this particular post is a magnet for searches about jvm options in the context of a Spring Boot app compiled / run under gradle. (All the advice I found for adding java.net.http.httpclient logging said "add it to bootRun's jvmArgs". Nothing happened, though.
So, if you happen to run your gradle-built Spring Boot app from a Docker container, you'll want to add your JVM params to an env var in your project's Dockerfile, like so, e.g.-
...
ENV JAVA_OPTS "${JAVA_OPTS} \
-server \
-Duser.timezone=UTC \
-XX:InitialRAMPercentage=50 \
-XX:MaxRAMPercentage=50 \
-Djavax.net.ssl.trustStorePassword=elvislives \
-Djavax.net.ssl.trustStoreProvider=BCFIPS \
-Djavax.net.ssl.trustStoreType=BCFKS \
-Djdk.internal.httpclient.debug=true \
-Djava.util.logging.manager=org.apache.logging.log4j2.jul.LogManager \
-Djdk.httpclient.HttpClient.log=errors,requests,headers,frames[:control:data:window:all..],content,ssl,trace,channel \
"
...
ENTRYPOINT java ${JAVA_OPTS} -cp app:app/lib/* com.mygreatcompany.theapp
For development as Docker Container add to run_script.sh as JAVA_OPTS
JAVA_OPTS="-XX:+UseG1GC
-Xms512m
-Xmx2048m
--add-opens java.base/java.util=ALL-UNNAMED -Dspring.profiles.active=$PROFILE,discovery"
I have an APT processor that display warnings on some conditions.
My project is using a maven1 build calling ant:apt
How to make maven fails when warning conditions are met ? (the processor can be modified)
Thanks.
The annotation processor needs to create a message of Kind ERROR. This results in a Compilation Failure, which in turn will abort the ant build (unless the failonerror parameter of the ant javac task is set to false). (And this in turn should fail the maven task)
processingEnvironment
.getMessager()
.printMessage(Kind.ERROR, "your error message here");
(Acquire the ProcessingEnvironment through the processor's init method)
Reference:
ProcessingEnvironment.getMessager()
Messager.printMessage(Kind, CharSequence)
Diagnostic.Kind.ERROR