I'm trying to get code coverage data from a remote server, so I added a JVM argument:
-javaagent:/opt/jacocoagent.jar=output=tcpserver,port=6300,includes="a pretty long list"
but unfortunately the includes list is too long, that the java command has exceeded the maximum length of our system limits.
Is there any way to specify a external property file so I can put the long "includes list” there?
I've read the jacoco document, it seems when running in "Offline Instrumentation", the jacoco agent will read properties from jacoco-agent.properties if it appears in classpath. But I don't want to use this mode.
Found a solution myself.
When oracle JVM startup, it picks an environment variable JAVA_TOOL_OPTIONS and
the JNI_CreateJavaVM function (in the JNI Invocation API) prepends the
value of the environment variable to the options supplied in its
JavaVMInitArgs argument.
So in my case, I defined:
JAVA_TOOL_OPTIONS=-javaagent:/path/to/jacocoagent.jar=output=tcpserver,address=*,port=6300,includes="a pretty long list"
For details, you can refer to:
https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/envvars002.html
Related
I'm using a Kerberos enabled Spark cluster for running our Spark applications. The Kerberos has been setup previously by other members of the organization, and I have no idea how it works. In the initial days, we had used the Kerberos debug logs to understand the exception "Unable to obtain password from user" which was being raised due to absence of a JCE certificate in the cacerts folder of jre security. However, we no longer require the logs and thus, used the -Dsun.security.krb5.debug=false parameter to disable the logging. However, this did not have any effect. Is there any other parameter that could do the trick? Please help me.
Excerpt from the GitBook "Hadoop and Kerberos: The Madness Beyond the Gate" by Steve Loughran, chapter Low-Level Secrets
JVM Library logging
You can turn Kerberos low-level logging on -Dsun.security.krb5.debug=true
This doesn't come out via Log4J, or java.util logging; it just comes
out on the console. Which is somewhat inconvenient —but bear in mind
they are logging at a very low level part of the system. And it does
at least log. If you find yourself down at this level you are in
trouble. Bear that in mind.
If you want to debug what is happening in SPNEGO, another system
property lets you enable this: -Dsun.security.spnego.debug=true
You can ask for both of these in the HADOOP_OPTS environment variable
export HADOOP_OPTS="-Dsun.security.krb5.debug=true -Dsun.security.spnego.debug=true"
Hadoop-side JAAS debugging
Set the env variable HADOOP_JAAS_DEBUG to true and UGI will set the
"debug" flag on any JAAS files it creates.
You can do this on the client, before issuing a hadoop, hdfs or yarn
command, and set it in the environment script of a YARN service to
turn it on there.
export HADOOP_JAAS_DEBUG=true
On the next Hadoop command, you'll see a trace like (.........)
Caveat: the Java properties starting with sun.security. apply to Sun/Oracle Java run-time, and also OpenJDK run-time and its variants. But not to IBM Java, etc.
Excerpt from the Java 8 documentation under Troubleshooting Security
If you want to monitor security access, you can set the
java.security.debug System property.
(.......) Separate multiple options with a comma.
When troubleshooting Kerberos specifically, I personally use that combination:
-Djava.security.debug=gssloginconfig,configfile,configparser,logincontext
Excerpt from the Oracle JDK 9 Release Notes section tools/launcher
JDK 9 supports a new environment variable JDK_JAVA_OPTIONS to
prepend options to those specified on the command line. The new
environment variable has several advantages over the
legacy/unsupported _JAVA_OPTIONS environment variable including the
ability to include java launcher options (...)
These two env variables are a very dirty (and utterly difficult to detect) way to inject Java system properties without them appearing on the command line.
What does that mean for you? Well, you have to search for multiple Java system props and environment variables, which might be set
for env variables: globally (cf. /etc/profile.d/*.sh), or at account level (cf. ~/.bashrc and friends), or inside Hadoop "include files", or directly inside a shell script that runs your Spark job
for system props: in any shell-or-env variable that is later developed in a shell script (...) or any env var picked up by Java on startup, or in YARN configuration files (when using Spark-on-YARN), or directly on a Java command-line
Good luck.
I personally would run a dummy Spark job that just dumps all env variables and Java system props; then inspect the dump to detect what to search for; then run a brute-force find ... -exec grep ... on the Linux filesystem (repeat as needed).
My Java Webstart application runs in a controlled trusted environment. This is a closed internal network where I have some control on how the application is started.
How can I pass JVM arguments to the application, even if they are considered 'unsecure' for use by webstart by the JVM?
There are several options to pass JVM arguments to webstart.
Through JNLP file.
Through the JAVA_TOOL_OPTIONS environment variable.
Through the deployment settings on the local computer.
Through the javaws command (I was unable to get this to work).
Note that I have included links to the java 8 version of this documentation. All of these ideas are supported and documented in other Java versions, however sometimes they work a tiny bit different, or have slightly changed restrictions.
Through JNLP file.
The JNLP supports many JVM arguments through the JNLP file. Some can be done though direct settings, such as initial-heap-size and max-heap-size. For other settings java-vm-args can be used.
The JNLP File Syntax documentation lists some supported java-vm-args for 'this version' however it is unsure if that is the version 1.4+ of the example, or JRE 8. I know some unlisted settings are actually supported, such as -XX:MaxGCPauseMillis and activating the G1 garbage collector. You can make a JNLP and then use jinfo -flag MaxGCPauseMillis <pid> to test if a setting has been correctly propagated.
This is the preferred method, because it does not require any direct control of the JVM. The down-side is that only supports specific parameters that are considered 'safe'.
Through the JAVA_TOOL_OPTIONS environment variable
When you start Java Webstart by using the javaws command, you can use the JAVA_TOOL_OPTIONS to set any parameter you want on all JVM started from that environment.
So in Linux you can do to set an unsupported parameter:
export JAVA_TOOL_OPTIONS=-XX:SoftRefLRUPolicyMSPerMB=2000
javaws <my jnlp>
Note that this will affect all java applications ran with this system variable. So setting this for all users, or a specific user should be done with great care. Setting this only for a single application as the example above is much safer.
The advantage of this solution is that you can pass any parameter you want. The downside is that it requires a specific way of launching the application, or a very broad setting. It also requires control over the client system.
Through the deployment settings on the local computer
You can also pass JVM arguments by changing the deployment settings of the JVM. This can be done through the Java Control Panel, which allows you to set default runtime settings.
If you want to automate these settings you can use the deployment properties file. Unfortunately the JRE specific section of this file is undocumented. Manually it is very easy to adapt this file:
deployment.javaws.jre.0.args=-XX\:SoftRefLRUPolicyMSPerMB\=2000
When automating this file, you have to watch very carefully that it contains these settings for all detected JVMs, so you have to be sure to change the correct one. Also this will be used for all Webstart and applets on your system.
Through the javaws command (I was unable to get this to work)
There should be another way (besides the JAVA_TOOL_OPTIONS method) to change the parameters using the command line. The javaws documentation lists the -J option to pass arguments to the JVM, for example by running your JNLP as follows:
javaws -J-XX:SoftRefLRUPolicyMSPerMB=2000 <my jnlp>
However I have not been able to get this to actually set the JVM parameters.
Currently, I am trying to pass a system property to an executable in the following format: ./executable -Dvar="value" other parameters, since this is what I've seen people do for java files. I keep getting an error in the executable saying that -Dvar="value" doesn't exist as a parameter. Where am I going wrong? Are system properties exclusive to Java or something?
The -D parameter sets a system property. The system properties can be accessed through System.getProperty("<your parametername>");
A tutorial is given here https://docs.oracle.com/javase/tutorial/essential/environment/sysprop.html
The -D is consumed by the java runtime (java.exe) and will be invisible to your application on the commandline.
I have a jenkins job which uses buckminster to build an eclipse product.
At the beginning I have an "Extended Choice Parameter" where the "customer" key can be selected.
In the buckminster configuration I use this "customer" variable to select the right cquery:
import '${WORKSPACE}/source/scodi-customer/${customer}/server/features/ch.scodi.${customer}.server.feature/site.cquery'
Since the variable "customer" is per default not available in the commands, I added the following to the "JVM arguments":
-Dcustomer=${customer}
This all used to work well, but now I updated the server and build environment from Java 1.7 32-Bit to Java 1.8 64-Bit.
Since then I get the following error trying to build:
java.io.FileNotFoundException: [Path to job]\source\scodi-customer\${customer}\server\features\ch.scodi.${customer}.server.feature\site.cquery (The system cannot find the path specified)
Before the variable was resolved fine.
Is this a buckminster or java8 problem, not being able to resolve the ${customer} variable? Is there maybe another (cleaner) way to pass the variable to the buckminster configuration?
Edit: I did some further testing and added the following to the JVM arguments, I get the same exception referring to a missing ${customer}. It looks to me that JVM arguments are ignored.
-Dcustomer=CUSTOMER
Finally I found a workaround.
I had to rename my "customer" variable to "CUSTOMER", now it is resolved in the buckminster command area.
From:
import
'${WORKSPACE}/source/scodi-customer/${customer}/server/features/ch.scodi.${customer}.server.feature/site.cquery'
To:
import
'${WORKSPACE}/source/scodi-customer/${CUSTOMER}/server/features/ch.scodi.${CUSTOMER}.server.feature/site.cquery'
Also my JVM parameters from:
-Dcustomer=${customer}
To:
-Dcustomer=${CUSTOMER}
Found out the JVM parameters are not relevant to the buckminster command interface but are used later in "cspex" files.
Running Jenkins on a Windows machine this might cause this issue...
So the actual problem was the naming of the "Extended Choice Parameter", not being capitalized.
According to Oracle, the only way to set system properties is through command line -D parameters like that :
java -Dmy.prop=value com.package.MyClass
Is it really the only way ? Isn't it possible to create some system.properties file that will contain all these properties, and that would be automagically read when the JVM starts ?
I precise I can have no use of the System.setProperty(String,String) function.[1]
Setting this file through a command line parameter would be fine as well :
java -Fsystem.properties com.package.MyClass
I have searched where I know (and found there is a way with IBM's JVM), but I'm still empty-handed...
[1] : The goal is to set the default Charset, and this is primarily done through the file.encoding property, but only at the VM startup phase. Setting this property in runtime doesn't change the default Charset, and there is also no way to change it 'programmatically'.