We Keep Coding

Strange behavior with different java application export option

I have java server application wich uses many libs (netty, guava, etc). I always export this application as one single .jar. When I run application in Eclipse, I didn't have any problems. But if I start app in console (Windows, or Ubuntu, doesn't matter), I have strange problem: ALL connection processes via sockets last toooo long. For example, simple http connection via HttpAsync or others (rabbitmq connection, etc.) lasts 1-2 min. But after connection completed, data sends/receives fast. I can't figure what the problem. As mentioned before, I use Eclipse for development.
As you know, you can export project 3 dif ways (in Eclipse):
Extract required libraries into JAR.
Package required libraries into JAR.
Copy required libraries into sub folder next to JAR.
So, when I used 2 option, I had problem. When I switched to 3d option (all .jars in folder near main .jar), problem was solved.
Generally there are no big difference between 2 and 3 option (in 2 all .jars just inside one jar). I thought that it was cause of extra time needed to load new classes in execution time from the jars. But problem occurs not only at start, but for all new connections.
Can someone explain this behavior?
UPD: Eclipse Luna. Doesn't matter what OS I'm using (Windows, or Ubuntu), even doesn't matter what jvm (tried with different Oracle jdk, even tried open jdk).

This all talks about difference in performance when packaging into JAR v/s extracting into JAR & difference in performance when running from Eclipse v/s running from console.
Difference in performance when packaging into JAR v/s extracting into JAR:
Extract required libraries into JAR:
What it does:
In this option Eclipse will extract all the classes from the referenced JARs and package into the generated JAR.
If you open the JAR then you will find that there are NO referenced JARs packaged but all the classes of referenced JARs are arranged as per the package structure and then packaged inside the JAR at root level. This brings the key difference in performance as compared to the "Packaging required libraries into a jar file" where there is additionally cost of runtime parsing and loading of JAR in memory etc..
When exporting as JAR through Eclipse then it is best option if performance is concern. Also this is scalable option because you can ship this JAR
MANIFEST.MF Main thing to note in this file is you main class. When you run the JAR you are directly running the class you need.
Main-Class: com.my.jar.TestSSL
Package required libraries into JAR:
What it does:
In this option Eclipse will:
package all the referenced JARs into the generated JAR.
employ Eclipse's JAR loading mechanism through org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader and you can also see org.eclipse.jdt.internal.jarinjarloader package into your generated JAR and this package is just under the root directory of the generated JAR.
Now of course this is the additional cost which comes when you choose this option because when you run the JAR then it is not you main class getting executed but JarRsrcLoader will be executed which will load your main class and other libraries, and all the referenced libraries are packaged. See MANIFEST.MF section below
MANIFEST.MF Main thing to note in this file is you main class. When you run the JAR, JarRsrcLoader will run and will do further job.
Rsrc-Main-Class: com.cgi.tmi.TestSSL
Main-Class: org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader
Now for last Eclipse export option - "Copy required libraries into sub folder next to JAR", I don't think it is a very scalable solution to consider because this imposes your file system dependency, so I would say don't do it.
Difference in performance when running from Eclipse v/s running from console:
When you run application from Eclipse then it is quiet similar to 1st export option where Eclipse doesn't need to parse and load JARs at runtime and all.
This is however a very trivial point, key is the consideration of Eclipse JAR export option 1 v/s option 2.
Final words:
Use "Extract required libraries into JAR" for exporting JAR and you will see substantial performance gain.
It is highly improbable that your socket connections are lasting long when you run from console because JVM runs code then it would have same or very comparable performance when running from Eclipse and console (considering same Java versions in both case). You could be feeling because of packaged JAR performance. Try extracted JAR and you should be fine.
Also, consider the amount of logging you are doing. When running through, depending upon configuration Eclipse may mask a lot of logging and hence saving you i/o time.
Do understand how classes are accessed from JAR class path, which is like additional computational cost when you are referencing classes from JAR.

As we don't know the exact structure of your JAR here is a more general explanation (assumed you run your application with java -jar your_app.jar).
case Copy required libraries into sub folder next to JAR.
if a class needs to be loaded the class loader (after the runtime JAR) first checks your_app.jar to find a required class
if the class is not found it traversed over all JAR files in the subfolder
all JAR files could be kept in the filesystem cache for further reading
case Package required libraries into JAR
if a class needs to be loaded the Eclipse class loader JarRsrcLoader (after the runtime JAR) first checks your_app.jar to find a required class
if the class is not found it traversed over all embedded JAR files, which means as first they need to be decompressed from your_app.jar before the content can be read
the extracted embedded JAR files are not kept in the filesystem cache for further reading (as they are not files in the filesystem)
If you have a bigger number of hugh embedded library JARs this might lead in a slow down of class loading (but only for the first time a class is loaded by a class loader).
You can see the difference in the class loading if you compare the outpout of
java -verbose:class -jar your_app_external_library_jars.jar
with
java -verbose:class -jar your_app_embedded_library_jars.jar
The performance might be improved by generating an INDEX.LIST file for each JAR file (e.g. your_app.jar and the embedded library JARs).

It happens because when you go with "uber jar" approach, some metadata might be lost.
It's just an example, but if you download this and this, take a look inside the jar. There are a few files with the same name in the same META-INF folder.
Those files might be important, and when eclipse repackages things for you, he might not be the doing a decent job on merging such files.
That is what might be happening to you.

In the 2nd approach, You have all dependency jars in the main.jar.
So it won't load any of the dependency jars unless required.
Whereas, in case of 3rd option, your main.jar and other dependency jars are independent (unlike 2nd way), and hence gets loaded for connections and is available.
try adding a log statement or syso by manipulating a dependency jar to see this working.

How to refer a jar outside exe in Java desktop application?

I have a Java application installed. The jar is bundled into an .exe file using Launch4J. Now I want to create a patch for this application.
If I create another jar containing only updated files, how can I refer it in the original code?

I have java application installed. ..Now I want to create a patch for this application.
This is one of the strengths of the Java Web Start launch technology that comes with the J2SE. Simply update the Jar on the server, and the next time the app. launches, it will be updated.
The update can be honed for your use-case, configured to be done lazily or eagerly, before or after launch, or even programatically controlled using the JNLP API's DownloadService.
..And the jar is bundlled into an .exe file ..
'Unfortunately', JWS works on Windows, ..and Mac., and *nix - so you may have to expand your horizons.
BTW - I have no idea how to do the same with Launch4J, but then, that is really the wrong question. I aim to provide an answer to the right question, which is "How do I deploy & update a Java rich client?". ;)

I've never worked with Launch4J, however I think you should try to affect the classpath. JRE always loads the classes from the classpath. From this point of view, jars have no added value and just serve as a containers for your *.class files and resources.
Now, if you succeed to configure your tool to do something like:
classpath = C:\Temp\my_patch_path;$your_current_classpath
then its enough to put your changed files into C:\Temp\my_patch_path (of course preserving the package structure). JRE will load your classes first in this case.
Hope, this helps
Mark

It is might not be possible to do this without changing the contents of the exe.

How to load a library that depends on another library, all from a jar file

I would like to ship my application as a self-contained jar file. The jar file should contain all the class files, as well as two shared libraries. One of these shared libraries is written for the JNI and is essentially an indirection to the other one (which is 100% C).
I have first tried running my jar file without the libraries, but having them accessible through the LD_LIBRARY_PATH environment variable. That worked fine.
I then put the JNI library into the jar file. I have read about loading libraries from jar files by copying them first to some temporary directory, and that worked well for me (note that the 100% C library was, I suppose, loaded as before).
Now I want to put both libraries into the jar, but I don't understand how I can make sure that they will both be loaded. Sure I can copy them both to a temporary directory, but when I load the "indirection" one, it always gives me:
java.lang.UnsatisfiedLinkError: /tmp/.../libindirect.so: /libpure.so: cannot open shared object file: No such file or directory
I've tried to force the JVM to load the "100% C" library first by explicitely calling System.load(...) on its temporary file, but that didn't work better. I suspect the system is looking for it when resolving the links in libindirect.so but doesn't care about what the JVM loaded.
Can anyone help me on that one?
Thanks

One way would be to spawn another Java process from the first, generating the appropriate invocation script.
The jar is invoked by the user
The libraries are extracted to a temp directory
A (bash) script is written to the temp directory
this sets/exports the necessary environment variables
this launches the second JRE instance
The code makes the script executable
The code invokes the script
I know, spawning two JRE instances to launch one app would not be my first choice either.

If you are using Eclipse IDE, then this answer might help you.
I had same problem in eclipse windows that I couldn't added dependant .class files from the JNI.
After searching for a while I came to know that "Its a known bug inside Eclipse", In order resolve the same, I ported all the code to NetBeans IDE.
Can not add all the classes files from the JNI folder in Eclipse (JAVA, Windows 7)

The conventional location for storing my Java libraries and applications in UNIX based systems

I usually store the Java applications and JAR files that I download from the Web in the ~/Java folder on my computer (an OS X machine). I have been doing this since the days when I was a Windows user. However I think in UNIX based systems user local apps are conventionally stored in another directory. I have a feeling that this directory should either be /usr/local/, /usr/local/USERNAME, /opt/local, or /opt/local/USERNAME but I am not sure. Any ideas which directory can I use for this purpose?
Please note that, I am talking about archive files that I download from the Web, unpack and use locally and not programs that have installation scripts or MacPorts, etc.

The para-answer is that you shouldn't be downloading jars by hand at all, you should be using Gradle, Ivy, Maven, or something similar to manage your jars for you. These tools take a simple specification of your dependencies as input, and go and find, download, store, and make available all the necessary jar files. This takes a little bit of getting used to, but are rather wonderful once you're in the swing of it.
A direct answer, though, is that on orthodox unix, these files belong in /usr/local/share. usr because they're read-only data that is not part of the base operating system, local because they are not being supplied by the operating system (which owns the rest of /usr), and share because jar files are architecture-independent.
Note that on FreeBSD, the 'ports' package management system puts files in /usr/local, but i believe it shares it with the local administrator. There isn't some other location where purely local files go.
If the system has a convention for where package-managed jars go, then copy that under /usr/local. For example, on Ubuntu, there is /usr/share/java, so you should use /usr/local/share/java.
Further, if the system has a convention for handling versions, copy that. Again, on Ubuntu, jars are all stored in one directory, with version numbers in the name, but with a versionless symlink pointing to the default/latest version. So, i have a file at /usr/share/java/xstream-1.3.1.jar, and a symlink at /usr/share/java/xstream.jar pointing to it. I'd use the same approach in /usr/local/share/java.
Now, that's for orthodox unix. You're on OS X, which is not orthodox unix. Still, the principles apply: find how the system stores jars it provides, and transpose that into a user-managed filesystem space.

There isn't any blessed way to do it. You may, however, run into having multiple versions of a jar, and then it just goes downhill from there.
I usually download the jars I need as a distribution, and unpack it to its own folder, and then add the jars to the projects I need them for in my IDE. For libraries a common approach is to use Maven and its dependency handling.
So, my suggestion is to keep your current way of doing it, if you like that, but have each project in its own folder, like
~/Java/jakarta-commons-net-1.1.8/commons-net.jar

Apple have a note giving /Library/Java/Extensions as the directory for shared jars and ~/Library/Java/Extensions for jars just for yourself. These paths are on the classpath.
The jars can be anywhere as long as that directory is on your class path. (I use a version of
Thorbjørn Ravn Andersen's style I use ~/Library/Jar/jakarta-commons-net-1.1.8/commons-net.jar)
See Tom Anderson's answer for a better way to do it using ivy etc

You might wish to read the Filesystem Hierarchy Standard /opt or /usr/local are probably appropriate but you should read there FHS definitions first.

On FreeBSD systems the location is ${LOCALBASE}/share/java -- and subdirectories thereof. Not that all software-ports respect it, but they should.
LOCALBASE is usually /usr/local -- unless overwritten by the system administrator.

Java distribuion as jar file containg config, libs and deps

I am developing a framework that needs a lot of stuff to get working. I have several folders inside of my Eclipse project that are needed
[root]
- config
- src
- lib
- serialized
Also there are important files like the log4j.properties and the META-INF dir inside the src directory.
I wonder if there is a way to distribute one JAR containing all essential files so my gui will just have to import one jar. I guess that I have to exclude the config folder in order to make the framework configurable.
I also wonder, if there is a way to move for example the log4j.properties to the config dir so that I have one config folder containg all needed configurations?
Thanks for help and advise on this matter!
Marco

Yes, but not really. You can take all your dependencies, unpack them and simply merge them into a bigger jar. This is what the maven jar plugin does if you make a jar with dependencies. The only problem is that this might result in conflicting files (suppose two of your dependencies contain a log4j.properties). This is one of the problems when doing the above with some of the spring libraries for instance.
I think someone actually wrote a classloader that allows you to bundle the whole jar inside of your jar and use it as is. I'm not sure how mature that is though and can't at the moment recall the name.
I think you're better off distributing all your dependencies separately. Setting up the classpath is a bit of a pain but surely java programmers are used to it by now. You can add dependencies to the Class-Path header in your manifest file, in simple cases. Bigger libraries have to rely on the classpath being set up for them though.
As to the second part of your question, probably dropping the conf/ directory under META-INF is enough for its contents to be picked up. I'm not sure about this. I'm fairly sure it will always be picked up if you put its contents at the top level of the jar. In any case, this is a distribution problem. You can easily have a conf/ directory inside your source tree and have your build scripts (whatever you might be using) copy the files in it to wherever is most convenient.
As to your users configuring. Try to establish some conventions so they have to configure as little as possible. For things that must be configured, it's best to have a basic default configuration and then allow the user to override and add options through his/her own configuration file.

In terms of the resources, it is possible except that if you do that you are not going to be able to load resources (non class files) from the filesystem (via a file path).
It's likely that you're currently loading these resources from the file system. Once in the jar you need to load them as class path resources via the class.getResourceAsStream or similar.
As for the dependent jars you may have, it's common practice for these to be placed as extra jars on the classpath. I know it's complicates things but developers are used to doing this. The nature of the java landscape is that this is inevitable. What the spring framework for example does is supply a bundled zip file with the core jar and the jar dependencies included.
Is your library going to be used in an EE context or an SE context? If it is an EE context then you really don't have to worry about configuration and class path issues as the container takes care of that. In an SE context it is a lot more tricky as that work has to be done manually.