We Keep Coding

How to unzip a library to a temporary directory, load it and all of its dependencies from that directory

Essentially I want to zip up collection of shared objects in a jar file, unzip them to a temporary directory at runtime, and load from there. Say for example I have the libraries:
libApp.so
libBasicPlugin.so => libApp.so
libPlugin1.so => libApp.so, libBasicPlugin.so, libPlugin2.so
libPlugin2.so => libApp.so, libBasicPlugin.so
Java unzips them to an arbitrary, unique temporary directory $USER_HOME/.my-app/2011_12_05_001/, and then loads libApp.so, which in turn uses dlopen to load libPlugin1 and libPlugin2 (libBasicPlugin isn't explicitly loaded). The problem is that libPlugin1 doesn't know where to find libBasicPlugin and libPlugin2.
Here are the various solutions I have tried/considered:
Putting the libraries in a known location and setting LD_LIBRARY_PATH in a sh file before invoking Java. This works but precludes packaging the libraries in a jar
Using Java's System.load to "preload" the dependencies. This is the approach I've seen suggested, but only seems to work if you know a priori what plugins are going to be loaded and know their dependencies. Unless you iterated over all the files in a directory, loading them until you stop getting UnsatisfiedLinkExceptions...
Somehow tell the shared object that its dependencies will be in the same directory (via DT_RPATH?). This is what I think I would like in an ideal world, but it seems the best you can do is set the library location relative to the executable, which in this case is /usr/lib/jvm/.../java.
Statically link each library with its dependencies. I'm worried this would result in two copies of libPlugin2 (a dynamic one and a static one) which has caused all sorts of problems for us in the past.
On Windows, calling SetDllDirectory before loading the plugins solves this nicely. Are there any solutions I am overlooking/misunderstanding? Am I going about this entirely the wrong way?

You could set the LD_LIBRARY_PATH env var from within the Java process itself.

How to refer a jar outside exe in Java desktop application?

I have a Java application installed. The jar is bundled into an .exe file using Launch4J. Now I want to create a patch for this application.
If I create another jar containing only updated files, how can I refer it in the original code?

I have java application installed. ..Now I want to create a patch for this application.
This is one of the strengths of the Java Web Start launch technology that comes with the J2SE. Simply update the Jar on the server, and the next time the app. launches, it will be updated.
The update can be honed for your use-case, configured to be done lazily or eagerly, before or after launch, or even programatically controlled using the JNLP API's DownloadService.
..And the jar is bundlled into an .exe file ..
'Unfortunately', JWS works on Windows, ..and Mac., and *nix - so you may have to expand your horizons.
BTW - I have no idea how to do the same with Launch4J, but then, that is really the wrong question. I aim to provide an answer to the right question, which is "How do I deploy & update a Java rich client?". ;)

I've never worked with Launch4J, however I think you should try to affect the classpath. JRE always loads the classes from the classpath. From this point of view, jars have no added value and just serve as a containers for your *.class files and resources.
Now, if you succeed to configure your tool to do something like:
classpath = C:\Temp\my_patch_path;$your_current_classpath
then its enough to put your changed files into C:\Temp\my_patch_path (of course preserving the package structure). JRE will load your classes first in this case.
Hope, this helps
Mark

It is might not be possible to do this without changing the contents of the exe.

lambdaj installation

we have downloaded jar files for lambdaj and its dependencies which are again jar files.
we do not know how to go about it. we have copied these files in the
C:\Program Files\Java\jre6\lib\ext
have set the class path in environment variales as:
variable: classpath
path: C:\Program Files\Java\jre6\lib\ext
but we do not know how to go further. we want to run some lambdaj programs.
can anyone suggest how to run lambdaj programs?

You would run a Java program that requires lambdaj in exactly the same way you'd run any other java program with an external dependency, i.e. by invoking the java executable passing in the fully-qualified name of the Main class, or the JAR with an appropriate manifest, or by deploying it in a servlet container, etc. Additionally you should be putting the LambdaJ JAR on the classpath for this invocation, not in the lib folder for your entire JVM.
What have you tried so far and why/how is it not working? Your question at the moment is a bit analogous to "I want to use Microsoft Word to view some Word documents, how do I do this?".
Update for comment 1: You said "it's not working". That doesn't help anyone address your problem as it gives no clue what you expected to happen and what you observed, only that they were different. As for where JAR files can be stored - you can put them in any directory, so long as that directory is on the classpath (or you add it to the classpath) of the Java application that runs. The canonical place to put external dependencies is in a folder called lib below the root of your project, but the important thing is that you choose somewhere consistent and sensible.
It sounds like you don't quite grok Java and classpaths yet. If you have followed some tutorials and are still stuck, ask for help to let you understand. Adding more detail to your question, including the layout of your files, the commands you issued, and the response that came back would be useful too.

If you are using Netbeans create a project and right click on the Libraries folder within the desired project. Click Add JAR/Folder...

The conventional location for storing my Java libraries and applications in UNIX based systems

I usually store the Java applications and JAR files that I download from the Web in the ~/Java folder on my computer (an OS X machine). I have been doing this since the days when I was a Windows user. However I think in UNIX based systems user local apps are conventionally stored in another directory. I have a feeling that this directory should either be /usr/local/, /usr/local/USERNAME, /opt/local, or /opt/local/USERNAME but I am not sure. Any ideas which directory can I use for this purpose?
Please note that, I am talking about archive files that I download from the Web, unpack and use locally and not programs that have installation scripts or MacPorts, etc.

The para-answer is that you shouldn't be downloading jars by hand at all, you should be using Gradle, Ivy, Maven, or something similar to manage your jars for you. These tools take a simple specification of your dependencies as input, and go and find, download, store, and make available all the necessary jar files. This takes a little bit of getting used to, but are rather wonderful once you're in the swing of it.
A direct answer, though, is that on orthodox unix, these files belong in /usr/local/share. usr because they're read-only data that is not part of the base operating system, local because they are not being supplied by the operating system (which owns the rest of /usr), and share because jar files are architecture-independent.
Note that on FreeBSD, the 'ports' package management system puts files in /usr/local, but i believe it shares it with the local administrator. There isn't some other location where purely local files go.
If the system has a convention for where package-managed jars go, then copy that under /usr/local. For example, on Ubuntu, there is /usr/share/java, so you should use /usr/local/share/java.
Further, if the system has a convention for handling versions, copy that. Again, on Ubuntu, jars are all stored in one directory, with version numbers in the name, but with a versionless symlink pointing to the default/latest version. So, i have a file at /usr/share/java/xstream-1.3.1.jar, and a symlink at /usr/share/java/xstream.jar pointing to it. I'd use the same approach in /usr/local/share/java.
Now, that's for orthodox unix. You're on OS X, which is not orthodox unix. Still, the principles apply: find how the system stores jars it provides, and transpose that into a user-managed filesystem space.

There isn't any blessed way to do it. You may, however, run into having multiple versions of a jar, and then it just goes downhill from there.
I usually download the jars I need as a distribution, and unpack it to its own folder, and then add the jars to the projects I need them for in my IDE. For libraries a common approach is to use Maven and its dependency handling.
So, my suggestion is to keep your current way of doing it, if you like that, but have each project in its own folder, like
~/Java/jakarta-commons-net-1.1.8/commons-net.jar

Apple have a note giving /Library/Java/Extensions as the directory for shared jars and ~/Library/Java/Extensions for jars just for yourself. These paths are on the classpath.
The jars can be anywhere as long as that directory is on your class path. (I use a version of
Thorbjørn Ravn Andersen's style I use ~/Library/Jar/jakarta-commons-net-1.1.8/commons-net.jar)
See Tom Anderson's answer for a better way to do it using ivy etc

You might wish to read the Filesystem Hierarchy Standard /opt or /usr/local are probably appropriate but you should read there FHS definitions first.

On FreeBSD systems the location is ${LOCALBASE}/share/java -- and subdirectories thereof. Not that all software-ports respect it, but they should.
LOCALBASE is usually /usr/local -- unless overwritten by the system administrator.

Java distribuion as jar file containg config, libs and deps

I am developing a framework that needs a lot of stuff to get working. I have several folders inside of my Eclipse project that are needed
[root]
- config
- src
- lib
- serialized
Also there are important files like the log4j.properties and the META-INF dir inside the src directory.
I wonder if there is a way to distribute one JAR containing all essential files so my gui will just have to import one jar. I guess that I have to exclude the config folder in order to make the framework configurable.
I also wonder, if there is a way to move for example the log4j.properties to the config dir so that I have one config folder containg all needed configurations?
Thanks for help and advise on this matter!
Marco

Yes, but not really. You can take all your dependencies, unpack them and simply merge them into a bigger jar. This is what the maven jar plugin does if you make a jar with dependencies. The only problem is that this might result in conflicting files (suppose two of your dependencies contain a log4j.properties). This is one of the problems when doing the above with some of the spring libraries for instance.
I think someone actually wrote a classloader that allows you to bundle the whole jar inside of your jar and use it as is. I'm not sure how mature that is though and can't at the moment recall the name.
I think you're better off distributing all your dependencies separately. Setting up the classpath is a bit of a pain but surely java programmers are used to it by now. You can add dependencies to the Class-Path header in your manifest file, in simple cases. Bigger libraries have to rely on the classpath being set up for them though.
As to the second part of your question, probably dropping the conf/ directory under META-INF is enough for its contents to be picked up. I'm not sure about this. I'm fairly sure it will always be picked up if you put its contents at the top level of the jar. In any case, this is a distribution problem. You can easily have a conf/ directory inside your source tree and have your build scripts (whatever you might be using) copy the files in it to wherever is most convenient.
As to your users configuring. Try to establish some conventions so they have to configure as little as possible. For things that must be configured, it's best to have a basic default configuration and then allow the user to override and add options through his/her own configuration file.

In terms of the resources, it is possible except that if you do that you are not going to be able to load resources (non class files) from the filesystem (via a file path).
It's likely that you're currently loading these resources from the file system. Once in the jar you need to load them as class path resources via the class.getResourceAsStream or similar.
As for the dependent jars you may have, it's common practice for these to be placed as extra jars on the classpath. I know it's complicates things but developers are used to doing this. The nature of the java landscape is that this is inevitable. What the spring framework for example does is supply a bundled zip file with the core jar and the jar dependencies included.
Is your library going to be used in an EE context or an SE context? If it is an EE context then you really don't have to worry about configuration and class path issues as the container takes care of that. In an SE context it is a lot more tricky as that work has to be done manually.