We Keep Coding

Geotools "Failed to load the GDAL native libs." at runtime, OK in Eclipse

I'm trying to run the GeoTools ImageLab.java exmaple. It works fine inside Eclipse and I'm using Maven as in the instructions. I'm trying to load a GeoTIFF. But when I use Eclipse to build a runnable jar file and run that from a DOS command line I get:
Jun 11, 2018 6:49:57 PM it.geosolutions.imageio.gdalframework.GDALUtilities loadGDAL
WARNING: Failed to load the GDAL native libs. This is not a problem unless you need to use the GDAL plugins: they won't be enabled.
java.lang.UnsatisfiedLinkError: no gdaljni in java.library.path
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
Caused by: java.lang.UnsupportedOperationException: Trying to get a reader from an unknown format.
at org.geotools.coverage.grid.io.UnknownFormat.getReader(UnknownFormat.java:53)
at org.geotools.tutorial.raster.ImageLab.displayLayers(ImageLab.java:103)
at org.geotools.tutorial.raster.ImageLab.getLayersAndDisplay(ImageLab.java:87)
at org.geotools.tutorial.raster.ImageLab.main(ImageLab.java:61)
The JNI reference makes me think I need a gdal dll file, but I'm having trouble finding one if that's the case.
All help appreciated! Thanks in advance.

If you are using the gt-imageio-ext-gdal module then you will need to make sure that the JVM can see your gdal libraries and that they are the exact version expected by the module. There are detailed instructions on how to install that on the related GeoServer page.
However, for a simple GeoTiff there is no need for GDAL as this is handled by the gt-geotiff module.
Finally, if you are trying to build a single jar application you should make sure you have read and understood this FAQ entry.

"No dependency satisfies type class org.neo4j.kernel.api.index.SchemaIndexProvider" when building über-jar

I have manually downloaded and added the libraries log4j, jsoup and neo4j for a project in Intellij Idea.
Before adding neo4j, I could build a jar as an artifact with all the libraries extracted to it so i could directly copy it somewhere and run with java -jar file.jar.
When I run the application in IntelliJ Idea it always works!
Now i get the following error when running the jar:
Error: A JNI error has occurred, please check your installation and try again Exception in thread "main" java.lang.SecurityException: Invalid signature file digest for Manifest main attributes
at sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:314)
at sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:268)
at java.util.jar.JarVerifier.processEntry(JarVerifier.java:316)
at java.util.jar.JarVerifier.update(JarVerifier.java:228)
at java.util.jar.JarFile.initializeVerifier(JarFile.java:383)
at java.util.jar.JarFile.getInputStream(JarFile.java:450)
at sun.misc.JarIndex.getJarIndex(JarIndex.java:137)
at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:839)
at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:831)
at java.security.AccessController.doPrivileged(Native Method)
at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:830)
at sun.misc.URLClassPath$JarLoader.<init>(URLClassPath.java:803)
at sun.misc.URLClassPath$3.run(URLClassPath.java:530)
at sun.misc.URLClassPath$3.run(URLClassPath.java:520)
at java.security.AccessController.doPrivileged(Native Method)
at sun.misc.URLClassPath.getLoader(URLClassPath.java:519)
at sun.misc.URLClassPath.getLoader(URLClassPath.java:492)
at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:457)
at sun.misc.URLClassPath.getResource(URLClassPath.java:211)
at java.net.URLClassLoader$1.run(URLClassLoader.java:365)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:495)
I already searched for the error and found that a library seems to be signed and I can't just pack it into another jar, because it thinks it was modified then.
The suggested solution was to exclude the signing files through the build system.
I tried to manually delete the BCKEY.dsa and the BCKEY.sf files (only signing files in META-INF folder) in the resulting jar but then I get the following error:
Exception in thread "main" java.lang.RuntimeException: Error starting org.neo4j.kernel.impl.factory.CommunityFacadeFactory, /home/xuiqzy/Documents/uni/4.Semester/PRG_practicum/WikiXtractor/data
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:144)
at org.neo4j.kernel.impl.factory.CommunityFacadeFactory.newFacade(CommunityFacadeFactory.java:40)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:108)
at org.neo4j.graphdb.factory.GraphDatabaseFactory.newDatabase(GraphDatabaseFactory.java:100)
at org.neo4j.graphdb.factory.GraphDatabaseFactory.lambda$createDatabaseCreator$0(GraphDatabaseFactory.java:89)
at org.neo4j.graphdb.factory.GraphDatabaseBuilder.newGraphDatabase(GraphDatabaseBuilder.java:183)
at org.neo4j.graphdb.factory.GraphDatabaseFactory.newEmbeddedDatabase(GraphDatabaseFactory.java:65)
at de.bened.wikixtractor.DatabaseManager.initialize(DatabaseManager.java:43)
at de.bened.wikixtractor.Main.main(Main.java:83)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.NeoStoreDataSource#5c67716' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:444)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
at org.neo4j.kernel.impl.transaction.state.DataSourceManager.start(DataSourceManager.java:99)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:434)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:140)
... 8 more
Caused by: org.neo4j.kernel.impl.util.UnsatisfiedDependencyException: No dependency satisfies type class org.neo4j.kernel.api.index.SchemaIndexProvider
at org.neo4j.kernel.impl.util.Dependencies.resolveDependency(Dependencies.java:71)
at org.neo4j.kernel.impl.util.Dependencies.resolveDependency(Dependencies.java:67)
at org.neo4j.kernel.NeoStoreDataSource.start(NeoStoreDataSource.java:418)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:434)
... 13 more
I also searched that error, a META-INF/services/org.neo4j.kernel.extension.KernelExtensionFactory file exists in my jar, there are also the jars lucene-analyzers-commen, lucene-codecs, lucene-core, lucene-queryparser, neo4j-lucene-index and neo4j-lucene-upgrade all integrated in the über-jar and the build in IntelliJ Idea seems to have enough dependencies, too.
So I'm a bit clueless why the jar doesn't work as opposed to the build in the IDE.
Long term I'm willing to switch to a build system but for now:
Is there a way to get the jar to run?
Any help or pointers in the right direction are appreciated! :)

[Eclipse not starting]:An error has occured. See the log file null

I'm unable to get my Eclipse Luna running. It was working fine till now and suddenly crashed. Now when i try to start it, I get the error saying:
An error has occured. See the log file null
I tried launching eclipse from terminal and also tried to delete the metadata folder from my workspace but nothing helped me.
Here is the crash log:
org.osgi.framework.BundleException: Error initializing container.
at org.eclipse.osgi.container.SystemModule.init(SystemModule.java:93)
at org.eclipse.osgi.internal.framework.EquinoxBundle$SystemBundle.init(EquinoxBundle.java:209)
at org.eclipse.osgi.internal.framework.EquinoxBundle$SystemBundle.init(EquinoxBundle.java:201)
at org.eclipse.osgi.launch.Equinox.init(Equinox.java:168)
at org.eclipse.core.runtime.adaptor.EclipseStarter.startup(EclipseStarter.java:296)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:231)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:648)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:603)
at org.eclipse.equinox.launcher.Main.run(Main.java:1465)
Caused by: java.lang.ExceptionInInitializerError
at org.eclipse.osgi.internal.framework.EquinoxBundle$SystemBundle$EquinoxSystemModule.initWorker(EquinoxBundle.java:137)
at org.eclipse.osgi.container.SystemModule.init(SystemModule.java:83)
... 12 more
Caused by: java.security.ProviderException: setSeed() failed
at sun.security.provider.NativePRNG$RandomIO.implSetSeed(NativePRNG.java:458)
at sun.security.provider.NativePRNG$RandomIO.access$300(NativePRNG.java:329)
at sun.security.provider.NativePRNG.engineSetSeed(NativePRNG.java:212)
at java.security.SecureRandom.getDefaultPRNG(SecureRandom.java:209)
at java.security.SecureRandom.<init>(SecureRandom.java:190)
at org.eclipse.osgi.internal.framework.UniversalUniqueIdentifier.computeNodeAddress(UniversalUniqueIdentifier.java:127)
at org.eclipse.osgi.internal.framework.UniversalUniqueIdentifier.<clinit>(UniversalUniqueIdentifier.java:35)
... 14 more
Caused by: java.io.IOException: Operation not permitted
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:313)
at sun.security.provider.NativePRNG$RandomIO.implSetSeed(NativePRNG.java:456)
... 20 more

It seems you're using Solaris? This OS has an alternative entropy source called EGD, which needs special protocol to write to, which Sun JDK does not implement...
Can you try adding -Djava.security.egd=/dev/random to the vmargs section of your eclipse.ini.
If you still see the same issue. Make sure your user can write to /dev/random.

I fixed it myself. Few files in my security folder /jre/lib/security had got messed up.
Downloaded latest Java Cryptography Extension (JCE) and put the jars (Download US_export_policy.jar and local_policy.jar) in the security folder. My eclipse works fine now.

Java RMI ClassNotFoundException on stub occurs in Linux but not Windows

I'm getting the following exception when attempting to use Naming.lookup() to create an RMI object:
java.rmi.UnmarshalException: error unmarshalling return; nested exception is:
java.lang.ClassNotFoundException: project.server.data.RmiMainObjImpl_Stub
at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
at java.rmi.Naming.lookup(Naming.java:101)
... 2 more
Caused by: java.lang.ClassNotFoundException: project.server.data.RmiMainObjImpl_Stub
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.rmi.server.LoaderHandler$Loader.loadClass(LoaderHandler.java:1206)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at sun.rmi.server.LoaderHandler.loadClassForName(LoaderHandler.java:1219)
at sun.rmi.server.LoaderHandler.loadClass(LoaderHandler.java:452)
at sun.rmi.server.LoaderHandler.loadClass(LoaderHandler.java:185)
at java.rmi.server.RMIClassLoader$2.loadClass(RMIClassLoader.java:637)
at java.rmi.server.RMIClassLoader.loadClass(RMIClassLoader.java:264)
at sun.rmi.server.MarshalInputStream.resolveClass(MarshalInputStream.java:214)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1612)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
... 4 more
Curiously, this error only happens under Linux and AIX, but never under Windows. The error happens consistently, but seemingly works under some versions of Java:
The original jar was built with 1.6.0u30 on a Windows machine
The original jar will not work under Linux using 1.7.0u60, but will work with 1.6.0u24
The original jar will not work under AIX using 1.6 64-bit, but will work with 32-bit
Building the jar on a Linux machine using 1.7.0u60 will actually not work with the same runtime on that machine
My policy file seems to be set up correctly and is being recognized (though, I'm guessing I'd have a different error if it was not):
grant {
permission java.security.AllPermission "", "";
};
I'm executing java with a command line similar to this:
java -cp ./.:./JProjApi.jar:./MyRMI.jar -Djava.security.policy=./policy.all com/project/rmi/Main
And the code looks like this:
System.setSecurityManager(new RMISecurityManager());
rmiObj = (project.server.data.RmiMainObj_1_0) Naming.lookup("rmi://172.17.44.45/RMIMain");
I do not have access to the server-side (and I'm consuming a jar that has the necessary client-side interfaces as well).
Any ideas on what might be going wrong?

the stub class [is] not in the client-side jar.
The stub class must be in the client JAR file, unless you're using the codebase feature.
why does the Windows version work?
If it's working on one platform and not others, the stub class is clearly in the CLASSPATH on the platforms where it works, and not in the platforms where it doesn't. Or, if you're using the codebase feature, which you haven't said anything about, the codebase URL isn't accessible from all the platforms.

Figuring this out became a much lower priority, but it came back.
Network scanning showed nothing unusual.
It turned out that it did, in fact, have at least something to do with the Java version or new settings on newer Java versions.
This:
-Djava.rmi.server.useCodebaseOnly=false
Had to be added to the command line and it now works for those machines it didn't previously work on.

Why would Java classloading fail on Linux, but succeed on Windows?

I've got a Java web application (using Spring), deployed with Jetty. If I try to run it on a Windows machine everything works as expected, but if I try to run the same code on my Linux machine, it fails like this:
[normal startup output]
11:16:39.657 INFO [main] org.mortbay.jetty.servlet.ServletHandler$Context.log>(ServletHandler.java:1145) >16> Set web app root system property: 'webapp.root' = [/path/to/working/dir]
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.mortbay.start.Main.invokeMain(Main.java:151)
at org.mortbay.start.Main.start(Main.java:476)
at org.mortbay.start.Main.main(Main.java:94)
Caused by: java.lang.ExceptionInInitializerError
at org.springframework.web.util.Log4jWebConfigurer.initLogging(Log4jWebConfigurer.java:129)
at org.springframework.web.util.Log4jConfigListener.contextInitialized(Log4jConfigListener.java:51)
at org.mortbay.jetty.servlet.WebApplicationContext.doStart(WebApplicationContext.java:495)
at org.mortbay.util.Container.start(Container.java:72)
at org.mortbay.http.HttpServer.doStart(HttpServer.java:708)
at org.mortbay.util.Container.start(Container.java:72)
at org.mortbay.jetty.Server.main(Server.java:460)
... 7 more
Caused by: org.apache.commons.logging.LogConfigurationException: org.apache.commons.logging.LogConfigurationException: No suitable Log constructor [Ljava.lang.Class;#15311bd for org.apache.commons.logging.impl.Log4JLogger (Caused by java.lang.NoClassDefFoundError: org/apache/log4j/Category) (Caused by org.apache.commons.logging.LogConfigurationException: No suitable Log constructor [Ljava.lang.Class;#15311bd for org.apache.commons.logging.impl.Log4JLogger (Caused by java.lang.NoClassDefFoundError: org/apache/log4j/Category))
at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:543)
at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
at org.springframework.util.SystemPropertyUtils.(SystemPropertyUtils.java:42)
... 14 more
Caused by: org.apache.commons.logging.LogConfigurationException: No suitable Log constructor [Ljava.lang.Class;#15311bd for org.apache.commons.logging.impl.Log4JLogger (Caused by java.lang.NoClassDefFoundError: org/apache/log4j/Category)
at org.apache.commons.logging.impl.LogFactoryImpl.getLogConstructor(LogFactoryImpl.java:413)
at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:529)
... 18 more
Caused by: java.lang.NoClassDefFoundError: org/apache/log4j/Category
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
at java.lang.Class.getConstructor0(Class.java:2699)
at java.lang.Class.getConstructor(Class.java:1657)
at org.apache.commons.logging.impl.LogFactoryImpl.getLogConstructor(LogFactoryImpl.java:410)
... 19 more
Caused by: java.lang.ClassNotFoundException: org.apache.log4j.Category
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
... 24 more
[shutdown output]
I've run the app with java -verbose:class, and according to that output, org.apache.log4j.Category is loaded from the log4j JAR in my /WEB-INF/lib, just before the first exception is thrown.
Now, the Java versions on the two machines are slightly different. Both the machines have Sun's java, the Linux machine has 1.6.0_10, while the Windows machine has 1.6.0_08, or maybe 07 or 06, I can't remember the exact number right now, and don't have the machine at hand. But even though the minor versions of the Javas are slightly different, the code shouldn't break like this. Does anyone understand what's wrong here?

You must understand that a classloader can't see everything; they can only see what a parent classloader has loaded or what they have loaded themselves. So if you have two classloaders, say one for Jetty and another for your webapp, your webapp can see log4j (since the JAR is the WEB-INF/lib) but Jetty's classloader can't.
If you manage to make a class available to Jetty (for example something in the DB layer) which uses log4j but which ends up running in the context (and classloader) of Jetty, you will get an error.
To debug this, set a breakpoint in org.springframework.web.util.Log4jWebConfigurer.initLogging(). If you can, copy the source of this class into your project (don't forget to delete it afterwards) and add this line:
ClassLoader cl = Thread.currentThread().getContextClassLoader();
Have a look at the cl object in your debugger. That should give you some information who created it. My guess is that this is the classloader from Jetty.
[EDIT] Note that you get in a different mess if you have log4j in both classloaders: In this case, you will have two classes with the same name which create objects which are not assignment compatible! So make sure there is only a single instance of this jar or that instances of log4j will never be passed between the two contexts (which is usually not possible).

This seems like a classic classloader problem. It could be due to another web app being loaded first which also uses log4j but a version that is different to the one used by the app you are testing. The classloader uses the first version of the class it finds. The server class loading policy can usually be changed in the config files. Sorry I am a bit rusty on this but maybe it can point you in the correct direction.
Make sure there are no other installed apps on the web server,
Make sure the log4j being loaded is the correct version,
Make sure you don't have a log4j lurking somewhere in the classpath of the server.
HTH

You're using the same WAR on both machines? Have you checked if the WAR files are identical (no transfer errors occured)?

Some random things to consider:
(1) Check if there are any other versions of log4j floating around on the linux instance, outside of the web-app directories?
(2) Is apache commons logging being used at all? You might want to consider SLF4J instead?
(3) Did the JAR/WAR become corrupt in some way - was it FTP'ed in ASCII or Binary?
(4) Print out the classloader hierarchy in each case, just to see if there are any discrepancies?

Even though the original problem was solved for the asker, I'll point out that a common source of problems when running the same code on Windows vs Linux (or Unix) is case-sensitivity issues. Windows ignores case while Linux or Unix is case-sensitive. This has bitten me more than once.
So if you specify a jar or directory on the classpath, but it isn't the right case then it will fail on Linux but succeed on Windows. This can also be the source of FileNotFoundExceptions.

Had the same problem and found an easy solution/workaround:
In Eclipse in Preferences > Java > Installed JREs, select the JRE > Edit and Add External JARs... and browse to your log4j.jar.
The other workaround is to add log4j.jar to every launch definition in Classpath tab.