We Keep Coding

Java webapp produces StackOverflow exception on Linux

I have a Java-web application running on Tomcat 9. It works fine on a Windowns machine. But after deploying this app on a Debian Linux server I've encountered java.lang.StackOverflow exceptions on some particular pages. Here is the beginning of the stacktrace log:
27-Dec-2017 08:54:43.746 SEVERE [https-jsse-nio-9443-exec-3]
org.apache.catalina.core.ApplicationDispatcher.invoke Servlet.service() for
servlet [jsp] threw exception
java.lang.StackOverflowError
at java.io.UnixFileSystem.canonicalize0(Native Method)
at java.io.UnixFileSystem.canonicalize(UnixFileSystem.java:172)
at java.io.File.getCanonicalPath(File.java:618)
at org.apache.catalina.webresources.AbstractFileResourceSet.file(AbstractFileResourceSet.java:90)
Please, see full stacktrace here:
https://pastebin.com/0AmFDY8F
As far as I understand, the exception occurs while compiling the JSP page source by Jasper and it's somehow related to the resolving of Linux paths / directories. Could anybody please help me to figure out what the exact problem is? I didn't create any symbolic links, if the problem can be related to this.

According to your stack trace, the problem is caused by not being able to find a specific path as resolved by your Spring framework:
org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:154)
...
Caused by: java.io.IOException: JSPException including path '/struct/context/tutorials.jsp'.
The Spring framework is likely using the canonical paths to resolve the file, and it fails perhaps because there is some quirk in the framework related to the type of Unix system you are deploying to.
Please reference this in the Spring documentation:
https://docs.spring.io/spring-boot/docs/current/reference/html/deployment-install.html#deployment-install-supported-operating-systems
The default script supports most Linux distributions and is tested on CentOS and Ubuntu. Other platforms, such as OS X and FreeBSD, will require the use of a custom embeddedLaunchScript.
Notice that Debian was not listed. Hopefully this puts you on the right path.
If I'm chasing a red herring here, I'm sorry - but you haven't given us much to go on.

Finally I've found what caused the StackOverflow exception. I have a recursive JSP tag call to render nested options for a <select> tag. After removing the recursion, the exception disappeared. Sure, it's just a workaround, so I'm still wondering why it works fine on Windows and doesn't on Linux.

NoClassDefFoundError "wrong name" for a class in the java.lang package

I'm running Cassandra 2.2.11 (and won't be upgrading) on a host. Periodically, in a cron job, I run nodetool commands for monitoring. nodetool is implemented as just another java process that uses JMX to talk to the Cassandra java process. I launch five or so commands every minute.
Once in a while (not in any recognizable pattern), the execution of nodetool will fail with a NoClassDefFoundError that refers to a class from java.lang. For example,
java.lang.NoClassDefFoundError: java/lang/Thread (wrong name: java/lang/Thread)
at java.lang.Class.getDeclaredFields0(Native Method)
at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
at java.lang.Class.getDeclaredField(Class.java:2068)
at java.util.concurrent.FutureTask.<clinit>(FutureTask.java:476)
at java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:590)
at sun.rmi.transport.tcp.TCPChannel.free(TCPChannel.java:347)
at sun.rmi.server.UnicastRef.free(UnicastRef.java:431)
at sun.rmi.server.UnicastRef.done(UnicastRef.java:448)
at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:132)
at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:205)
at javax.naming.InitialContext.lookup(InitialContext.java:417)
at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1955)
at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1922)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:183)
at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:150)
at org.apache.cassandra.tools.NodeTool$NodeToolCmd.connect(NodeTool.java:302)
at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:242)
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:158)
In this stack trace, the error happens during class initialization for FutureTask. I've also seen
java.lang.NoClassDefFoundError: java/lang/Object (wrong name: java/lang/Object)
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.getDeclaredMethod(Class.java:2128)
at java.lang.invoke.MethodHandleImpl$Lazy.<clinit>(MethodHandleImpl.java:614)
[...]
but also
java.lang.NoClassDefFoundError: java/lang/String (wrong name: java/lang/String)
at java.lang.Class.getDeclaredFields0(Native Method)
at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
at java.lang.Class.getDeclaredField(Class.java:2068)
at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1703)
at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:72)
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:484)
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472)
[...]
So it's not only happening during class initialization, but, in the few samples I've collected, something in the reflection implementation does seem to be the culprit.
Java is at version 8
java version "1.8.0_144"
The nodetool launcher always uses the same classpath. And there are no weird classes in there (or additional class loaders). The same installation is done across hundreds of identical nodes (on Linux).
My top search results for NoClassDefFoundError wrong name refer to executions where a simplified class name was used to launch java, rather than the fully qualified name. That's not the issue here. Also, the names in the error messages are identical.
So what can cause such "wrong name" NoClassDefFoundError errors for "bootstrap" classes?

i think it is the lack of resource that cause the problems like the connector timeout or something. Do you see the log from your example?;
nodeprobe is connecting through the jmx or trying to connect then the error occurs?
Those are very typical error that can also cause other intermiten error on the shit.(usually OS/netowrk OS shit) thus :
includes your string and even object based error ;in conclusion it make sense.
may be you should check your resource when the error happen.
i know this is kind of catch 22 that the resource monitor is causing the lack of resource instead; but it happen hehe

According to the stacktraces, the exception id being thrown in a calls to getDeclaredFields0. However, this is not where the exception came from originally. According to the OpenJDK source code, there is nothing in the codebase that throws an exception with "wrong name" in the exception message. The message has come from somewhere else.
I strongly suspect that this is actually re-reporting a problem that happened the first time that some class was loaded or initialized. What happens is that the classloader finds the problem the first time, marks the offending internal class object as "bad" and then the throws the error. According to the javadoc, applications should not attempt to recover from this. But if one does, and then attempts to use the "bad" class in some way, the original problem will be reported again as a NoClassDefFoundError with the original reason.
So what does the reason mean?
It is hard to tell because we don't have the stacktrace for the original exception; i.e. then one where the classloading / initialization first failed. If you can find that stacktrace, we can track down the 3rd-party library that did it. It is almost certainly happening in a classloader.
The obvious meaning is that a class file has a classname in it that doesn't match the name in the classes bytecodes. However, we'd need to examine the classloader code to be sure.
So why is it happening intermittently?
Possibly because the application JVM has many classloaders and only a subset of them have "polluted" their class namespace with this bad class.
That could be bad news. It suggests there may be some kind of synchronization issue in the core of the application.
Anyhow, there is not enough evidence to draw sound conclusions.
Bottom line
Based on the evidence, I would guess that this is a result of some kind of "code weaving" or "byte code engineering" that has gone wrong. As a further guess, I would say that some child classloader is not delegating properly, and has mistakenly attempted to process a built-in class. (It could even be that the classloader in question knows that it should never process a "java.lang.*" class and it has a obscure way of saying this.)
Why? possibly because someone / something explicitly added the "rt.jar" to some classpath that it shouldn't be on.
For further diagnosis, the first thing we need is the original stacktrace that tells us which classloader did the initial damage.

As none of basic java library found,I think there is problem in your java installation or you have not set CLASSPATH and JAVA_HOME environment variables. Try to set CLASSPATH and JAVA_HOME environment variables.
export JAVA_HOME="/usr/lib/jvm/java-8-oracle/bin"
export CLASSPATH="/usr/lib/jvm/java-8-oracle/lib"
If not worked, try reinstall java and set environment variables.

Tomcat: JRStyledTextParser not Initialized. java.awt is set to true

I keep getting java.lang.NoClassDefFoundError: Could not initialize class net.sf.jasperreports.engine.util.JRStyledTextParser
The full stacktrace is:
java.lang.NoClassDefFoundError: Could not initialize class net.sf.jasperreports.engine.util.JRStyledTextParser
net.sf.jasperreports.engine.fill.JRBaseFiller.<init>(JRBaseFiller.java:108)
net.sf.jasperreports.engine.fill.JRVerticalFiller.<init>(JRVerticalFiller.java:69)
net.sf.jasperreports.engine.fill.JRVerticalFiller.<init>(JRVerticalFiller.java:57)
net.sf.jasperreports.engine.fill.JRFiller.createBandReportFiller(JRFiller.java:200)
net.sf.jasperreports.engine.fill.JRFiller.createReportFiller(JRFiller.java:215)
net.sf.jasperreports.engine.fill.JRFiller.fill(JRFiller.java:115)
net.sf.jasperreports.engine.JasperFillManager.fill(JasperFillManager.java:583)
net.sf.jasperreports.engine.JasperRunManager.runToPdf(JasperRunManager.java:455)
net.sf.jasperreports.engine.JasperRunManager.runReportToPdf(JasperRunManager.java:870)
com.evnica.interop.main.ReportServlet.createReport(ReportServlet.java:119)
com.evnica.interop.main.ReportServlet.doGet(ReportServlet.java:96)
com.evnica.interop.main.ReportServlet.service(ReportServlet.java:79)
javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
I have searched for the solution and found a bunch.
Could not initialize class net.sf.jasperreports.engine.util.JRStyledTextParser
JasperReports NoClassDefFoundError exception on net.sf.jasperreports.engine.util.JRStyledTextParser
and about 10 more (this is my first question so I can't enlist them all).
I have tried:
To set java.awt.headless=true.
I did it within code:
static {
System.setProperty("java.awt.headless", "true");
System.out.println(java.awt.GraphicsEnvironment.isHeadless());
}
(returns true as expected).
I did it in JVM Options:
I set CATALINA_OPTS to -Djava.awt.headless=true. Echoes as expected.
It didn't help.
I tested it with.jrxml with DejaVu Sans and included jasperreports-fonts-6.2.2.jar in the path, and I tested it with Sans Serif. I checked which fonts were available to JVM and tested with them. No success.
There were suggestions to check versions of jasperreports jars - I checked it too. I have all the jars of 6.2.2. version:
jasperreports-6.2.2.jar,
jasperreports-fonts-6.2.2.jar,
jasperreports-javaflow-6.2.2.jar.
I use joda-time-2.9.3 (have no idea how it's connected to the problem, but one advice was to use joda; I used it from the beginning anyway), project SDK is java 1.8.0_51 (there was an advice to update to 8 - not relevant as I already use it), and my Tomcat is OK (the absent work folder caused similar problem in one of the cases).
What am I missing?
EDIT:
While debugging I came to a method initializeGrophEnv() in JRGraphEnvInitializer class, which throws a util.graphic.environment.initialization.error:
AVAILABLE_FONT_FACE_NAMES.addAll(Arrays.asList(GraphicsEnvironment.getLocalGraphicsEnvironment().getAvailableFontFamilyNames()));
It's interesting that the method getAvailableFontFamilyNames() in SunGraphicsEnvironment returns a valid array of 274 elements (printscreen from debugger under link):

First of all, thanks everyone for help. You responded so fast, and made me dig deeper =) The problem is solved.
My error was to have both jasperreports-6.2.2.jar and jasperreports-javaflow-6.2.2.jar in the libraries. As you remember, the last method in the stack trace was the initialization of JRBaseFiller. The presence of two jars caused a conflict, as both contain a package net.sf.jasperreports.engine.fill with a JRBaseFiller class in it.
I left the jasperreports-javaflow-6.2.2.jar only. It didn't work on its own. I added two more libraries: jfreechart-1.0.19.jar (previous versions didn't contain all the needed classes) and jcommon-1.0.8. The java.lang.NoClassDefFoundError is gone.

UnsatisfiedLinkError and NoClassDefFoundError : Running Java Mapscript on Ubuntu 11.10

We have coded and run a dynamic web application using MAPSERVER(Version 6.0.1) on windows platform using Java Technology. Now, there is need of deploying it on Ubuntu 11.10. We have installed Apache Tomcat 6.0, Mapserver 6.0.1, Apache 2.0, and FWTools-2.0.1(As this package contain all required tools for mapserver if I am not wrong, so I didn't feel any other tools to be installed). We have deployed the war file(and put application without) in Apache Tomcat 's Webapps folder. I even got the index page which dont have code related with mapscript api. But while fetching the other servlet with mapscript we are getting following error...
java.lang.UnsatisfiedLinkError: no mapscript in java.library.path
java.lang.ClassLoader.loadLibrary(ClassLoader.java:1681)
java.lang.Runtime.loadLibrary0(Runtime.java:840)
java.lang.System.loadLibrary(System.java:1047)
edu.umn.gis.mapscript.mapscriptJNI.<clinit>(mapscriptJNI.java:23)
edu.umn.gis.mapscript.mapObj.<init>(mapObj.java:283)
Again while refreshing the browser page where the above error was displayed, I got a change,
java.lang.NoClassDefFoundError: Could not initialize class
edu.umn.gis.mapscript.mapscriptJNI
edu.umn.gis.mapscript.mapObj.<init>(mapObj.java:283)
I searched on net about the above problem. But finally blank. Please, provide idea about the above problem.

I'm not going to explain why you're getting the UnsatisfiedLinkError, but instead I'll explain why you are getting the NoClassDefFoundError when you reload the page.
A NoClassDefFoundError with a message Could not initialize class ... is thrown by the JVM when it attempts to initialize a class that it has already tried and failed to initialize.
The first of your two stacktraces contains the line
edu.umn.gis.mapscript.mapscriptJNI.<clinit>(mapscriptJNI.java:23)
The method name <clinit> denotes the static initializer, of the class mapscriptJNI. So, at the point that the UnsatisfiedLinkError was thrown, the JVM was trying to initialize this class. Looking at the error message, it seems that this static initializer tried to load the native code library mapscript but failed.
This UnsatisfiedLinkError causes the mapscriptJNI class to fail to initialize successfully. The JVM keeps a record of all classes that fail to initialize, and if you attempt to initialize one of those classes again, you'll get a NoClassDefFoundError with a message saying that it could not initialize that class.
When you refresh the page, you end up causing the JVM to attempt to initialize the class mapscriptJNI a second time. Of course, this class failed to initialize the previous time. Your second stacktrace contains exactly the error I've described.
In short, the UnsatisfiedLinkError is the real error here. Fix that and the other one will go away.

I would check the following 2 items:
Is the mapscript.jar file on Tomcat or at least your webapp's classpath? (NoClassDefFoundError is your big clue here)
Is the libmapscript.so on either your LD_LIBRARY_PATH or -Djava.library.path? (UnsatisfiedLinkError since the shared object cannot be found)
Try having a look at this post, near the Running Java Mapscript (on Linux) section.
Hope that helps!

Deploying WAR with CXF in Glassfish V3

I'm trying to deploy a WAR in GF V3 that has CXF as a dependency and I get the following exception:
[#|2011-02-08T07:34:15.736-0800|WARNING|oracle-glassfish3.0.1|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=32;_ThreadName=http-thread-pool-8080-(2);|StandardWrapperValve[MyServlet]: PWC1406: Servlet.service() for servlet MyServlet threw exception
com.sun.xml.ws.util.ServiceConfigurationError: com.sun.xml.ws.api.pipe.TransportPipeFactory: Provider com.sun.enterprise.jbi.serviceengine.bridge.transport.JBITransportPipeFactory is specified in bundle://254.0:0/META-INF/services/com.sun.xml.ws.api.pipe.TransportPipeFactory but not found
at com.sun.xml.ws.util.ServiceFinder.fail(ServiceFinder.java:241)
at com.sun.xml.ws.util.ServiceFinder.access$100(ServiceFinder.java:141)
at com.sun.xml.ws.util.ServiceFinder$LazyIterator.next(ServiceFinder.java:376)
at com.sun.xml.ws.api.pipe.TransportTubeFactory.create(TransportTubeFactory.java:129)
at com.sun.xml.ws.transport.DeferredTransportPipe.<init>(DeferredTransportPipe.java:82)
at com.sun.xml.ws.api.pipe.ClientTubeAssemblerContext.createTransportTube(ClientTubeAssemblerContext.java:311)
at com.sun.xml.ws.assembler.jaxws.TransportTubeFactory.createTube(TransportTubeFactory.java:62)
at com.sun.xml.ws.assembler.TubeCreator.createTube(TubeCreator.java:77)
at com.sun.xml.ws.assembler.TubelineAssemblerFactoryImpl$MetroTubelineAssembler.createClient(TubelineAssemblerFactoryImpl.java:121)
at com.sun.xml.ws.client.Stub.createPipeline(Stub.java:224)
at com.sun.xml.ws.client.Stub.<init>(Stub.java:201)
at com.sun.xml.ws.client.Stub.<init>(Stub.java:174)
at com.sun.xml.ws.client.sei.SEIStub.<init>(SEIStub.java:81)
at com.sun.xml.ws.client.WSServiceDelegate.createEndpointIFBaseProxy(WSServiceDelegate.java:602)
at com.sun.xml.ws.client.WSServiceDelegate.getPort(WSServiceDelegate.java:344)
at com.sun.xml.ws.client.WSServiceDelegate.getPort(WSServiceDelegate.java:326)
at com.sun.xml.ws.client.WSServiceDelegate.getPort(WSServiceDelegate.java:308)
at javax.xml.ws.Service.getPort(Service.java:99)
at com.mycompany.myapp.webserviceclient.SomeWebserviceService.getSomeWebservicePort(SomeWebserviceService.java:58)
This sorta indicates to me that there is a bundle in my WAR\lib directory that is causing problems, but when I expand the WAR and do a search on META-INF\services I do not find anything that is close what is outlined in the exception.
I originally had CXF and it's transitive dependencies in the war\lib dir, but I've since removed that and still encounter the same error.
Nothing useful comes up when searching google and I'm at a loss here.
Does anyone know what might be going on here?
EDIT #1
Another symptom of this is that the tester pages for deployed web services will not work properly.

I had a similar problem with my glassfish3.1. Below is a stack trace
Caused by: com.sun.xml.ws.util.ServiceConfigurationError: com.sun.xml.ws.api.pipe.TransportPipeFactory: Provider com.sun.enterprise.jbi.serviceengine.bridge.transport.JBITransportPipeFactory is specified in bundle://96.0:0/META-INF/services/com.sun.xml.ws.api.pipe.TransportPipeFactory but not found
at com.sun.xml.ws.util.ServiceFinder.fail(ServiceFinder.java:245)
at com.sun.xml.ws.util.ServiceFinder.access$100(ServiceFinder.java:145)
at com.sun.xml.ws.util.ServiceFinder$LazyIterator.next(ServiceFinder.java:380)
at com.sun.xml.ws.api.pipe.TransportTubeFactory.create(TransportTubeFactory.java:133)
at com.sun.xml.ws.transport.DeferredTransportPipe.<init>(DeferredTransportPipe.java:86)
at com.sun.xml.ws.api.pipe.ClientTubeAssemblerContext.createTransportTube(ClientTubeAssemblerContext.java:315)
at com.sun.xml.ws.assembler.jaxws.TransportTubeFactory.createTube(TransportTubeFactory.java:67)
at com.sun.xml.ws.assembler.TubeCreator.createTube(TubeCreator.java:84)
at com.sun.xml.ws.assembler.TubelineAssemblerFactoryImpl$MetroTubelineAssembler.createClient(TubelineAssemblerFactoryImpl.java:130)
at com.sun.xml.ws.client.Stub.createPipeline(Stub.java:228)
at com.sun.xml.ws.client.Stub.<init>(Stub.java:205)
at com.sun.xml.ws.client.Stub.<init>(Stub.java:178)
at com.sun.xml.ws.client.sei.SEIStub.<init>(SEIStub.java:85)
at com.sun.xml.ws.client.WSServiceDelegate.createEndpointIFBaseProxy(WSServiceDelegate.java:608)
at com.sun.xml.ws.client.WSServiceDelegate.getPort(WSServiceDelegate.java:348)
at com.sun.xml.ws.client.WSServiceDelegate.getPort(WSServiceDelegate.java:330)
at com.sun.xml.ws.client.WSServiceDelegate.getPort(WSServiceDelegate.java:312)
at javax.xml.ws.Service.getPort(Service.java:134)
at org.glassfish.webservices.monitoring.WebServiceTesterServlet.initializePort(WebServiceTesterServlet.java:563)
at org.glassfish.webservices.monitoring.WebServiceTesterServlet.doGet(WebServiceTesterServlet.java:169)
at org.glassfish.webservices.monitoring.WebServiceTesterServlet.invoke(WebServiceTesterServlet.java:104)
at org.glassfish.webservices.EjbWebServiceServlet.service(EjbWebServiceServlet.java:114)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at com.sun.grizzly.http.servlet.ServletAdapter$FilterChainImpl.doFilter(ServletAdapter.java:1002)
... 20 more
The problem was traced to the conflicting mappings for "com.sun.xml.ws.api.pipe.TransportPipeFactory" inside two different modules.
prompt> grep -rl com.sun.xml.ws.api.pipe.TransportPipeFactory /usr/local/glassfish-3.1/glassfish/domains/modules*
modules/webservices-osgi.jar
modules/sun-javaee-engine.jar
prompt> grep -rl com.sun.enterprise.jbi.serviceengine.bridge.transport.JBITransportPipeFactory /usr/local/glassfish-3.1/glassfish/domains/modules/*
modules/sun-javaee-engine.jar
Clearly, the required file was sitting in the wrong jar "sun-javaee-engine.jar".
Above, "sun-javaee-engine.jar" is loaded by the "Java EE Service Engine Module". Once I removed the module (using glassfish update tool) everything started working fine.
EDIT #1:
Here is how to remove the module using the update tool on the Unix command line:
sudo ./pkg uninstall sun-javaee-engine