We are a Microsoft shop and use C# exclusively. We have a Windows Service that runs and sends WCF SOAP requests to one of our production application servers. Those application servers in turn make a call out to a 3rd party SOAP endpoint. In our Windows Service logs we started noticing some java.util.ConcurrentModificationException exceptions that seem to succeed on a retry. java.util.ConcurrentModificationException is a Java language exception, so my initial thought was that one of the 3rd party SOAP endpoints that we call was written in Java and was returning that exception. We then wrapped all calls to 3rd party services in a try-catch, and throw a new wrapper exception with a description of which endpoint was being called when the exception occurred, along with adding the inner exception to it.
However, after doing this we are still seeing the same exception being thrown without the outer wrapper exception on it, so it is not happening when calling a 3rd party service. The only other thing our C# code does is read from a SQL database using the System.Data assembly, which I'm fairly confident is written in C# (or C++) and not Java.
The exception stack trace ends at the Windows Service's function to call the application server; We don't see any of the application server code in the stack trace of the exception caught on the Windows Service. Also, this error seems to happen intermittently, I'm guessing when there is lots of traffic, so it's not easy to reproduce. Out of thousands of requests a day, we might see it happen 50 times.
So I'm kind of at a loss from where this exception is coming from. It's possible that our production load balancer (F5) and/or firewall product (Cisco ASA) are written in java, but I wouldn't expect their exception details to leak into the exception I'm seeing; I would instead expect a regular C# "connection failed/lost" exception.
Our application servers are Windows 2012 R2 using IIS 8, and we are running ASP.Net WCF SOAP services.
Any thoughts/ideas are appreciated. Thanks.
Update
It turns out that we had some code that was actually unwrapping the original exception and throwing it back with a truncated stack trace. So the problem was originating from calling a 3rd party service that is written in Java after all; we were just doing something dumb with our error handling. Mystery solved.
Related
I have a very unusual error condition that I can't seem to find the solution to. I'm hoping someone out here in StackOverflowland can help. Before you just look at the title and say "your CLASSPATH is wrong," read on!
I work with a server-based Java application that utilizes a number of third-party libraries. The particular library that I'm getting the error with is a file transfer library. Part of the functionality of the application allows the user to connect to remote servers (FTP, SFTP and FTPS) and send/receive files.
One of our customers is using the application and occasionally gets a "Class Not Found Exception" (or more specifically, "No Class Definition Found") when attempting to connect to a remote host. What's weird is that they don't get it all the time. They have sent me logs that show a successful connection and then another one which shows this error. There was no restart of the server between the two and both connections are to the same host using the same connection settings.
What would cause a "Class Not Found Exception" to occur occasionally within a Java application on a class that has already been successfully used? I've contacted the third-party vendor and they are just as puzzled.
I wasn't able to determine anything from the stack trace, but since it was requested, here it is:
java.lang.NoClassDefFoundError: com.enterprisedt.net.j2ssh.configuration.ConfigurationLoader (initialization failure)
at java.lang.J9VMInternals.initialize(J9VMInternals.java:139)
at com.enterprisedt.net.j2ssh.SshThread.a(SshThread.java:93)
at com.enterprisedt.net.j2ssh.SshThread.<init>(SshThread.java:73)
at com.enterprisedt.net.j2ssh.transport.TransportProtocolCommon.startTransportProtocol(TransportProtocolCommon.java:515)
at com.enterprisedt.net.j2ssh.SshClient.connect(SshClient.java:593)
at com.enterprisedt.net.ftp.ssh.SCPClient.connectSSH(SCPClient.java:1137)
at com.enterprisedt.net.ftp.ssh.SSHFTPClient.connect(SSHFTPClient.java:920)
at com.enterprisedt.net.ftp.async.internal.ConnectTask.connect(ConnectTask.java:154)
at com.enterprisedt.net.ftp.async.internal.ConnectTask.run(ConnectTask.java:216)
at com.enterprisedt.net.ftp.async.internal.FTPTaskProcessor$b.run(FTPTaskProcessor.java:590)
I have a Java-based client that receives data from a Tomcat 6.0.24 server webapp via JAX-WS. I recently upgraded the server with new functionality that can take a long time (over 30 seconds) to run for certain inputs.
It turns out that for these long operations, some kind of timeout is occurring. The client gets an HTTP 400 Bad Request error, and shortly afterwards (according to my log timestamps, at least) the server reports a Broken Pipe.
Here's the client's error message:
com.sun.xml.internal.ws.client.ClientTransportException: The server sent HTTP status code 400: Bad Request
And the server's:
javax.xml.ws.WebServiceException: javax.xml.stream.XMLStreamException: ClientAbortException: java.net.SocketException: Broken pipe
I've tried experimenting with adding timeout settings on the service's BindingProvider, but that doesn't seem to change anything. The default timeout is supposed to be infinite, right?
I don't know if it's relevant, but it might be worth noting that the client is an OSGI bundle running in a Karaf OSGI framework.
Bottom line, I have no idea what's going on here. Note that the new functionality does work when it doesn't have to run for too long. Also note that the size of the new functionality's response is not any larger than usual - it just takes longer to calculate.
In the end, the problem was caused by some sort of anti-DoS measure on the server's public gateway. Unfortunately, the IT department refused to fix it, forcing me to switch to polling-based communication. Oh well.
I have simple client-server application using RMI. Server side is supposed to generate some sort of PDF (itext) file, and client should display it.
While trying to generate my pdf on server side by invoking proper method of my own common remote interface, ClassNotFoundException is thrown on client side.
Caused by: java.lang.ClassNotFoundException: com.itextpdf.text.ExceptionConverter (no security manager: RMI class loader disabled)
I thought, that using RMI would be like using black box - client invokes method, and receives response, without knowlage of how it is done, to the only common class for both sides would be remote interface. More over i thought that this will allow me to shrink required dependencies on client side to minimum.
Is it really mandatory for client to have all server's libraries in its classpath to work?
EDIT:
Another strange fact is that there are no exceptions cought and logged on server side.
If I add server side project as dependency to client, that everything works fine.
Just a guess here, it's probably not catching and handling ExceptionConverter on the server side properly and it's propagating to the client which has no definition for it.
If you enable RMI-classloader AND set the appropriate system properties (google for: "java.rmi.server.codebase") then RMI will dynamically load the missing class from server to client - and your problem will go away. No need to store and statically link server libraries on client side.
As far as I know the RMI-classloader tutorial describes this all in detail.
EDIT:
I have given the hint to the great RMI-feature of dynamic classloading. But I would say in your special case it is probably not a good idea to try to propagate server classes to clients. Imagine there is another client like a JMX-browser in parallel which might get problems with unknown classes. Better: Catch the itext-Exception on server side just before sending the result of RMI-invocation and wrap it in another well-known exception like for example IllegalStateException by only using the error message but not the stack trace (you should log it anyway on the server-side).
My idea for exception handling on server-side is (pseudo-code):
try {
// your server code
} catch (Throwable ex) {
// a) log error completely including cause (stacktrace) with tools like Log4J etc.
// b) throw new RemoteException(error.getMessage()); // without cause!
}
Ok I found the solution.
It appears that I thought right - client does not have to know anything about what is under the hood of server implementation.
The key to everything, was static initialization block in one of our classes - resource used there could not beed resolved. As it was in static block, IOException was not cought and Error was generated insteed of exception. After changing all catches from Excetpion to Throwable server did log proper stacktrace with the root cause (this was not included in stacktrace logged by client). Fixing it was just a matter of secodns.
Thank you for your time!
I've got a RESTful Web-Service that does some IO and database activity in order to return a result.
There are some Exceptions that I would like to see, ie exception thrown because an expected variable is not set, or my database is no longer exists, etc...
However when I throw these exceptions they get caught and wrapped in a HTTP 500 (Internal Server Error) exception and that is what the client gets (As opposed to the root exception).
I would like to see my exception here, instead of going to have to look through the Application server logs.
You can implement an ExceptionMapper that returns anything you want. So, for example, it can put the whole exception with the stacktrace to the response.
Is it what you want at client side? I'm not sure.
It can be quite a difficult job for a client to rebuild the exceptions and rethrow it.
In addition, it's quite unsecure, since the stacktrace will expose the server's internal code to the client.
I suggest you declare some server error codes that make sense to client and that you want to expose. Example:
1000 - connection to database failed.
2000 - something else
So for specific exceptions, in the ExceptionMapper you can map the server exceptions to error codes and then remap them at the client side.
I have a setup in which some applications communicate with each other via Tibco rendezvous. The applications communicate using certified messaging. My problem is that two of my receivers have recently started exhibiting the behavior that they will get an Error 27, Not Permitted when they want to confirm a message (the first message in a certified message exchange isn't certified, we've accounted for that).
I've been looking around the internet to find people with the same error, and I have found many, but they all get the error when trying to create the tibco transport. I can create the transport just fine, but I can't confirm any messages received over it.
Our environment uses both tibco 7.X and 8.X, some times intermingled. This problem appears both when the peers use the same tibco version and when they use different versions. It doesn't show up for all applications, but when it does show up for an application, it remains "broken". Discarding the ledger files for both sender and receiver does nothing. We still get the error. Both sender and receiver have proper permissions to write to (and create the) ledger files. We are connecting to permanently running rvds. The sender and receiver are on different machines. Communication has worked flawlessly in the past, but at some point, it stopped doing so. The application is in java, and we're using the tibrvj.jar auto-native libraries.
The error is
...
Caused by: TibrvException[error=27,message=Not permitted]
at com.tibco.tibrv.TibrvImplCmTPortC.natConfirmMsg(Native Method)
at com.tibco.tibrv.TibrvImplCmTPortC.confirmMsg(TibrvImplCmTPortC.java:304)
at com.tibco.tibrv.TibrvCmListener.confirmMsg(TibrvCmListener.java:88)
....
I know you're going to ask me "what did you do to make it start happening", and my response is "I don't know".
Any input would be appreciated.
Thanks.
It may be possible that TCP connections between the two RVD servers is not possible. Can you check if you can connect from one to the other (connect from the subscriber host back to the publisher)? In my experience, CM acknowledgments are handled over TCP (please take this with a grain of salt as I'm more an end user than a Middleware support guy).
As it turns out, it was a screw-up on the application level.
Due to some old code lying around, after having updated a dependency (our messaging layer), we had moved from an application level confirmation to a container level confirmation, but we had forgotten to remove an explicit message confirmation in the application code.
To summarize: We tried to confirm the message twice, and the second time it threw this exception.
I recently encountered the same exception - application had been working for months, suddenly was throwing exception. In my case some maintenance had been done on the Windows server the application ran on and directories had been marked read-only. Once that was cleared the exception went away.
Discovered this after trouble-shooting hours worth of other potential causes.
Just my two cents: This exception also occurs when you try to explicitly confirm message on non-CM transport.