We are running tomcat application server that handle over 100 concurrent sessions.
In the last 2 months the most active users noticed that sometimes they are getting kicked off from the system.
As I understood from the log tomcat sessions are getting expired without any reason.
I see no problems from the web application side.
Is there any problem from the tomcat side?
Tomcat 6.0.18.
If there has been no code change triggering this, I would look at memory usage. This might be the effect of Tomcat running out of memory and invalidating sessions to recover.
If at all possible monitor garbage collections, and/or put surveillance on with jconsole Java Flight Recorder or jvisualvm.
A possible cause is that you put in the session an object that does NOT implement the Serializable interface. Tomcat occasionally writes some of the sessions on the disk. If a session contains non-serializable objects it will simply be removed from the container (because of the NotSerializableException). If this is happening you should see the Exception in the tomcat log file.
I would increase the monitoring of the server in general and the sessions specifically.
A good monitoring application is lambda probe - it allows you to view the current sessions and their data. I would also add a HttpSessionListener to log session creation and destruction.
Edit
Is is possible that you add some non serializble objects to the session and Tomcat fails to passivate them to the disk?
Edit 2
Lambda probe seems to be dead, and there a much better fork of the project over at http://code.google.com/p/psi-probe/
there is a timeout, that you can configure in your web.xml:
<web-app>
...
<session-config>
<session-timeout>-1</session-timeout>
</session-config>
</web-app>
use -1 for no timeout
Increase your logging of sessions, which may shed some light onto your problem.
The Tomcat config page's Logging in Tomcat includes an example of increasing logging of sessions.
We just ran into this with tomcat 6_0_18 and ibm 1.5 jvm
turns out it was an ibm jvm issue with atomic operations.
There is a fix in tomcats greater than 6_0_19 to handle it.
It also doesn't occur in sun 1.5 jvm
here are some more details
tomcat bugzilla case
I have seen similar problems when the following prerequisites exist:
multiple instances of the tomcat app are installed across multiple JVMs
load balancing (between the web server and the Tomcat JVMs) is not configured correctly.
Tomcat's session replication feature is not enabled
Due to the incorrect load balancing config, the web server may randomly decide to break session affinity and send an incoming request to a Tomcat JVM that has never seen the session before. The Tomcat JVM will issue a new session and the user will lose all of their previouS session data and effectively start over.
You can search Tomcat's bug database, but it would be better to have another look at your web application first. The chances that there is something wrong with Tomcat are very low.
Try to investigate what causes session invalidation. Are you using filters? Do you have cross-context requests? Try adding logging information for every request to find out, when exactly the session is lost.
I had a similar symptom (but more consistent). Users would sign in ok, go to the dashboard ok, then loose the session.
I realized I had done a Java 16 to 18 upgrade and had not recompiled everything. Cleaned (mvn clean) and recompiled the code for ALL my .jars/.war.
Problem solved, for me.
I hope this helps someone.
Eventhough I do not know the cause of the problem, one possible fix (which I had done at my previous project) would be to run the application on a tomcat cluster and have session failover. Sessions can be by default sticky and when one node goes down, the healthy nodes pick up the sessions and all this is transparent to the end user.
Related
I am trying to understand the tomcat's (version 8.0.21) session persistence after I execute kill -9 . When I start my tomcat ( using startup.sh), I observe (randomly) http sessions of my web app ( which were created before the kill command execution) still being maintained.
I understand this session persistence across restart, if I execute shutdown.sh to stop the tomcat and then bring the tomcat up again. My understanding of kill -9 is some what similar to 'power-off'. So my question is :
Whether the standard manager implementation gets invoked just after kill -9, which tries to persist session before termination.
Or I just get the previous session because my tomcat may have serialized the few session, (somewhere into its temp directory) while it was up earlier (before execution of kill -9).
The Tomcat session manager documentation has lots to say on the topic. Here's a few relevant gems:
Tomcat provides two standard implementations of Manager for use — the default one stores active sessions, while the optional one stores active sessions that have been swapped out (in addition to saving sessions across a restart of Tomcat) in a storage location...
...
Whenever Apache Tomcat is shut down normally and restarted, or when an application reload is triggered, the standard Manager implementation will attempt to serialize all currently active sessions...
Emphasis mine - I think it's important to note that active sessions get serialized when you reload applications. If you've reloaded the application recently, and then do a kill -9 (which, as correctly noted in the comments, gives Tomcat no warning and no time to clean anything up, and thus should only be done as a last resort), then you will indeed see sessions persisting across the restart.
It is also possible that you may have the persistent storage manager configured:
In addition to the usual operations of creating and deleting sessions, a PersistentManager has the capability to swap active (but idle) sessions out to a persistent storage mechanism, as well as to save all sessions across a normal restart of Tomcat.
Though I will say that unless you did this configuration, or this install of Tomcat was preconfigured for you, this would not be the case.
I've got a system with multiple jboss (4.2.3) servers behind multiple apache servers.
In the situation where one of the jboss servers goes down we still get users with sessions for those servers coming in, however they "bounce" around the servers due to the application server ID in the JSESSIONID, causing apache to send the request to random servers for each request, resulting in session timeout errors.
My initial thought was to create an interceptor in the application to invalidate the session if we detect that the session is actually for a different server (i.e. session.invalidate())
But that doesn't cause a new JSESSIONID to be generated (even with the fix described here https://issues.jboss.org/browse/JBAS-4436)
My next idea was to create a valve to do the same job but on the Request instead of the HttpServletRequest, however I can't find the library to add ValveBase to jboss 4.2.3 (and I'm not even certain that valves are supported by 4.2.3)
Is there either:
a) Anything I've missed in getting one of my 2 ideas above to work
or
b) Any better ideas I haven't thought of for solving the problem ? (I would like to cluster the sessions but can't at the moment due to infrastructure problems)
Thanks.
Use Session Stickyness in Apache so it won't switch to other application server until it dies.
My tomcat application randomly invalidates the session information and the user if forced to log back in again. This only happens when the webapp is deployed with other specific web applications. When it is run by itself with no other app running, it works fine. There is a conflict somewhere, but I am unable to find it.
The webapp in question uses both the Spring and Hibernate frameworks. The other webapps do as well. Here is an example of what happens.
When I first log on, everything is fine.
JSESSIONID Matches
The JSESSONID matches in both the cookie and the management console. However, when using the application, it randomly terminates the session.
Session gone in Tomcat Console
When I another web request, the login page is shown and a new JSESSIONID has been generated.
From the research I have done, the JSESSIONID is generated whenever a JSP file is called and there is no current session. I don't understand why the session is being invalidated, but only when it is run alongside other applications. I have checked the memory usage, and I didn't notice a correlation between a memory drop and the sessions being invalidated. I guess that means that Tomcat isn't clearing sessions to free up memory.
Any help is much appreciated
Check that session.setMaxInactiveInterval() is not programmatically configured in the other web applications. This would elicit the behavior you describe.
This method is slightly different from the session timeout value that you would find in your web.xml files. (Here's a good rundown.)
the following steps may help you:
Are you sure that you did not override the default session time out value, Please check out your session time-out at runtime.
from tomcat manager page (~/manager/html/list) see the status of your web application sessions and press on the value of your app sessions column.
I'm debugging an issue where calls to our weblogic server often return with a new JSESSIONID value, replacing the JSESSION that was originally sent.
The issue occurs frequently, but not consistently. (Ie., when it affects you, it affects ~90% of your traffic.)
We've shown that the issue appears to be unrelated to the actual server call being made, which makes it hard for me to debug any specific method.
Is there a way to hook into the JSESSION creation in WebLogic, to see what is causing the session to be created / destroyed / invalidated?
Attaching a remote debugger would be great, but I'd settle for some form of logging that might point me in the right direction?
This is a standard part of the servlet spec (not WebLogic specific). You want the HttpSessionListener interface.
Note that the Servlet session stuff is really bad - the kind of behavior you're seeing can happen when memory problems kick in, etc. For a deconstruction, check out the presentation at SeaJUG (Rethinking Users and User Session Management). Slides and video are posted.
Nodemanager/WLST is something that is also useful in debugging and monitoring servers with Weblogic Admin, Enterprise Fusion Middleware Control, or Cloud Control.
I was wondering if there is a 'smooth way' of redeploying a Java WAR to a production server (no cluster, no OSGi)?
All I can come up with is stop server, update file, restart server. And 10 minutes beforehand I need to display a maintenance warning on the site.
What's your approach?
First, hot-deploy doesn't always work. We spent so much time to make sure every new module is loaded and decided it's not worth the trouble. So what you are doing may sound bad but it's the most reliable way to deploy a new WAR.
Our current approach is to use a switch with load-balancer in front of all servers. We run at least 2 instances of the application servers. When we shutdown one server for maintenance, the traffic automatically goes to the other one.
Some of the switches are really inexpensive. If you don't have enough load to justify a new box and your 2 instances can run on the same box.
In some circumstances, the switches can actually save money. For example, we have a SSL page that used to use 6 boxes and now it runs fine on 2 boxes with SSL acceleration in the switch.
You might have a look at JRebel, though I wouldn't use it in production. In production we do basically the same, though our boss keeps on dreaming of hot redeploys. Unfortunately they are supported mostly on paper - in most complex applications something always goes wrong with hot redeploys. The same is even truer for incremental hot redeploys...
Some application servers do support redeployment without interruption of service. This is at least true for WebLogic, see Using Production Redeployment to Update Applications. Note that this is not hot deploy (and I would NEVER use hot deploy with production servers).
Without application server support, I'm afraid you won't be able to do real "smooth" redeployments. If you want to minimize downtime, one approach is to deploy the new application in parallel (on the same server or another one) and to change the routing rules when done. But clients will loose their session.
Usually it's possible to optimize start-up time. Our web application starts with Jetty in 5-7 seconds. Other Java web servers are worse, because they start very slow.
Also, as I'm aware (not I did it), the front-end web server (such as apache, we use lighttpd) could be configured to hold request some period of time (up to 30 seconds on ours) while the Jetty is not ready. So, we just easily restart the Jetty while deploying, and users just have several seconds delay in worst case, which usually just looks like an Internet connection glitch.
Usually, mv old.war new.war and let the AS take it from there, although with very busy 24/7 services, I imagine this wouldn't be an option.
As ZZ Coder has already mentioned load balancer is a good solution especially for big deployments. For my own project I use reverse http proxy functionality of nginx web server. It redirects all http packets from a specified web context (seen from the Internet) to a server inside my network. Configuration is really easy:
location /best-app-ever/ {
proxy_pass host-address:8080/some-app-1.1
root /home/www/some-app-1.1
}
Switching version should be smooth as well. Assuming that you have already deployed new version of application just change nginx configuration file and apply changes:
location /best-app-ever/ {
proxy_pass host-address:8080/some-app-1.2
root /home/www/some-app-1.2
}
sudo nginx -t
sudo service nginx restart
Be warned that in case your web application is stateful and/or contains some running or scheduled processes, the deployment and undeployment might not be as smooth.