I have a central load balancing server and several application servers running on Apache Tomcat. The load balancing server receives request and forwards them to the application servers in round robin fashion. If one these application servers goes down, the load balancing server should stop forwarding requests to it.
My current solution for this is to ping the application servers every few minutes and if I don't receive a response, remove them from a list of available servers. Is there a better way to monitor the status of these servers? Should I ping more often or should the application servers constantly inform the load balancing server?
Execute a null transaction on it regularly. Pinging really isn't enough, it only exercises the TCP/IP stack, and I have seen operating systems in states where TCP/IP was up but no applications and not even part of the OS stack itself. Executing a transaction exercises everything. Include the database in the null transaction.
First ensure your server isn DDOS attrack protected , if the depends on you application connection avg time edit keep alive time
Then you should study about precock mpm , i think it will give you best solution
Related
I have a Windows application (written in Java) that connects to the server (spring + httpd) via websockets. Another websocket connection is established as soon as a user authenticates to the same server from a web browser. If server notice that both clients have same IP address, it "pairs" them so both applications can talk with each other.
The challenge I'm facing currently is that when multiple Windows application are starting up, all of them establish new websocket connections that exceeds httpd limitation of 255 active connections and the server goes down.
I'm looking for some feasible solution that would not overwhelm the server. A perfect scenario: a user logs into the system using a web browser, the server tries to connect the Windows application running on a clients machine afterwards and everyone is happy.
Have you any idea how to achieve it?
What I've tried already is to not create a new websocket connection on the Windows application startup but send a GET request to the request to the server and wait for the response that will occur after authenticating a user from a web browser. Hanging GET requests still need resources and httpd keeps a separate process for each of them. Also, it turned out that httpd has a 5 minutes timeout for hanging requests and sends 502 back after reaching it out.
I thought that maybe it is possible to handle GET requests in Spring by only one process / thread, but I haven't found any info for that.
Another limitation worth noting is that the Windows application runs on customer machines and customer's security policy may not allow for any wise tricks.
I am using Elasticsearch 1.5.1 and Tomcat 7. Web application creates a TCP client instance as Singleton during server startup through Spring Framework.
Just noticed that I failed to close the client during server shutdown.
Through analysis on various tools like VisualVm, JConsole, MAT in Eclipse, it is evident that threads created by the elasticsearch client are live even after server(tomcat) shutdown.
Note: after introducing client.close() via Context Listener destroy methods, the threads are killed gracefully.
But my query here is,
how to check the memory occupied by these live threads?
Memory leak impact due to this thread?
We have got few Out of memory:Perm gen errors in PROD. This might be a reason but still I would like to measure and provide stats for this.
Any suggestions/help please.
Typically clients run in a different process than the services they communicate with. For example, I can open a web page in a web browser, and then shutdown the webserver, and the client will remain open.
This has to do with the underlying design choices of TCP/IP. Glossing over the details, under most cases a client only detects it's server is gone during the next request to the server. (Again generally speaking) it does not continually poll the server to see if it is alive, nor does the server generally send a "please disconnect" message on shutting down.
The reason that clients don't generally poll servers is because it allows the server to handle more clients. With a polling approach, the server is limited by the number of clients running, but without a polling approach, it is limited by the number of clients actively communicating. This allows it to support more clients because many of the running clients aren't actively communicating.
The reason that servers typically don't send an "I'm shutting down" message is because many times the server goes down uncontrollably (power outage, operating system crash, fire, short circuit, etc) This means that an protocol which requires such a message will leave the clients in a corrupt state if the server goes down in an uncontrolled manner.
So losing a connection is really a function of a failed request to the server. The client will still typically be running until it makes the next attempt to do something.
Likewise, opening a connection to a server often does nothing most of the time too. To validate that you really have a working connection to a server, you must ask it for some data and get a reply. Most protocols do this automatically to simplify the logic; but, if you ever write your own service, if you don't ask for data from the server, even if the API says you have a good "connection", you might not. The API can report back a good "connections" when you have all the stuff configured on your machine successfully. To really know if it works 100% on the other machine, you need to ask for data (and get it).
Finally servers sometimes lose their clients, but because they don't waste bandwidth chattering with clients just to see if they are there, often the servers will put a "timeout" on the client connection. Basically if the server doesn't hear from the client in 10 minutes (or the configured value) then it closes the cached connection information for the client (recreating the connection information as necessary if the client comes back).
From your description it is not clear which of the scenarios you might be seeing, but hopefully this general knowledge will help you understand why after closing one side of a connection, the other side of a connection might still think it is open for a while.
There are ways to configure the network connection to report closures more immediately, but I would avoid using them, unless you are willing to lose a lot of your network bandwidth to keep-alive messages and don't want your servers to respond as quickly as they could.
I'm attempting to use JMeter with some custom samplers to load test a Java application that is normally accessed via Weblogic RMI over T3 from a Swing-based GUI. Intention is to load the application server and measure the response time of particular transactions by simulating many concurrent user connections/interactions (up to ~500).
I have implemented a couple of JMeter samplers that acquire a RMI connection to the server via a JNDI lookup and that works fine. However I've noticed that, even if I acquire two contexts on different threads using different credentials, only one T3 connection is opened.
Is there a way to effectively create multiple independent connections to the app server from within one JVM, or will I be forced to run one user per JVM?
App is running in WLS 11g, currently on Hotspot 32bit but will be moving to JRockit 64bit.
Thanks.
You are running up against RMI connection pooling. There are ways to turn it down, see the RMI Home Page and the Properties pages linked from them, but it's still an unrealistic test for other reasons such as port exhaustion on the client host. You should really look at using as many client hosts as possible with as many separate JVMs as possible.
Given the following scenario, please suggest me a way to implement memcache in my application.
Currently, I have 10 webservers in which the same application is being run and a load balancer to decide upon to which web server the request be sent.
On each webserver, I am maintaining a local cache i.e. there is some class XYZ which controls the MySQL table xyz and this class has initialize method
which will warm up the local cache.
Now, suppose the webservers are X,Y,Z. The load balancer sends a request to X and this request adds some values to db & updates the cache. Again the same request was sent by the load balancer to Y. But since server Y doesnot have the value in the cache, it hits the database.
So, given this scenario, how should I implement memcache in my application so that I could minimize db hits.
Should I have a separate memcache server and all the other 10 webservers will get the cached data from this memcacher server?
One work around (not ideal though), would be to implement sticky session on the load balancer so that request from one user always go through to the same server (for the duration of their session). This doesn't help much if a server dies or you need cached data shared between sessions (but it is easy and quick to do if your load balancer supports it).
Otherwise the better solution is to use something like memcached (or membase if your feeling adventurous). Memcached can either be deployed on each or your servers or on separate servers (use multiple servers to avoid the problem of one servers dying and taking your cache with it). Then on each of your application servers specify in your memcached client connection details for all of the memcached servers (put them in the same order on each server and use a consistent hashing algorithm in the memcached client options to determine on which server(s) the cache key will go).
In short - you now have a working memcached set-up (as long as you use it sensibly inside your application)
There are plenty of memcached tutorials out there that can help with the finer points of doing all this but hopefully my post will give you some general direction.
I would like to inform all logged in users that the server will shutdown. This special interest would be nice in an ajaxfy application (RIA).
What are the possible solutions? What are the best practice solutions?
There were two possible end-scenarios:
Send a text $x to the server ergo to all users. ("The server will not be available for some minutes.")
Send a key $y to the server which will used to generate a (custom) text to all users. ("SERVER_SHUTDOWN")
Environment: Tomcat (6/7), Spring 3+
Messaging to users: with polling or pseudo-pushing via an async servlet.
Ideas
1. Context.destroy(): Implementing a custom ContextListener's destroy
I don't think it is a good solution to block within a "destroy()" -- blocking, because we should wait about 5-10 seconds to make sure that all logged in users receive a message.
2. JMX Beans
This would mean, that any server service operation (start, stop) have to invoke a special program which sends the message.
3. Any other messaging queues like AMQP or ActiveMQ
Like 2.
Unless the server shuts down regularly and the shutdown has a significant impact on users (for e.g. they will lose any unsubmitted work - think halfway through editing a big post on a page) then notifying of server shutdown won't really be of much benefit.
There are a couple of things you could do.
First, if the server is going to be shutdown due to planned maintenance then you could include a message on web pages like;
Server will be unavailable Monday 22nd Aug 9pm - 6am for planned
maintenance. Contact knalli#example.com for more information.
Second, before shutting down the server, redirect requests to a static holding page (just change your web server config). This holding page should have information on why the server is down and when it will be available again.
With both options, its also important to plan server downtime. It's normal to have maintenance windows outside of normal working hours. Alternatively, if you have more than one server you can cluster them. This allows you to take individual servers out of the cluster to perform maintenance without having any server downtime at all.