First time posting so hopefully I can give enough info.
We are currently using Apache ActiveMQ to setup a pretty standard Producer/Consumer queue app. Our application is hosted on various different client servers and at random times/loads we experience issues where the JMS connection permanently dies, so our producer can no longer connect to the consumer, so we have to restart the producer. We're pretty sure on the issue, that we're running our of connections on the JMS cached connection factory, so need to do a better cleanup/recycling of these connections. This is a relatively common issue described here (Our setup is pretty similar):
Is maven shade-plugin culprit for my jar to not work in server
ActiveMQ Dead Connection issue
Our difficulty is that this problem is only experienced on our application when it's deployed on servers, however we don't have access to these, as they house confidential client info, so we can't do any debugging/reproduction on the servers where the issues occur, however it's not possible so far for us to reproduce the issue on our local environment.
So in short is there any way that we could forcefully kill/corrupt our JMS connection threads so that we can reproduce and test various fixes and approaches? Unfortunately we don't have the option to add in fixes without testing/demo'ing any solutions, so replication of the issue on our local setup is our only option.
Thanks in advance
Related
I'm new to java and tomcat. I'm developing a website in java using spring mvc. It's deployed to a linux server that's running Tomcat 8. Everything works fine when I deploy, it connects to the database great. The issue is that the site seems to go idle very quickly. I haven't been able to time it exactly, but it seems like it only takes about a minute of inactivity for the entire site to go idle. Then the next request is extremely slow, loading in all my classes. I'm losing my sessions as well.
Is this a common occurrence? Does it sound like I'm doing something wrong in java? Tomcat? Both?
EDIT: In light of StuPointerException's comment, I've updated my database connection management. I'm now using Apache dbcp. I will update if this resolves the problem. I want to give my QA tester ample time to hit my site some more.
It's difficult to answer your question directly without more information about your server setup.
For what it's worth though, every time I see this kind of behaviour it's down to a mis-configured database connection pool. There can be a significant overhead in creating new database connections.
If you don't use a connection pool or you're allowing connections in the pool to die (due to missing validation queries/checks) then you will start to see performance problems over time due to connection timeouts.
I have an webapp built with Jboss Seam, which runs on cluster Jboss EAP. Webapp have a client library, which should stay connected with server for receiving events. When event arrived from client, it fires JMS message. The question is how can I achieve only one client connection per cluster(to avoid JMS message duplication) in this environment?
Possibly Clustering Singleton Services might work for your problem. See reference https://docs.jboss.org/jbossas/docs/Clustering_Guide/4/html/ch05s11.html
For a more details reference, see https://access.redhat.com/documentation/en-US/JBoss_Enterprise_Application_Platform/6/html/Development_Guide/Implement_an_HA_Singleton.html
Hope that helps
I have a Producer Consumer based application based on Netty. The basic requirement was to build a message oriented middleware (MOM)
MOM
So the MOM is based on the concept of queuing (Queuing makes systems loosely coupled and that was the basic requirement of the application).
The broker understands the MQTT protocol. We performed stress testing of the application on our local machine. These are the specs of the local machine.
We were getting great results. However, our production server is AWS Ubuntu based. So we stress tested the same application on AWS Ubuntu server. The performance was 10X poor than the local system. This is the configuration of the AWS server.
We have tried the following options to figure out where the issue is.
Initially we checked for bugs in our business logic. Did not find any.
Made the broker, client and all other dependencies same on mac as well as aws. What I mean by same dependencies is that we installed the same versions on aws as on mac.
Increased the ulimit on AWS.
Played with sysctl settings.
We were using Netty 4.1 and we had a doubt that it might be a Netty error as we do not have stable release for Netty 4.1 yet. So we even built the entire application using Netty 3.9.8 Final (Stable) and we still faced the same issue.
Increased the hardware configurations substantially of the AWS machine.
Now we have literally run out of options. The java version is the same on both machines.
So the last resort for us is to build the entire application using NodeJS but that would require a lot of effort rather than tweaking something in Netty itself. We are not searching for Java based alternatives to Netty as we think this might even be a bug in JVM NIO's native implementation on Mac and Ubuntu.
What possible options can we try further to solve this bug. Is this a Netty inherent issue. Or is this something to do with some internal implementations on Mac and Ubuntu which are different and are leading to perfomance differences as we see them ?
EDIT
The stress testing parameters are as follows.
We had 1000 clients sending 1000 messages per second (Global rate).
We ran the test for about 10 minutes to note the latency.
On the server side we have 10 consumer threads handling the messages.
We have a new instance of ChannelHandler per client.
For boss pool and worker pool required by Netty, we used the Cached Thread pool.
We have tried tuning the consumer threads but to no avail.
Edit 2
These are the profiler results provided by jvmtop for one phase of load testing.
I am trying to run down a problem with consumer connections to RabbitMQ being dropped in our production environment. This problem seems to occur after running for a few days and by restarting our application it seems to connect and work fine for a few more days. My guess is that there is a period of inactivity that is causing the issue. It seems the AMQP heartbeat was designed just for this problem. We are using spring-amqp 1.3.2.RELEASE and setting the requestedHeartbeat on the ConnectionFactory to 10 however we are still seeing connections drop.
The spring-amqp client will reconnect if I completely disconnect from the internet and reconnect, or block the connection with a firewall, however it does not even seem to throw an Exception in the log when this happens in production. Of course that may be because we are using slf4j and logback for our logging mechanism and spring is using commons logging, so it is appearing in System.out and not going to the log. I have added the jcf-over-slf4j bridge to fix that but have not rolled it out yet so I do not have a stack trace to contribute.
One more piece of info about our architecture: we have HA proxy in front of RabbitMQ.
I would like to somehow put the app in debug and run within eclipse to see if the heartbeats are actually going out. I tried to verify with Wireshark but our traffic has two way SSL encryption and I haven't been able to decrypt the traffic yet.
Does anyone have any suggestions? I have been trying to run this down for weeks and I'm running out of ideas. I would greatly appreciate your input.
Thanks!
on 11-Feb-2015 days ago rabbit released 3.4.4 which has support for automatic reconnections. You could roll your own solution like we did a while back but it seems easier to just upgrade to the newest version of rabbit.
https://github.com/rabbitmq/rabbitmq-dotnet-client/blob/rabbitmq_v3_4_4/projects/client/RabbitMQ.Client/src/client/impl/AutorecoveringConnection.cs
If you have rabbitmq's autorecovery enabled, Spring AMQP prior to 1.4 is not compatible with it; the problem being that rabbit restores the connections/channels, but Spring AMQP doesn't know about them. Generally, though, this just causes extra connections/channels - Spring AMQP just establishes new channels. I have not heard of it causing the problems you describe.
If you can't figure it out by fixing your logging configuration, another thing to try is to jstack your process to see what the threads are doing. But you should be able to figure it out from the logs.
I'm working with a a framework that has a Core application, and secondary applications that all communicate using JMS via ActiveMQ (through Camel). It all seems to work fine on Windows, but the moment I moved it to our CentOS environment it failed. Let me note that our guy who programmed it and who was our ActiveMQ guy has left so I don't know quite how to diagnose the problem. It seems to be establishing a connection but then does nothing else. It is supposed to begin an exchange of messages but it doesn't. When I set logging to DEBUG I get messages saying "urlList connectionList:[]" and "waiting x ms before attempting connection" as though it's not connecting. I've made sure there's no firewall, there's no security policy to block it, ActiveMQ is shown to be running. I've tried everything I can think of but I have no idea what the problem could be. Any recommendations?
try testing ActiveMQ via the web console to make sure it's functioning properly. next, try connecting remotely via JMX and verify that works as well. It's likely an environment issue that will be difficult to diagnose via the info given...
see these pages for more information...
http://activemq.apache.org/web-console.html
http://activemq.apache.org/jmx.html