Here is the thing,
I'm creating an SQS Connection. I'm using the same connection to create consumers to listen to two different queues(Q1, Q2).
Enabling and disabling to queue is handled by the Admin user of application through a UI.
So, Whenever I disable Q1 consumer, I shouldn't close the connection, and close the connection only when both Q1 & Q2 Consumers are disabled, I can't afford to write complex code to check if both consumers are disabled.
Is there a way to check idle time of an open SQSConnection.
or
I would like to know the cost of keeping an SQSConnection open all the time
or
How about opening two different connections
here is how I'm creating the connection
SQSConnectionFactory connectionFactory = new SQSConnectionFactory(
new ProviderConfiguration(), ((AmazonSQSClientBuilder)
AmazonSQSClientBuilder.standard().withRegion(sqsRegion)).
withCredentials(
_getCredentialsProvider(awsSecretKey, awsAccessKey)));
_connection = connectionFactory.createConnection();
The entire question, here, seems premised on the unfortunate name SQSConnectionFactory, which isn't what this really is. A more accurate name might have been something like SQSConfiguredClientFactory.
None of the createConnection methods set-up the physical connection to SQS
https://github.com/awslabs/amazon-sqs-java-messaging-lib/blob/master/src/main/java/com/amazon/sqs/javamessaging/SQSConnectionFactory.java
...because SQS doesn't actually use established/continuous "connections."
The service API interactions take place over HTTPS, with transient connections being created, kept alive, and destroyed as other methods (e.g. receiveMessage(queueUrl)) need them.
So with regard to your questions: 1. connections are not left "open" in any meaningful/relevant sense, so there is nothing to check; 2. the only cost comes from actually using the connections to send/receive/delete messages; and 3. this seems unnecessary for the reasons indicated above.
Related
I have a Spring Boot (v2.0.8) application which makes use of a HikariCP (v2.7.9) Pool (connecting to MariaDB) configured with:
minimumIdle: 1
maximumPoolSize: 10
leakDetectionThreshold: 30000
The issue is that our production component, once every few weeks, is repeatedly throwing SQLTransientConnectionException " Connection is not available, request timed out after 30000ms...". The issue is that it never recovers from this and consistently throws the exception. A restart of the componnent is therefore required.
From looking at the HikariPool source code, it would seem that this is happening because every time it is calling connectionBag.borrow(timeout, MILLISECONDS) the poolEntry is null and hence throws the timeout Exception. For it to be null, the connection pool must have no free entries i.e. all PoolEntry in the sharedList are marked IN_USE.
I am not sure why the component would not recover from this since eventually I would expect a PoolEntry to be marked NOT_IN_USE and this would break the repeated Exceptions.
Possible scenarios I can think of:
All entries are IN_USE and the DB goes down temporarily. I would expect Exceptions to be thrown for the in-flight queries. Perhaps at this point the PoolEntry status is never reset and therefore is stuck at IN_USE. In this case I would have thought if an Exception is thrown the status is changed so that the connection can cleared from the pool. Can anyone confirm if this is the case?
A flood of REST requests are made to the component which in turn require DB queries to be executed. This fills the connection pool and therefore subsequent requests timeout waiting for previous requests to complete. This makes sense however I would expect the component to recover once the requests complete, which it is not.
Does anyone have an idea of what might be the issue here? I have tried configuring the various timeouts that are in the Hikari documentation but have had no luck diagnosing / resolving this issue. Any help would be appreciated.
Thanks!
Scenario 2 is most likely what is happening. I ran into the same issue when using it with cloud dataflow and receiving a large amount of connection requests. The only solution I found was to play with the config to find a combination that worked for my use case.
I'll leave you my code that works for 50-100 requests per second and wish you luck.
private static DataSource pool;
final HikariConfig config = new HikariConfig();
config.setMinimumIdle(5);
config.setMaximumPoolSize(50);
config.setConnectionTimeout(10000);
config.setIdleTimeout(600000);
config.setMaxLifetime(1800000);
config.setJdbcUrl(JDBC_URL);
config.setUsername(JDBC_USER);
config.setPassword(JDBC_PASS);
pool = new HikariDataSource(config);
Given the following code :
public void doSomething(Channel channel1){
DefaultConsumer consumer1 = new DefaultConsumer(channel1).
channel1.basicConsume("Queue1Name",..., "Consumer1Tag", consumer1)
}
If channl1 get's disconnected for some reason, which will cause messages to not be consumed(?) :
Can the same consumer instanse(consumer1) be used with a different channel in some way or is a new instance and a new channel required?
Will a redecleration of Queue1Name be aware of any changes? i.e -
DeclareOk queueResponse = someOtherChannel.queueDeclare("Queue1Name", ........);
int consumerCount = queueResponse.getConsumerCount()
What will be the consumer count?
The reason for this question is that i have a background thread running and checking connection to RabbitMQ once in a while. I check that channels and connections are open, and in case they are not - i reset them. I want to keep a single consumer on Queue1 and still be able to handle a scenario when i find it's channel is closed.
If channel1 gets disconnected it's messages won't be consumed, that is correct. They will remain in the Queue.
Yes you can reconnect to the same Queue, the messages left behind will have to be handled before you can receive the new messages though.
I think you need to remove the Queue before you can make a new one with the same name (I'm not 100% sure). Consumer count should be 1 after you reconnect. But if you do that, any messages left in the Queue will be lost.
You should really try this out yourself in your application and observe behaviour.
I'd like to listen on a websocket using akka streams. That is, I'd like to treat it as nothing but a Source.
However, all official examples treat the websocket connection as a Flow.
My current approach is using the websocketClientFlow in combination with a Source.maybe. This eventually results in the upstream failing due to a TcpIdleTimeoutException, when there are no new Messages being sent down the stream.
Therefore, my question is twofold:
Is there a way – which I obviously missed – to treat a websocket as just a Source?
If using the Flow is the only option, how does one handle the TcpIdleTimeoutException properly? The exception can not be handled by providing a stream supervision strategy. Restarting the source by using a RestartSource doesn't help either, because the source is not the problem.
Update
So I tried two different approaches, setting the idle timeout to 1 second for convenience
application.conf
akka.http.client.idle-timeout = 1s
Using keepAlive (as suggested by Stefano)
Source.<Message>maybe()
.keepAlive(Duration.apply(1, "second"), () -> (Message) TextMessage.create("keepalive"))
.viaMat(Http.get(system).webSocketClientFlow(WebSocketRequest.create(websocketUri)), Keep.right())
{ ... }
When doing this, the Upstream still fails with a TcpIdleTimeoutException.
Using RestartFlow
However, I found out about this approach, using a RestartFlow:
final Flow<Message, Message, NotUsed> restartWebsocketFlow = RestartFlow.withBackoff(
Duration.apply(3, TimeUnit.SECONDS),
Duration.apply(30, TimeUnit.SECONDS),
0.2,
() -> createWebsocketFlow(system, websocketUri)
);
Source.<Message>maybe()
.viaMat(restartWebsocketFlow, Keep.right()) // One can treat this part of the resulting graph as a `Source<Message, NotUsed>`
{ ... }
(...)
private Flow<Message, Message, CompletionStage<WebSocketUpgradeResponse>> createWebsocketFlow(final ActorSystem system, final String websocketUri) {
return Http.get(system).webSocketClientFlow(WebSocketRequest.create(websocketUri));
}
This works in that I can treat the websocket as a Source (although artifically, as explained by Stefano) and keep the tcp connection alive by restarting the websocketClientFlow whenever an Exception occurs.
This doesn't feel like the optimal solution though.
No. WebSocket is a bidirectional channel, and Akka-HTTP therefore models it as a Flow. If in your specific case you care only about one side of the channel, it's up to you to form a Flow with a "muted" side, by using either Flow.fromSinkAndSource(Sink.ignore, mySource) or Flow.fromSinkAndSource(mySink, Source.maybe), depending on the case.
as per the documentation:
Inactive WebSocket connections will be dropped according to the
idle-timeout settings. In case you need to keep inactive connections
alive, you can either tweak your idle-timeout or inject ‘keep-alive’
messages regularly.
There is an ad-hoc combinator to inject keep-alive messages, see the example below and this Akka cookbook recipe. NB: this should happen on the client side.
src.keepAlive(1.second, () => TextMessage.Strict("ping"))
I hope I understand your question correctly. Are you looking for asSourceOf?
path("measurements") {
entity(asSourceOf[Measurement]) { measurements =>
// measurement has type Source[Measurement, NotUsed]
...
}
}
I have a publisher that is pushing messages to a topic. I have multiple subscribers each doing a different task once they consume the message from the topic.
Now I want my system to scale to multiple instances of the same process running on different hosts/same host. e.g. I want to run multiple copies of my application A on different hosts so that if one instance of A is slow, then the other instances can pull in subsequent messages and make forward progress..
I found out that this is possible using virtual destinations. I followed the steps here -
http://activemq.apache.org/virtual-destinations.html
But how do i setup my multiple subscribers to the same topic with the same client id? when i try to do that, i get errors. when i try some other way, it doesn't work. can someone help?
Normally, I start a subscriber by doing the below steps -
ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory(ActiveMQConnection.DEFAULT_USER, ActiveMQConnection.DEFAULT_PASSWORD, ActiveMQConnection.DEFAULT_BROKER_URL;);
activeMQConnection = connectionFactory.createConnection();
activeMQConnection.setClientID("subscriber1");
activeMQConnection.setExceptionListener(exceptionListener);
activeMQSession = activeMQConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE);
activeMQTopic = activeMQSession.createTopic("myTopic");
activeConsumer = activeMQSession.createDurableSubscriber(activeMQTopic, "myTopic");
activeConsumer.setMessageListener(messageListener);
activeMQConnection.start();
when i try to create a 2nd subscriber and pass the topic name as "VirtualTopic.myTopic", nothing happens.
thanks
The virtual topics feature is very simple and quite powerful once you understand it.
When using virtual topics - there is no need for durable consumers. That is because for each client you will get an instance of regular queue created. If you have 5 clients (application A, B, C, D, E) you will get 5 queues created and populated with the copy of the messages every time message is sent to the virtual topic.
Actually it is a limitation of durable consumer - that only ONE connection is allowed per clientId. Being a regular queue, you can create as many consumers as you like and queue will guarantee that 1 message will be received only by 1 consumer. So if you have application A that takes 1 minute to process a message, you can create 5 instances of it listening to the same queue. When you will post 5 messages within 1 second, each of your application will receive its own message to process.
There are not well documented requirements which are not intuitive. To make virtual topic work you need
Use VirtualTopic. in your topic name, for example VirtualTopic.Orders (this prefix can be configured)
Use Consumer. in the name of the queue you. Like Consumer.ApplicationA.VirtualTopic.Orders where ApplicationA is actually your client id
Use regular subscribers not durable ones for the queue above.
Example:
string activeMqConsumerTopic = "Consumer.AmqTestConsumer.VirtualTopic.Orders";
IQueue queue = SessionUtil.GetQueue(session, activeMqConsumerTopic);
IMessageConsumer consumer = session.CreateConsumer(queue);
Queue is created for automatically whenever the first instance of consumer is subscribed to it. Since that moment all messages that are sent to topic are duplicated/copied into all associated queues.
Hope this helps.
Virtual Topics is the answer for you. However you have to define a naming standard for all virtual topic queues. Here is the answer for this:
Virtual Topics helps with following prospective:
1. Load Balancing of messages
2. Fast Failover of Subscriber
3. Re-using same connection Factory for different Producers and Consumers. (Durable Subscribers needs a Unique JMS Client Id and same cannot be reused for any other Producer or consumer)
here is the way to do it, below example creates prefix VTCON.*. So every queue with this prefix and Topic Name at the end will consumer the message.
<virtualDestinations>
<virtualTopic name="TEST.TP01" prefix="VTCON.*." selectorAware="false"/>
</virtualDestinations>
http://workingwithqueues.blogspot.com/2012/05/activemq-virtual-topics-or-virtual.html
We have a scenario where we need to send a synchronous message over our an in memory activemq broker. Our synchronous client side code looks like this:
Session responseSession = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
TemporaryQueue responseQ = responseSession.createTemporaryQueue();
msg.setJMSReplyTo(responseQ);
QueueReceiver qReceiver = ((QueueSession) responseSession).createReceiver(responseQ);
sendMessage(msg, false, timeout);
Message response = qReceiver.receive(timeout);
Most of the time our server response code works fine but occasionally we get a stacktrace like:
javax.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID:<removed>
at org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1632)
at org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:231)
at org.apache.activemq.ActiveMQMessageProducerSupport.send(ActiveMQMessageProducerSupport.java:269)
I'm suspecting that the root problem is that the temporary queue isn't fully setup or that it hasn't been published or whatever it is that it does by the time the service attempts to publish a message to it. In my server code I've wrapped the send call in a loop catching the InvalidDestinationException, sleeping for a second and trying again until it succeeds. Since I've added this, whenever I see the exception on the second try it works.
Am I doing something wrong? Should I be doing something on the client to sync or flush or otherwise ensure that the temporary queue is up before sending the message to the server? Is there something else I should be doing on the server side to ensure the queue is up? Can I otherwise safely attempt to create the temporary queue on the server side if it thinks it isn't already there?
Note: We were using ActiveMQ 5.3.0 but today I tried 5.5.0 with the same results.
That's rather strange and I don't think it should be happening as the temp destination is created locally and should be in the Connections map of temp destinations. If you could create a simple JUnit test case the demonstrates the issue you should open a new Jira and attach it for review.