In my team we are using ActiveMQ 5.15.11 in production and our messages consumers are connected through HTTP protocol. Since we have network issues on the consumer side, we tried to disable the InactivityMonitor in the activemq.xml (see below)
<transportConnectors>
<transportConnector name="openwire" uri="tcp://host:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="http" uri="http://host:61617?wireFormat.maxInactivityDuration=0"/>
</transportConnectors>
It seems like the "wireFormat.maxInactivityDuration=0"is not applied since we have logs from InactivityMonitor in the consumer :
2020-03-18 00:00:03,426 [DEBUG] - - - WriteChecker: 10000ms elapsed since last write check.
2020-03-18 00:00:03,426 [DEBUG] - - - Running WriteCheck[https://host:443/activemq]
2020-03-18 00:00:03,509 [DEBUG] - - - WriteChecker: 10000ms elapsed since last write check.
2020-03-18 00:00:03,509 [DEBUG] - - - Running WriteCheck[https://host:443/activemq]
2020-03-18 00:00:03,514 [DEBUG] - - - WriteChecker: 10000ms elapsed since last write check.
2020-03-18 00:00:03,514 [DEBUG] - - - Running WriteCheck[https://host:443/activemq]
2020-03-18 00:00:03,642 [DEBUG] - - - WriteChecker: 10000ms elapsed since last write check.
2020-03-18 00:00:03,642 [DEBUG] - - - Running WriteCheck[https://host:443/activemq]
2020-03-18 00:00:03,706 [DEBUG] - - - WriteChecker: 10000ms elapsed since last write check.
2020-03-18 00:00:03,706 [DEBUG] - - - Running WriteCheck[https://host:443/activemq]
2020-03-18 00:00:03,738 [DEBUG] - - - WriteChecker: 10000ms elapsed since last write check.
2020-03-18 00:00:03,738 [DEBUG] - - - Running WriteCheck[https://host:443/activemq]
Furthermore, as soon as we get long network delay, we also get log :
2020-03-01 20:43:17,578 [WARN ] - - - Transport (https://host:443/activemq) failed , attempting to automatically reconnect: {}
org.apache.activemq.transport.InactivityIOException: Channel was inactive for too (>30000) long: https://host:443/activemq
at org.apache.activemq.transport.AbstractInactivityMonitor$5.run(AbstractInactivityMonitor.java:246)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Notice 1 : ActiveMQ clients are JMS Consumers.
Notice 2 : we tried to disable InactivityMonitor on TCP transport and it works well with this protocol.
Thanks for your help in advance.
For the HTTP transport I don't think that the standard OpenWire protocol negotiation step is performed which likely means that the server side configuration of max-inactivity-duration is visible at the client side. This results in the client using the defaults for the inactivity checks which I recall as being 30 seconds or so which it then splits into smaller chunks for read and write checks.
You would likely need to configure the client end to not do the checking using the same wireformat options. I don't think any of that is extensively tested though so there could likely be issues with altering defaults over the HTTP transport wrappers. If it continues to misbehave then likely you'd need to open a JIRA with the ActiveMQ project to get something implemented to let you control it better.
On the client side you should be able to disable the inactivity monitor with transport.useInactivityMonitor=false or maybe just useInactivityMonitor=false on the URI.
Related
I am developing a Spring Boot Application that uses HTTPS Only. I am using AWS services and the corresponding AWS Java SDK. How can I view the HTTP(S) request that the java sdk methods call on the backend of my application? I want to make sure when doing uploads to S3, etc, that everything is done over HTTPS only as the security of this application is important. A little confused on how to see when the backend of the application is interacting with the AWS services. Thanks in advance.
AWS uses HTTPS by default for all communications, and gives you options (such as VPC endpoints) that prevent traffic from leaving the AWS VPC.
Unfortunately, I couldn't find a reference in the Java SDK documentation that says it follows this practice. You can find guarantees for individual services (for example, S3). And it's implied that the SDK uses TLS by the page that describes how to enforce using TLS 1.2.
However, if you really want to be sure, you need to enable logging at the wire level.
Update in response to comment:
I ran the following program with debugging on:
public static void main(String[] argv)
throws Exception
{
S3Client client = S3Client.builder().build();
for (Bucket bucket : client.listBuckets().buckets())
{
System.out.println(bucket.name());
}
}
Looking at the logs, it's definitely making an HTTPS connection:
2021-11-17 17:51:24,802 [main] DEBUG request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=GET, protocol=https, host=s3.amazonaws.com, port=443, encodedPath=/, headers=[amz-sdk-invocation-id, User-Agent], queryParameters=[])
...
2021-11-17 17:51:24,832 [main] DEBUG PoolingHttpClientConnectionManager - Connection request: [route: {s}->https://s3.amazonaws.com:443][total kept alive: 0; route allocated: 0 of 50; total allocated: 0 of 50]
2021-11-17 17:51:24,839 [main] DEBUG PoolingHttpClientConnectionManager - Connection leased: [id: 0][route: {s}->https://s3.amazonaws.com:443][total kept alive: 0; route allocated: 1 of 50; total allocated: 1 of 50]
2021-11-17 17:51:24,840 [main] DEBUG MainClientExec - Opening connection {s}->https://s3.amazonaws.com:443
2021-11-17 17:51:24,871 [main] DEBUG DefaultHttpClientConnectionOperator - Connecting to s3.amazonaws.com/52.216.251.102:443
2021-11-17 17:51:24,871 [main] DEBUG SdkTlsSocketFactory - Connecting socket to s3.amazonaws.com/52.216.251.102:443 with timeout 2000
2021-11-17 17:51:24,902 [main] DEBUG SdkTlsSocketFactory - Enabled protocols: [TLSv1.2]
That's followed by the TLS negotiation, and then the actual request. One thing that may be confusing, if you're not familiar with the HTTP protocol, is this:
2021-11-17 17:51:25,022 [main] DEBUG MainClientExec - Executing request GET / HTTP/1.1
That's the HTTP request line, and the "HTTP/1.1" indicates the protocol version. This information is sent over the secure connection.
I have a standalone JAVA application.
And have integrated it successfully with Elastic APM (+ElasticSearch +Kibana) for capturing telemetries.
Java Version: 8 - OpenJDK
Elastic Agent & Library Version: 1.16
Elastic Search, APM and Kibana Version: 7.7.1
Below are the relevant JVM Options being used:
JAVA_OPTS="$JAVA_OPTS -javaagent:$BASE_HOME/agent-lib/elastic-apm-agent-1.16.0.jar -Delastic.apm.service_name=my-app -Delastic.apm.server_urls=http://elastic-apm-server:8200"
JAVA_OPTS="$JAVA_OPTS -Delastic.apm.application_packages=com,org -Delastic.apm.span_frames_min_duration=-1ms"
JAVA_OPTS="$JAVA_OPTS -Delastic.apm.log_file=$BASE_HOME/logs/apm.log -Delastic.apm.log_level=DEBUG"
I am generating custom transactions and spans using the Tracer/Transaction/Span APIs as suggested in the official documentation.
And as per the generated debug logs. These spans and transactions are getting captured as expected.
I have validated the same by DEBUGGING it over the IDE, that transactions are being captured as expected.
Problem: The custom transactions are not shown on the Kibana APM Dashboard
However some out of the box transactions from Quartz(which is being used in the application) are shown as expected. Which should mean the integration with the Elastic APM Server is fine.
It appears to me, even though the transactions are being captured successfully, those are not reported(sent) to the APM Server
Refer some relevant apm logs:
2020-07-01 12:33:09.569 [pool-1-thread-1] DEBUG co.elastic.apm.agent.impl.ElasticApmTracer - startTransaction '' 00-d0025079170e4f03698702f4e68be4ac-cf792454fbef1c77-01 (16970dc3) {
2020-07-01 12:33:09.569 [pool-1-thread-1] DEBUG co.elastic.apm.agent.impl.ElasticApmTracer - Activating 'ExtractionRequestHandler#invokeExtraction' 00-d0025079170e4f03698702f4e68be4ac-cf792454fbef1c77-01 (16970dc3) on thread 26
2020-07-01 12:33:09.569 [pool-1-thread-1] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - increment references to 'ExtractionRequestHandler#invokeExtraction' 00-d0025079170e4f03698702f4e68be4ac-cf792454fbef1c77-01 (16970dc3) (2)
2020-07-01 12:33:09.569 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving SPAN event (sequence 86)
2020-07-01 12:33:09.570 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'ExtractionRequestHandler#invokeExtraction' 00-98a1d8f4970d585915eb03a414b7b14c-994dd2823198f1ef-01 (33d448b5) (4)
2020-07-01 12:33:09.570 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'BOpFileUtils#authorizeFilePath' 00-98a1d8f4970d585915eb03a414b7b14c-133200d1793fbaab-01 (67fba8aa) (0)
2020-07-01 12:33:09.570 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving SPAN event (sequence 87)
2020-07-01 12:33:09.570 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'ExtractionRequestHandler#invokeExtraction' 00-98a1d8f4970d585915eb03a414b7b14c-994dd2823198f1ef-01 (33d448b5) (3)
2020-07-01 12:33:09.570 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'SCR#init' 00-98a1d8f4970d585915eb03a414b7b14c-77cf207c33eb24ab-01 (2f1f25c3) (0)
Need help in finding what I am doing wrong? And how to fix it?
I got the answer after posting the same on Elastic Support Forum.
It was a very prompt response.
This was not a problem from Elastic APM side, and was more of a silly problem from my side.
Refer the discussion to find the problem and solution.
I want to start my app even if the RabbitMQ is not reachable. Currently my app hangs while AmqpInboundChannelAdapter tries to establish connection and I see pattern how long it waits before it tries again. How to configure app using Spring AMQP to start regardless of RabbitMQ availability and configure this "back off policy"?
18/May/2015 16:52:57,666 INFO [main] - AmqpInboundChannelAdapter - started org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter#0
18/May/2015 16:54:47,769 INFO [main] - AmqpInboundChannelAdapter - started org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter#1
18/May/2015 16:57:07,818 INFO [main] - AmqpInboundChannelAdapter - started org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter#2
Can someone assist me in understanding and troubleshooting this issue? I do not know what is causing Hector to fail when it tries to connect to the Cassandra cluster.
How can I find out where the issue is?
0 [main] INFO me.prettyprint.cassandra.connection.CassandraHostRetryService - Downed Host Retry service started with queue size 10 and retry delay 30s
168 [main] INFO me.prettyprint.cassandra.service.JmxMonitor - Registering JMX me.prettyprint.cassandra.service_keyspace-name:ServiceType=hector,MonitorType=hector
399 [main] INFO me.prettyprint.cassandra.model.ConfigurableConsistencyLevel - READ ConsistencyLevel set to QUORUM for ColumnFamily Files
400 [main] INFO me.prettyprint.cassandra.model.ConfigurableConsistencyLevel - WRITE ConsistencyLevel set to QUORUM for ColumnFamily Files
406 [main] INFO me.prettyprint.cassandra.model.ConfigurableConsistencyLevel - READ ConsistencyLevel set to QUORUM for ColumnFamily FileList
407 [main] INFO me.prettyprint.cassandra.model.ConfigurableConsistencyLevel - WRITE ConsistencyLevel set to QUORUM for ColumnFamily FileList
From the trace it seems you are using QUORUM as consistency level. Try to use ONE and see if it works. It seems that one or more of the nodes that should satisfy your request are down. Use noderool ring/status or see if any node is down in your cluster.
I am using Camel to setup some routing using the file and jms-queue components. The problem that I am having is that I cannot disable polling messages sent to the console.
I tryed multiple ways to disable these messages by setting the logging level(runLoggingLevel = OFF) on the routes, trace = false on the context, set a logger on the routes and a few others but nothing works.
A message from the file component looks like this:
2013-08-26 09:34:47,651 DEBUG [Camel (camelContextOrder) thread #0 - file://order-import/order-in] o.a.c.c.f.FileConsumer Took 0.001 seconds to poll: order-import\order-in
And a messsage from the jms queue:
2013-08-26 09:34:46,281 DEBUG [ActiveMQ Journal Checkpoint Worker] o.a.a.s.k.MessageDatabase Checkpoint started.
2013-08-26 09:34:46,403 DEBUG [ActiveMQ Journal Checkpoint Worker] o.a.a.s.k.MessageDatabase Checkpoint done.
You have DEBUG logging level configured. You should change that to INFO etc so Camel / ActiveMQ will not log so much.
Check your logging configuration to adjust this.