I have a standalone JAVA application.
And have integrated it successfully with Elastic APM (+ElasticSearch +Kibana) for capturing telemetries.
Java Version: 8 - OpenJDK
Elastic Agent & Library Version: 1.16
Elastic Search, APM and Kibana Version: 7.7.1
Below are the relevant JVM Options being used:
JAVA_OPTS="$JAVA_OPTS -javaagent:$BASE_HOME/agent-lib/elastic-apm-agent-1.16.0.jar -Delastic.apm.service_name=my-app -Delastic.apm.server_urls=http://elastic-apm-server:8200"
JAVA_OPTS="$JAVA_OPTS -Delastic.apm.application_packages=com,org -Delastic.apm.span_frames_min_duration=-1ms"
JAVA_OPTS="$JAVA_OPTS -Delastic.apm.log_file=$BASE_HOME/logs/apm.log -Delastic.apm.log_level=DEBUG"
I am generating custom transactions and spans using the Tracer/Transaction/Span APIs as suggested in the official documentation.
And as per the generated debug logs. These spans and transactions are getting captured as expected.
I have validated the same by DEBUGGING it over the IDE, that transactions are being captured as expected.
Problem: The custom transactions are not shown on the Kibana APM Dashboard
However some out of the box transactions from Quartz(which is being used in the application) are shown as expected. Which should mean the integration with the Elastic APM Server is fine.
It appears to me, even though the transactions are being captured successfully, those are not reported(sent) to the APM Server
Refer some relevant apm logs:
2020-07-01 12:33:09.569 [pool-1-thread-1] DEBUG co.elastic.apm.agent.impl.ElasticApmTracer - startTransaction '' 00-d0025079170e4f03698702f4e68be4ac-cf792454fbef1c77-01 (16970dc3) {
2020-07-01 12:33:09.569 [pool-1-thread-1] DEBUG co.elastic.apm.agent.impl.ElasticApmTracer - Activating 'ExtractionRequestHandler#invokeExtraction' 00-d0025079170e4f03698702f4e68be4ac-cf792454fbef1c77-01 (16970dc3) on thread 26
2020-07-01 12:33:09.569 [pool-1-thread-1] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - increment references to 'ExtractionRequestHandler#invokeExtraction' 00-d0025079170e4f03698702f4e68be4ac-cf792454fbef1c77-01 (16970dc3) (2)
2020-07-01 12:33:09.569 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving SPAN event (sequence 86)
2020-07-01 12:33:09.570 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'ExtractionRequestHandler#invokeExtraction' 00-98a1d8f4970d585915eb03a414b7b14c-994dd2823198f1ef-01 (33d448b5) (4)
2020-07-01 12:33:09.570 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'BOpFileUtils#authorizeFilePath' 00-98a1d8f4970d585915eb03a414b7b14c-133200d1793fbaab-01 (67fba8aa) (0)
2020-07-01 12:33:09.570 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving SPAN event (sequence 87)
2020-07-01 12:33:09.570 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'ExtractionRequestHandler#invokeExtraction' 00-98a1d8f4970d585915eb03a414b7b14c-994dd2823198f1ef-01 (33d448b5) (3)
2020-07-01 12:33:09.570 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'SCR#init' 00-98a1d8f4970d585915eb03a414b7b14c-77cf207c33eb24ab-01 (2f1f25c3) (0)
Need help in finding what I am doing wrong? And how to fix it?
I got the answer after posting the same on Elastic Support Forum.
It was a very prompt response.
This was not a problem from Elastic APM side, and was more of a silly problem from my side.
Refer the discussion to find the problem and solution.
I am having problems running an app I have developed in an EC2 instance. When I execute the .jar (java -jar app.jar), the SpringBoot app starts but it fails when trying to connect to my MySQL RDS database. The thing is when I run the app locally on my machine, It has no issues with the DB connection.
I have opened the port where the app is running (8090) and MySql port as well (3306) for inbound and outbound traffic:
This is the error I get:
2016-09-23 17:46:38.132 INFO 10161 --- [main] .t.TomcatEmbeddedServletContainerFactory : Server initialized with port: 8090
2016-09-23 17:46:38.604 INFO 10161 --- [main] o.apache.catalina.core.StandardService : Starting service Tomcat
2016-09-23 17:46:38.605 INFO 10161 --- [main] org.apache.catalina.core.StandardEngine : Starting Servlet Engine: Apache Tomcat/7.0.54
2016-09-23 17:46:38.724 INFO 10161 --- [ost startStop 1] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2016-09-23 17:46:38.725 INFO 10161 --- [ost startStop 1] o.s.web.context.ContextLoader: Root WebApplicationContext: initialization completed in 5028 ms
2016-09-23 17:48:48.476 ERROR 10161 --- [ost startStop 1] o.a.tomcat.jdbc.pool.ConnectionPool: Unable to create initial connections of pool.
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
Any ideas how can i solve this problem?
Thank you very much for your help
Regards
Andres
From your description and log file, it's likely that network configuration is the cause here.
You might want to draw the network topology of your instances (region/availability zone, VPC, subnet, network acl, security group). This will be very helpful when you do more complex development work.
There are good references: VPC Introduction and Security in your VPC and Scenarios for Accessing a DB Instance in a VPC
I suggest the following actions for your troubleshooting:
Check security group (SG) configuration of your EC2 instance and RDS instance.
You can check this by going to EC2 Dashboard/RDS Dashboard -> Click on an instance and look at "Security Group" description, or you can click on the Setting icon (Show/Hide columns) and tick "Security Groups".
In RDS's SG configuration: make sure you have enable access from EC2 instance's SG to port 3306. You can do this by putting EC2 instance's SG ID into Source field of the config, as a "Custom IP" value. See the 1st scenario in the above reference for more detail.
Use mysql command line to test the connection between EC2 instance and RDS.
Hope it helps.
You need to perform following steps :
1) Go to EC2 instance and find security group you want access in RDS
2) Now go to your RDS security group and select inbound rules
Select ALL TCP and add your sg-xxx(security group)
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_VPC.Scenarios.html
I start an Hbase cluster for my test class. I use that helper class:
HBaseClusterSingleton.java
and use it as like that:
private static final HBaseClusterSingleton cluster = HBaseClusterSingleton.build(1);
I retrieve configuration object as follows:
cluster.getConf()
and I use it at Spark as follows:
sparkContext.newAPIHadoopRDD(conf, MyInputFormat.class, clazzK,
clazzV);
When I run my test there is no need to startup an Hbase cluster because Spark will connect to my dummy cluster. However when I run my test method it throws an error:
2015-08-26 01:19:59,558 INFO [Executor task launch
worker-0-SendThread(localhost:2181)] zookeeper.ClientCnxn
(ClientCnxn.java:logStartConnect(966)) - Opening socket connection to
server localhost/127.0.0.1:2181. Will not attempt to authenticate
using SASL (unknown error)
2015-08-26 01:19:59,559 WARN [Executor
task launch worker-0-SendThread(localhost:2181)] zookeeper.ClientCnxn
(ClientCnxn.java:run(1089)) - Session 0x0 for server null, unexpected
error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused at
sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
Hbase tests, which do not run on Spark, works well. When I check the logs I see that cluster and Spark is started up correctly:
2015-08-26 01:35:21,791 INFO [main] hdfs.MiniDFSCluster
(MiniDFSCluster.java:waitActive(2055)) - Cluster is active
2015-08-26 01:35:40,334 INFO [main] util.Utils
(Logging.scala:logInfo(59)) - Successfully started service
'sparkDriver' on port 56941.
I realized that when I start up an hbase from command line my test method for Spark connects to it!
So, does it means that it doesn't care about the conf I passed to it? Any ideas about how to solve it?
I have trouble connecting to my 3-node cassandra cluster via Datastax PHP- and Java-Driver.
Especially for the PHP driver it is crucial that i can connect fast to improve loading times of my website.
How can i debug this or what is the reason?
Java output shows this:
09:59:40,284 [main] DEBUG - com.datastax.driver.NEW_NODE_DELAY_SECONDS is undefined, using default value 1
09:59:40,284 [main] DEBUG - com.datastax.driver.NON_BLOCKING_EXECUTOR_SIZE is undefined, using default value 4
09:59:40,297 [main] DEBUG - com.datastax.driver.NOTIF_LOCK_TIMEOUT_SECONDS is undefined, using default value 60
09:59:40,357 [main] DEBUG - Starting new cluster with contact points [/XXX.XXX.XXX.XXX:9042, /XXX.XXX.XXX.YYY:9042, /XXX.XXX.XXX.ZZZ:9042]
09:59:40,402 [main] DEBUG - Using SLF4J as the default logging framework
09:59:40,489 [main] DEBUG - java.nio.Buffer.address: available
09:59:40,490 [main] DEBUG - sun.misc.Unsafe.theUnsafe: available
09:59:40,490 [main] DEBUG - sun.misc.Unsafe.copyMemory: available
09:59:40,490 [main] DEBUG - java.nio.Bits.unaligned: true
09:59:40,492 [main] DEBUG - Java version: 8
09:59:40,492 [main] DEBUG - -Dio.netty.noUnsafe: false
09:59:40,492 [main] DEBUG - sun.misc.Unsafe: available
09:59:40,492 [main] DEBUG - -Dio.netty.noJavassist: false
09:59:40,665 [main] DEBUG - Javassist: available
09:59:40,665 [main] DEBUG - -Dio.netty.tmpdir: /var/folders/4y/t4b47lbn1zjbjpb6x09l30wm0000gn/T (java.io.tmpdir)
09:59:40,666 [main] DEBUG - -Dio.netty.bitMode: 64 (sun.arch.data.model)
09:59:40,666 [main] DEBUG - -Dio.netty.noPreferDirect: false
09:59:40,708 [main] DEBUG - com.datastax.driver.FORCE_NIO is undefined, using default value false
09:59:40,710 [main] INFO - Did not find Netty's native epoll transport in the classpath, defaulting to NIO.
09:59:40,714 [main] DEBUG - -Dio.netty.eventLoopThreads: 8
09:59:40,723 [main] DEBUG - -Dio.netty.noKeySetOptimization: false
09:59:40,723 [main] DEBUG - -Dio.netty.selectorAutoRebuildThreshold: 512
09:59:40,747 [main] DEBUG - -Dio.netty.leakDetectionLevel: simple
09:59:41,035 [main] DEBUG - com.datastax.driver.DISABLE_COALESCING is undefined, using default value false
09:59:41,046 [main] DEBUG - Generated: io.netty.util.internal.__matchers__.com.datastax.driver.core.Message$ResponseMatcher
09:59:41,066 [main] DEBUG - -Dio.netty.allocator.numHeapArenas: 4
09:59:41,066 [main] DEBUG - -Dio.netty.allocator.numDirectArenas: 4
09:59:41,066 [main] DEBUG - -Dio.netty.allocator.pageSize: 8192
09:59:41,066 [main] DEBUG - -Dio.netty.allocator.maxOrder: 11
09:59:41,067 [main] DEBUG - -Dio.netty.allocator.chunkSize: 16777216
09:59:41,067 [main] DEBUG - -Dio.netty.allocator.tinyCacheSize: 512
09:59:41,067 [main] DEBUG - -Dio.netty.allocator.smallCacheSize: 256
09:59:41,067 [main] DEBUG - -Dio.netty.allocator.normalCacheSize: 64
09:59:41,067 [main] DEBUG - -Dio.netty.allocator.maxCachedBufferCapacity: 32768
09:59:41,067 [main] DEBUG - -Dio.netty.allocator.cacheTrimInterval: 8192
09:59:41,078 [main] DEBUG - Generated: io.netty.util.internal.__matchers__.com.datastax.driver.core.FrameMatcher
09:59:41,082 [main] DEBUG - Generated: io.netty.util.internal.__matchers__.com.datastax.driver.core.Message$RequestMatcher
09:59:41,104 [main] DEBUG - -Dio.netty.initialSeedUniquifier: 0x24d6f22f78c5a924 (took 8 ms)
09:59:41,130 [main] DEBUG - -Dio.netty.allocator.type: unpooled
09:59:41,130 [main] DEBUG - -Dio.netty.threadLocalDirectBufferSize: 65536
09:59:41,197 [cluster1-nio-worker-0] DEBUG - Connection[/XXX.XXX.XXX.YYY:9042-1, inFlight=0, closed=false] Connection opened successfully
09:59:41,218 [cluster1-nio-worker-0] DEBUG - -Dio.netty.recycler.maxCapacity.default: 262144
09:59:41,432 [main] DEBUG - [Control connection] Refreshing node list and token map
09:59:41,518 [main] DEBUG - [Control connection] Refreshing schema
09:59:42,137 [main] DEBUG - [Control connection] Refreshing node list and token map
09:59:42,315 [main] DEBUG - [Control connection] Successfully connected to /XXX.XXX.XXX.YYY:9042
09:59:42,315 [main] INFO - Using data-center name '168' for DCAwareRoundRobinPolicy (if this is incorrect, please provide the correct datacenter name with DCAwareRoundRobinPolicy constructor)
09:59:42,315 [main] INFO - New Cassandra host /XXX.XXX.XXX.XXX:9042 added
09:59:42,315 [main] INFO - New Cassandra host /XXX.XXX.XXX.YYY:9042 added
09:59:42,315 [main] INFO - New Cassandra host /XXX.XXX.XXX.ZZZ:9042 added
09:59:42,342 [cluster1-nio-worker-1] DEBUG - Connection[/XXX.XXX.XXX.XXX:9042-2, inFlight=0, closed=false] Connection opened successfully
09:59:42,345 [cluster1-nio-worker-2] DEBUG - Connection[/XXX.XXX.XXX.YYY:9042-1, inFlight=0, closed=false] Connection opened successfully
09:59:42,348 [cluster1-nio-worker-3] DEBUG - Connection[/XXX.XXX.XXX.ZZZ:9042-1, inFlight=0, closed=false] Connection opened successfully
09:59:42,580 [cluster1-nio-worker-2] DEBUG - Added connection pool for /XXX.XXX.XXX.XXX:9042
09:59:42,591 [cluster1-nio-worker-3] DEBUG - Added connection pool for /XXX.XXX.XXX.YYY:9042
09:59:42,609 [cluster1-nio-worker-1] DEBUG - Added connection pool for /XXX.XXX.XXX.ZZZ:9042
As you can see, it takes ~2.5 seconds which is too slow for my use case.
Same happens with the PHP driver, but i don't have a log for this.
Queries are very fast once the driver is connected. Only issue is the slow connection time. I have set up all the three nodes as contact points.
EDIT
Just to clarify: My PHP driver is the problem. I'm wondering why it isn't using pooling/persisting connections. When i call the script two times in a row, every call takes 2-5 seconds. I think the second call should be using the persisting pool. phpinfo() shows persistent clusters & sessions = 0. This is the code that i'm using:
$cluster = Cassandra::cluster()
->withContactPoints('XXX.XXX.XXX.XXX', 'XXX.XXX.XXX.YYY, 'XXX.XXX.XXX.ZZZ')
->withCredentials('USERNAME', 'PASSWORD')
->build();
$keyspace = 'myKeyspace';
$session = $cluster->connect($keyspace);
UPDATE
The problem was my network. Had too little bandwidth.
DataStax drivers are full featured drivers. They are aware of your custer topology and cluster state which requires some expensive operations in the cluster object build stage. It is common for the cluster object creation to take multiple seconds (depending on the size of your cluster/number of nodes).
The best practice is not to create the cluster object for every request (that would be extremely inefficient). Instead, you want to build the cluster object one time and maintain the connections open. Then when you receive a request from your front end, handle it with the existing cluster object.
Cassandra will give you very fast response times when used correctly.
For other c* client best practices, take a look at Brian's Cassandra Loader. This is a good reference application as well as a very efficient bulk loader.
Some key best practices include: Limit the number of async requests if you are using execute async, if you are using batches, ensure that the batches are token specific to avoid excessive coordination, do not use logged batches unless you need atomicity, and do not dynamically manipulate your schema from your application to avoid schema mismatches.
I am using Camel to setup some routing using the file and jms-queue components. The problem that I am having is that I cannot disable polling messages sent to the console.
I tryed multiple ways to disable these messages by setting the logging level(runLoggingLevel = OFF) on the routes, trace = false on the context, set a logger on the routes and a few others but nothing works.
A message from the file component looks like this:
2013-08-26 09:34:47,651 DEBUG [Camel (camelContextOrder) thread #0 - file://order-import/order-in] o.a.c.c.f.FileConsumer Took 0.001 seconds to poll: order-import\order-in
And a messsage from the jms queue:
2013-08-26 09:34:46,281 DEBUG [ActiveMQ Journal Checkpoint Worker] o.a.a.s.k.MessageDatabase Checkpoint started.
2013-08-26 09:34:46,403 DEBUG [ActiveMQ Journal Checkpoint Worker] o.a.a.s.k.MessageDatabase Checkpoint done.
You have DEBUG logging level configured. You should change that to INFO etc so Camel / ActiveMQ will not log so much.
Check your logging configuration to adjust this.