As I indicated in another post, I'm having trouble with some SPIN constructors taking an excessive amount of time to execute quite limited data. I thought I'd take a different approach and see if I can profile the execution of the constructors to gain insight into where specifically they are spending excessive time.
How do I go about profiling the execution of constructors under RDF4J Server? I'm instantiating via SPARQL update (INSERT DATA) queries. Here's the System Information on RDF4J workbench:
I've attempted to profile the Tomcat server under which the RDF4J Server runs using jvisualvm.exe, but I have not gained much insight. Ideally, I'd like to get down to the class/method level within RDF4J so that I can post a more detailed request for help on my slow execution problem or perhaps fix my queries to be more efficient themselves.
So here's the version of Java Visual VM:
RDF4J is running under Apache Tomcat 8.5.5:
I can see overview information on Tomcat:
I can also see the monitor tab and threads:
HOWEVER, what I really want to see is the profiler so that I can see where my slow queries are spending so much time. That hangs on Calibration since I don't have the profiler calibrated for Java 1.8.
This attempting to connect box will persist indefinitely. Canceling it leads to the Performing Calibration message which doesn't actually do anything and is a dead-end hang requiring the Java VisualVM to be killed.
After killing the Java Visual VM and restarting and looking at Options-->Profiling-->Calibration Data, I see that only Java 7 has calibration data.
I have tried switching Tomcat over to running on Java 7, and that did work:
The profiler did come up with Tomcat:
However, when I tried to access the RDF4J workbench while Tomcat ran on Java 7, I could not get the workbench running:
So, I'm still stuck. It would appear that RDF4J requires Tomcat running under Java 1.8, not 1.7. I can't profile under Java 1.8.
I have seen other posts on this problem with Java VisualVM, but the one applicable solution seems to be to bring everything up in a development environment (e.g. Eclipse) and dynamically invoke the profiler at a debugger breakpoint once the target code is running under Java 1.8. I'm not set up to do that with Tomcat and RDF4J and would need pointers. My intention was not to become a Tomcat or RDF4J contributer (because my tasking doesn't allow that... I wouldn't be paid for the time) but rather to get a specific handle on what's taking so long for my SPIN constructor(s) in terms of RDF4J server classes and then ask for help from the RDF4J developer community on gitub.
Can Java VisualVM calibration be bypassed? Could I load a calibration file or directory somewhere for Java VisualVM to use instead of trying to measure calibration data which fails? I'm only interested in the relative CPU loading of classes, not absolute metrics, and I don't need to compare to measurements on other machines.
Thanks.
Related
You have to contract for supporting non public available Java releases with Oracle for example Java6 ended with 1.6.0_45 publicly but with support you can use 1.6.0_80. Our aim is to stay supported and public available java platform. Therefore we want to migrate our jnlp based RIA application from Java6 to java8 platform. In migration testing we have experienced performance problem. We try to simulate this problem in Java 8 poor GUI performance compared to Java 6 but we realize that the problem not related the that question.
After the introduction, in order to find out the root cause of problem we profiled the application with java mission control(jmc) bundled with jdk. when we profile application launched at workspace we cannot see any performance degradation. And then we realize that problem occurs only JAVAWS environment. Our application runs as expected at java environment but in javaws environment some GUI's are really slow , it adds 6 seconds more to response time.
My first question is that how we profile java application launched by javaws? Because to use jmc we need to set some JVM arguments (java-vm-args="-XX:+UnlockCommercialFeatures -XX:+FlightRecorder") but javaws doesn't support this JVM parameters. You can find supported list at here
My Second question is that why application doesn't run identical at both environment(java and javaws)? We examined signed and unsigned version of our application and both have same situation. By the way our application uses all-permission.
You can pass any JVM arguments you want to javaws by typing them in:
Java Control Panel > Java > View > Runtime Parameters
As for the poor performance, are you referring to application load time? or is it also slow after the applicatio is started?
If you're talking about load time, then yes javaws is slower (about 3-5 seconds slower) because it has to contact Internet to check if your jar files are black listed or if the certificate is expired. I'm not sure exactly what it is doing in these few seconds but I agree that it is disappointing performance hit and I hope someone can shed more light on it.
Thanks Saeid. Problem is determined. We profiled finally and we found that javaws consume long time to load resources e.g. icons, images. to make a solution we cached the resources and for absence images we return a empty image.
You can see this at
To profile application launched by javaws we make a solution following steps in ubuntu
1. move java executable javam
2. create a script file named java and insert following script
#!/bin/bash
/full/path/of/javam XX:+UnlockCommercialFeatures -XX:+FlightRecorder "$#"
3. run application and check java is launched by ps -ef | grep javam
4. run jmc and profile it.
I'm using Google App Engine v.1.9.1, Java edition, running inside Eclipse Kepler SR2. I've got JDK 1.7. My logging.properties is sending the logs to the java.util.logging.ConsoleHandler only.
My [edit] development server running in Eclipse [/edit] receives a lot of data from another server and just dumps it into a database. This generates a lot of logs. I get OutOfMemoryError after only a few hours.
I've run JProfiler and I figured out the object being kept around is com.google.apphosting.api.logservice.LogServicePb$LogLine. Somehow this isn't being discarded, ever, keeping millions of instances in memory.
Sure I can reduce the amount of data logged but that will only delay the problem.
I've looked everywhere to figure out how to flush out the log lines but I can't find any setting for this. The only option available is for Python not Java.
Any idea what's causing this and how to fix it?
As #Martin Berends said, the development server inside Eclipse is only for developing. It seems that log statements are kept in memory in that environment. Once I moved my app to a test server, the memory usage seems to be flat.
So the bottom line is; when running in a development environment, reduce the amount of logging and restart the server once in a while to avoid OutOfMemoryErrors. Secondly, do your tests on a real test server.
I assume the latest update version of java would provide better performance.
I am looking for a way to implement isolation of software components from endless loops or memory leaks. Android isolates each app in it's own process, Google Chrome isolates each tab in it's own process.
My primary drawback is that java takes so long to start and also I would like to reduce memory consumption.
Is there any alternate build or more controlled startup that will accomplish this?
If quick startup is your goal, Java on a PC may not be your best bet. It's going to take a few seconds because that's how long it takes to load the VM from disk.
If you want your app to start more quickly it's easy to get a splash screen up, just create a module that only loads your splash screen, waits for it to fully display then uses reflection to link to your "Real" main module.
(Use reflection because otherwise it will pull in your entire program through references before it starts the main one--at least that's how it used to work).
If you're talking about run-time performance, you won't get quicker by changing languages, Java's about as fast as you can get. You MIGHT be able to get a boost by converting to C/C++ and rewriting it to suit those platforms (Less OO, stack allocations instead of heap, etc), but otherwise none of the other languages in general usage are close to Java in speed.
If you really need the quick startup, depending on what you are doing there may be some tricks. I've seen projects that try to keep a Java VM running in your toolbar and allow you to make requests (tell it to start an app). This was faster but made additional requirements of the user (Loading this additional tool)
Another possibility--if you are constantly starting up/shutting down small tasks and that's the reason the startup bothers you then you can definitely speed it up by keeping it running invisibly. Just have your Java app open a socket and listen for commands then create a little .EXE or shell script that can start your program if it's not running or send commands to that socket if it is. This would completely eliminate startups after the first run.
In general, Java has a much longer startup time than other languages. If you are sticking with Java on a desktop app, a lot of stuff like startup time is determined by the JRE installed on the client's computer, which you can't control.
As to "endless memory leaks"... Java doesn't leak memory. If your program does, fix it.
This is a second answer because it's completely different and my other got too long :)
Try compiling it--I think GCC can compile it. This could almost completely eliminate your startup. I believe Jikes used to be a windows java compiler by IBM, but I don't know if it's still maintained.
Note that compiled code will probably run slower than JVM code for long-running apps.
I am using Rational Application Developer v7.0 that ships with an integrated test environment. When I get to debugging my webapp, the server startup time in debug mode is close to 5-6 minutes - enough time to take a coffee break!
At times, it so pisses me off that I start cursing IBM for building an operating system instead of an app server! Spawning 20+ processes and useless services with no documented configuration to tuning it, to starting any faster.
I am sure there are many java developers out there who would agree with me on this. I tried to disable the default apps and a set of services via my admin console, however that hasn't helped much.
I have no webservices, no enterprise beans, no queues, just a simple web app which requires a connection pool. Have you done something in the past to make your integrated test environment, start fast in debug mode and there by consume less RAM?
UPDATE:
I tried disabling a few services (internationalization, default apps etc...) and now the WebSphere server went from bad to worse. Not only doesn't it take horrifying startup time, it keeps freezing every now and then for up to 2 minutes. :-( Sounds like, optimization is not such a good thing, always!
The best way to debug server code is to use remote debugging.
First you need to add the following to the JVM params in the server start script:
-Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005
This will cause the JVM to listen on the specified port, then from your IDE you can start a remote debug session against that port and debug as if the code was running in the same process.
Working this way prevent you restarting the server so frequently and hence side-steps your problem with Websphere's start-up time.
You can get some odd results if the binaries on the server and the source in the IDE get out of sync but on the whole that's not a problem.
One of the main reasons is that you have a large application with many modules, classes, manifests, XML descriptors so on, and the fact that Websphere application server start up process is single threaded per se (thus each application may be started in a separate thread if they has equal weight). One other reason is that the Eclipse EMF and JST frameworks are very I/O intensive during startup and publish/deploy.
One other reason for the tedious start up is the annotation scanning which will occur during publish/deploy. This annotation scanning can be controlled and modified in a various ways. Look at this site:
http://wasdynacache.blogspot.se/2012/05/how-to-speed-up-annotation-processing.html
First of all, examine and evaluate your hardware, both CPU, memory and HDD. Is your processor/s running 100% for a long time during start up? If so, the processor may be too weak. Is paging occur? then you may have to put in some more RAM. The Websphere/eclipse JST and EMF frameworks are very I/O intense so you should consider to invest in a SSD disc. You should also make sure that other processes on your machine (virus protection software etc.) don´t steal hardware resources from the Websphere java processes.
So for the hardware:
1. Processor - a pretty fast one, since the publish and the startup is mostly singlethreaded you do not need that many cpu cores
2. Memory - You will at least need 512Mb of physical RAM, this depends of the size of your application of course.
3. Storage - I would definitely go for a fast SSD since the underlying eclipse framework is I/O intensive.
Here are some tricks to reduce the footprint of the start up phase. Please before applying these settings make sure that you record a baseline start up so that you can observe the difference in start up, i.e. the reduced start up time.
JVM args : -Xverify:none -Xquickstart -Xnoclassgc -XX:+UseNUMA -XtlhPrefetch -Xgcthreads4 (I got 4 virtual processors installed on my machine)
Extend the heap size to match the demands of your application.
Disable the autostart of the application to reduce publish time.
Disable PMI and unnecessary tracing.
Profile your application during startup and fix bottlenecks if found any.
Other JVM arguments that may gain performance:
com.ibm.cacheLocalHost=true
com.ibm.ws.classloader.zipFileCacheSize=512
com.ibm.ws.classloader.resourceRequestCacheSize=1024
com.ibm.ws.management.event.pull_notification_timeout =20000
com.ibm.ws.amm.scan.context.filter.packages=true
org.eclipse.jst.j2ee.commonarchivecore.disableZip=true
Jvm arguments that will make the Websphere application server to stop immediately:
com.ibm.ejs.sm.server.quiesceTimeout=0
com.ibm.ejs.sm.server.quiesceInactiveRequestTime=1000
Webcontainer properties:
com.ibm.wsspi.jsp.disableTldSearch=true
com.ibm.wsspi.jsp.disableResourceInjection=true
JVM arguments that may be specified eclipse.ini (Note that the heap parameters is configured according to the conditions of my environment)
-Dcom.ibm.ws.management.event.max_polling_interval=5000
-Xquickstart
-Xverify:none
-Xmxcl25000
-Xjit:dataTotal=65536
-Xcodecache64m
-Xscmx48m
-Xnolinenumbers
-Xverify:none
-Xmnx64m
-Xmx1446m
-Xmnx64m
-XX:+UseCompressedOops
-XX:+UseNUMA
5 to 6 mins is not normal. I use RAD and WAS everyday and get decent startup times. Which version of WAS are you running and how much RAM do you have?
If you share several workspaces and projects for a same WAS profile, consider creating a new WAS profile for your workspace.
You probably tried that but here's a simple check list of things to try on first hand. Make sure that your server settings in RAD has the following options enabled:
Optimize server for testing and developing
Run server with resources on the workspace
Minimize application files copied to the server
Uncheck "Enable universal test client" if you don't need it.
In the admin console you can verify some server settings such as
Run in development mode
Parallel start
Start components as needed
You can also uninstall the ivt app that comes installed by default when creating a new WAS profile. Then the usual things such as a drive that is not too fragmented and a pagefile size that is properly set.
And one last thing that you probably know already, republish to your server instead of restarting it.
That's one reason why Spring was born.
You don't even have to give all the niceties like JMS, remoting, etc. You'd be better off with Tomcat, ActiveMQ, and OpenEJB.
Anything but WebSphere.
There's some hints and tips for tuning RAD 6 on developerworks that may help, many of these also apply for RAD 7.
I have seen a similar list for RAD 7, I'll post it if I can find it.
I did find some tuning tips for Portal on RAD 7.
I would say my experience with the test environment has been suboptimal. I now tend to use Tomcat/Pluto configured for remote debugging with an External launch configuration to manage it from within bare Eclipse and rely on having appropriate JNDI configurations to abstract the underlying server.
If you are coding to the relevant APIs it shouldn't matter for development purposes that you're not on Websphere. If you do have a Webpshere specific issue you can always crank up the beast to debug it.
If you have no EJBs, no JMS, etc., just deploy under a standalone servlet container such as Tomcat or Jetty, you'll be amazed how fast it is :-), being ironic here but it's true!
If the connection pool really is the only appserver feature you use then why don't you simply use apache commons dbcp (http://commons.apache.org/dbcp/) drop webfear alltogether and use jetty instead. That should reduce your startup time to about 5 seconds. You can then later easily switch to websphere again for your production environment if you should really feel the need to.
WAS V7 addresses some of these problems by allowing you to configure what starts up when the app server starts up.
So if and when you migrate to WAS V7 you might seem some improvements in this space.
We have a curious problem with our java processes dying.
The application doesn't stacktrace, or write anything to the logs, the process just randomly dies. It's a heavily used application, but the problem only appears about once a month.
We're currently looking into using Process Monitor but any other suggestions would be welcome.
Edit:
It's a distributed Java application, running on Weblogic with an in-house web framework (Yes, this is a terrible idea, but it's been running for eight years), connecting to Oracle.
-
Out of Memory?
Our logs would catch java.lang.OutOfMemoryException, according to Brian Agnew.
Write crashes to a log? I don't think Java ever gets the chance, the death is happening at a process level, rather than Java exiting.
Can you wrap it in some shell script that captures the log files (stdout/stderr) and the exit code (which should give some indication as to how it died) ? On JVM exit you can also capture machine level stats using WMI
IF the VM itself is crashing it'll leave behind an hs_err_pid... file that contains stacktraces, machine-level debug info. You can then use that to diagnose the VM issue. See this blog entry for further information.
If the problem is related to the app's behaviour, it may be worth looking at JConsole, although from your description of the issue, this sounds much more like a low level VM issue.
(I assume you're on the latest VM for your Java version number etc.)
You can use a Linux NAGIOS Server to monitor the health of your Windows machines and services! Have a look at: nagios-monitoring-windows.
If you have such problems with your java app! You should test it and debug it! Applications shouldn't die without a trace! Look for logfiles! From which vendor is the app? Or is it self written? Try to enforce another Log4J/Logger/Debug Level. Monitor your System with cacti etc. to reduce the possibilities for such a crash. Talk to the software vendor.
Is enogh memory available? Maybe the app runs out of memory? Is it a standalone java process or a java process from a tomcat/jboss server?
Have you written down the crash times to a log? Appear they in different time-slices? Or appear they nearly time-circular?
VisualVM is a new tool which makes monitoring Java applications easier:
https://visualvm.dev.java.net/description.html
"VisualVM is a tool that provides detailed information about Java applications while they are running. It provides an intuitive graphical user interface that allows you to easily see information about multiple Java applications."