We Keep Coding

How to Profile Execution of RDF4J Server?

As I indicated in another post, I'm having trouble with some SPIN constructors taking an excessive amount of time to execute quite limited data. I thought I'd take a different approach and see if I can profile the execution of the constructors to gain insight into where specifically they are spending excessive time.
How do I go about profiling the execution of constructors under RDF4J Server? I'm instantiating via SPARQL update (INSERT DATA) queries. Here's the System Information on RDF4J workbench:
I've attempted to profile the Tomcat server under which the RDF4J Server runs using jvisualvm.exe, but I have not gained much insight. Ideally, I'd like to get down to the class/method level within RDF4J so that I can post a more detailed request for help on my slow execution problem or perhaps fix my queries to be more efficient themselves.
So here's the version of Java Visual VM:
RDF4J is running under Apache Tomcat 8.5.5:
I can see overview information on Tomcat:
I can also see the monitor tab and threads:
HOWEVER, what I really want to see is the profiler so that I can see where my slow queries are spending so much time. That hangs on Calibration since I don't have the profiler calibrated for Java 1.8.
This attempting to connect box will persist indefinitely. Canceling it leads to the Performing Calibration message which doesn't actually do anything and is a dead-end hang requiring the Java VisualVM to be killed.
After killing the Java Visual VM and restarting and looking at Options-->Profiling-->Calibration Data, I see that only Java 7 has calibration data.
I have tried switching Tomcat over to running on Java 7, and that did work:
The profiler did come up with Tomcat:
However, when I tried to access the RDF4J workbench while Tomcat ran on Java 7, I could not get the workbench running:
So, I'm still stuck. It would appear that RDF4J requires Tomcat running under Java 1.8, not 1.7. I can't profile under Java 1.8.
I have seen other posts on this problem with Java VisualVM, but the one applicable solution seems to be to bring everything up in a development environment (e.g. Eclipse) and dynamically invoke the profiler at a debugger breakpoint once the target code is running under Java 1.8. I'm not set up to do that with Tomcat and RDF4J and would need pointers. My intention was not to become a Tomcat or RDF4J contributer (because my tasking doesn't allow that... I wouldn't be paid for the time) but rather to get a specific handle on what's taking so long for my SPIN constructor(s) in terms of RDF4J server classes and then ask for help from the RDF4J developer community on gitub.
Can Java VisualVM calibration be bypassed? Could I load a calibration file or directory somewhere for Java VisualVM to use instead of trying to measure calibration data which fails? I'm only interested in the relative CPU loading of classes, not absolute metrics, and I don't need to compare to measurements on other machines.
Thanks.

How can I setup Tomcat running on a virtual machine to enable remote debugging as well as hot swapping synced file changes?

Apologies in advance for the lengthy explanation.
We have an large web application which uses a RabbitMQ server, a Tomcat webserver, a MySQL database, and a Hadoop environment. Most of the team uses Windows, and one uses Ubuntu, but we use CentOS for our deployment environment. We've recently started using vagrant and ansible to standardise environment configuration and deployment across multiple environments. This is part of a move to improve the quality of our application, and helps us to avoid failure between different environments.
We have most of it working now, and over the last few days I've been looking into replicating (or maybe even improving) our current development process with regards to building, running, debugging, and hot-swapping minor code changes without restarting the application. Our existing setup uses embedded Jetty (within eclipse), and we can run Jetty in either Run or Debug mode. With this setup we can place breakpoints, navigate to a relevant page, and when the breakpoint activates we can step through the code, drop frames off the stack to run through a method again, and even make minor changes to the code (within the rules of JVM hotswapping). This is, of course, fantastically helpful for us.
I want to replicate as much of this behaviour as possible except instead of an embedded jetty instance, I want to remote debug the tomcat instance on the virtual machine, but still be able to make changes to the code without needing to restart tomcat each time. I should probably point out at this point that I am aware of JRebel, but it has been deemed unfeasible for our team at this point in time.
At this point in time I have managed to automatically sync (using vagrant); 'src/main/webapp/WEB-INF', 'target/classes' (compiled by eclipse), and 'target/lib' (created by the maven copy-dependencies plugin'. These are all synced to the correct place on the vm, so that tomcat believes it is an exploded war. I have correctly setup Tomcat so that it runs in debug mode, and I have connected to it with eclipse, and successfully entered debug mode by placing a breakpoint. Making any changes to one of the synced files causes tomcat to restart the application and use the new file.
The only problems left are that I lose all my session data when I save a change to a file, and I have to wait for the re-deployment to complete and then repeat appropriate steps to get back to my breakpoint. This is obviously a big waste of time.
So after that lengthy explanation, my questions are:
Has anyone managed to achieve what I'm trying to do?
Is what I'm trying to do even possible?
If so does anyone have any tips to help me finish my configuration?
Can I do hot class replacement without triggering a context reload?
Any help would be greatly appreciated and sorry for the essay! :)

Website developmet with JBOSS

In my company there are around 30-40 developers and all of them are working on same website developmet project. Project is devided in many modules. My problem is every developer has run his own JBOSS to run his JAVA code. With Ant, they build the Ear file and deploye them on JBOSS which is running on their local desktop. This require huge memory and slow down the desktop.
Isn't possible if we can run the JBOSS on different server machine and all developer use this machine to run and test their code simultaneously? Hope you guys understand my question?
Thanks in advance.

You can set up a continuous integration server like Jenkins. With Jenkins you have different "projects". Each project builds the code and runs an instance of JBoss.
With Jenkins you can have a different project for each developer (if that's what you want) and you can get the added benefits of Jenkins compiling and running unit-tests each time a developer checks in code.

it is possible.you can achieve this by running the JBOSS Application Server on a particular IP.to do this you have to use ./run.sh -b followed by the IP Address of the machine on which you want to run the application server.with this your server is ready.now every client has to connect to the server with the help of that IP address.for this you replace the loopback address(127.0.0.1) with the IP address on which application server is running.

You can but with many developers all together it can get tricky.
It is possible to run a single instance of jboss but then you run into problems with either hot deploys, slowing work and cross contaminating if different devs work on the same code, or no hot deploys, in which case the server has to be shutdown every time an app is deployed/changed. In a testing environment this doesn't work very well.
The other way would be to run a separate instance for every dev to run tests on but this can get very cost intensive in both money and server resources. You also have a nightmare when you try to merge changes from these various environments.
Some good organization might help. We run three layers:development, staging, production. Development is purely for your developers. Bug tracking and testing is handled in staging while production is for user and customer evaluations(assuming the product hasn't been rolled yet). Use a good repository to code up changes from local machines to development, then the admins roll groups of changes to staging. It doesn't really solve your problem of the memory on local machines but I thought I would try and help out with some enterprise level stuff.
One quick fix would be to put tons of RAM on each machine, its dirt cheap anyway. Then set your server memory settings up way higher. That will reduce your cpu loads a bit and reduce the number of time you see the dreaded permgen error from too many hot deploys

Tool to log (OS) events during EJB tests (analyze PC crash)

I have some issues with my PC and don't know what else to check or look for.
If the tags or description are off, feel free to edit/comment.
Basically the question is: Do you know of any tool that I could run during reproduction of the test, which logs frequently and might provide a hint on what's going on?
If anyone already has a clue of what could cause the problem, that would be great as well.
So here's the problem:
I have a running JBoss 4.2.3.GA server application which provides some EJBs with remote interfaces. Those EJBs write or read stuff to/from the database but that doesn't seem to matter since I also had methods that just did a System.out.println(...) and nothing more.
Now I run a test client from the console which basically just "remotely" calls one of those EJB methods in a loop (to take some timings etc.).
So far nothing too unorthodox should be done, it's basically just a bunch of remote EJB calls.
However, during the execution of the loop the computer freezes completely (keyboard doesn't respond as well, e.g. num-lock key) - the only thing that changes is the blinking cursor. :)
Unfortunately I didn't manage to find a reason for this and since I often do my tests from eclipse I'd like to not have that happen too often (workspace crashes etc.)
Here's what I tried so far:
Numerous hardware tests including Lenovo PC Doctor (it's a Lenovo PC) - all succeeded, so i seems like there's no hardware problem
Use different JDK versions: 1.5.0, 1.6.0, 1.7.0 - all crashed
JRockit JVM (Java 6) - crashed as well
make Java cause 100% CPU load (10 thread running constantly on 4 cores) - succeeded/no error
allocate as much memory as the JVM would allow me - succeeded
run the tests on other computers - succeeded, except one that has the same hardware and similar software setup
Windows logs don't provide a hint (except "system was not shut down correctly" ... well that helps :) )
After all these tests I assume it might be a problem with the system configuration (drivers etc.) but I don't know how to track that (and I can't just use brute force due to the massive time requirements).
So, did anyone experience similar problems?
Do you know of any tool that I might use to log what the system does and preferably get a log right before the crash?
Thanks in advance,
Thomas

Websphere Application Server - What on earth will it take to start any fast?

I am using Rational Application Developer v7.0 that ships with an integrated test environment. When I get to debugging my webapp, the server startup time in debug mode is close to 5-6 minutes - enough time to take a coffee break!
At times, it so pisses me off that I start cursing IBM for building an operating system instead of an app server! Spawning 20+ processes and useless services with no documented configuration to tuning it, to starting any faster.
I am sure there are many java developers out there who would agree with me on this. I tried to disable the default apps and a set of services via my admin console, however that hasn't helped much.
I have no webservices, no enterprise beans, no queues, just a simple web app which requires a connection pool. Have you done something in the past to make your integrated test environment, start fast in debug mode and there by consume less RAM?
UPDATE:
I tried disabling a few services (internationalization, default apps etc...) and now the WebSphere server went from bad to worse. Not only doesn't it take horrifying startup time, it keeps freezing every now and then for up to 2 minutes. :-( Sounds like, optimization is not such a good thing, always!

The best way to debug server code is to use remote debugging.
First you need to add the following to the JVM params in the server start script:
-Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005
This will cause the JVM to listen on the specified port, then from your IDE you can start a remote debug session against that port and debug as if the code was running in the same process.
Working this way prevent you restarting the server so frequently and hence side-steps your problem with Websphere's start-up time.
You can get some odd results if the binaries on the server and the source in the IDE get out of sync but on the whole that's not a problem.

One of the main reasons is that you have a large application with many modules, classes, manifests, XML descriptors so on, and the fact that Websphere application server start up process is single threaded per se (thus each application may be started in a separate thread if they has equal weight). One other reason is that the Eclipse EMF and JST frameworks are very I/O intensive during startup and publish/deploy.
One other reason for the tedious start up is the annotation scanning which will occur during publish/deploy. This annotation scanning can be controlled and modified in a various ways. Look at this site:
http://wasdynacache.blogspot.se/2012/05/how-to-speed-up-annotation-processing.html
First of all, examine and evaluate your hardware, both CPU, memory and HDD. Is your processor/s running 100% for a long time during start up? If so, the processor may be too weak. Is paging occur? then you may have to put in some more RAM. The Websphere/eclipse JST and EMF frameworks are very I/O intense so you should consider to invest in a SSD disc. You should also make sure that other processes on your machine (virus protection software etc.) don´t steal hardware resources from the Websphere java processes.
So for the hardware:
1. Processor - a pretty fast one, since the publish and the startup is mostly singlethreaded you do not need that many cpu cores
2. Memory - You will at least need 512Mb of physical RAM, this depends of the size of your application of course.
3. Storage - I would definitely go for a fast SSD since the underlying eclipse framework is I/O intensive.
Here are some tricks to reduce the footprint of the start up phase. Please before applying these settings make sure that you record a baseline start up so that you can observe the difference in start up, i.e. the reduced start up time.
JVM args : -Xverify:none -Xquickstart -Xnoclassgc -XX:+UseNUMA -XtlhPrefetch -Xgcthreads4 (I got 4 virtual processors installed on my machine)
Extend the heap size to match the demands of your application.
Disable the autostart of the application to reduce publish time.
Disable PMI and unnecessary tracing.
Profile your application during startup and fix bottlenecks if found any.
Other JVM arguments that may gain performance:
com.ibm.cacheLocalHost=true
com.ibm.ws.classloader.zipFileCacheSize=512
com.ibm.ws.classloader.resourceRequestCacheSize=1024  
com.ibm.ws.management.event.pull_notification_timeout =20000
com.ibm.ws.amm.scan.context.filter.packages=true
org.eclipse.jst.j2ee.commonarchivecore.disableZip=true
Jvm arguments that will make the Websphere application server to stop immediately:
com.ibm.ejs.sm.server.quiesceTimeout=0
com.ibm.ejs.sm.server.quiesceInactiveRequestTime=1000
Webcontainer properties:
com.ibm.wsspi.jsp.disableTldSearch=true
com.ibm.wsspi.jsp.disableResourceInjection=true
JVM arguments that may be specified eclipse.ini (Note that the heap parameters is configured according to the conditions of my environment)
-Dcom.ibm.ws.management.event.max_polling_interval=5000
-Xquickstart
-Xverify:none
-Xmxcl25000
-Xjit:dataTotal=65536
-Xcodecache64m
-Xscmx48m
-Xnolinenumbers
-Xverify:none
-Xmnx64m
-Xmx1446m
-Xmnx64m
-XX:+UseCompressedOops
-XX:+UseNUMA

5 to 6 mins is not normal. I use RAD and WAS everyday and get decent startup times. Which version of WAS are you running and how much RAM do you have?
If you share several workspaces and projects for a same WAS profile, consider creating a new WAS profile for your workspace.
You probably tried that but here's a simple check list of things to try on first hand. Make sure that your server settings in RAD has the following options enabled:
Optimize server for testing and developing
Run server with resources on the workspace
Minimize application files copied to the server
Uncheck "Enable universal test client" if you don't need it.
In the admin console you can verify some server settings such as
Run in development mode
Parallel start
Start components as needed
You can also uninstall the ivt app that comes installed by default when creating a new WAS profile. Then the usual things such as a drive that is not too fragmented and a pagefile size that is properly set.
And one last thing that you probably know already, republish to your server instead of restarting it.

That's one reason why Spring was born.
You don't even have to give all the niceties like JMS, remoting, etc. You'd be better off with Tomcat, ActiveMQ, and OpenEJB.
Anything but WebSphere.

There's some hints and tips for tuning RAD 6 on developerworks that may help, many of these also apply for RAD 7.
I have seen a similar list for RAD 7, I'll post it if I can find it.
I did find some tuning tips for Portal on RAD 7.
I would say my experience with the test environment has been suboptimal. I now tend to use Tomcat/Pluto configured for remote debugging with an External launch configuration to manage it from within bare Eclipse and rely on having appropriate JNDI configurations to abstract the underlying server.
If you are coding to the relevant APIs it shouldn't matter for development purposes that you're not on Websphere. If you do have a Webpshere specific issue you can always crank up the beast to debug it.

If you have no EJBs, no JMS, etc., just deploy under a standalone servlet container such as Tomcat or Jetty, you'll be amazed how fast it is :-), being ironic here but it's true!

If the connection pool really is the only appserver feature you use then why don't you simply use apache commons dbcp (http://commons.apache.org/dbcp/) drop webfear alltogether and use jetty instead. That should reduce your startup time to about 5 seconds. You can then later easily switch to websphere again for your production environment if you should really feel the need to.

WAS V7 addresses some of these problems by allowing you to configure what starts up when the app server starts up.
So if and when you migrate to WAS V7 you might seem some improvements in this space.