I am writing a microbanchmark in Java, following the ‘rules’ on this page. The last rule states:
“Reduce noise in your measurements. Run your benchmark on a quiet machine…”
So now I’m wondering how to create a system that is as quiet as possible. Is a clean installation of windows 7 with unnecessary services disabled a good example of a quiet system, or are there better options?
A quiet machine is good, but perhaps unrealistic.
You should test a system which is as close to how you intend to use it as possible.
e.g. you should run the same hardware, OS and services as your target system.
If possibly run it on your target system, possible after hours if needed.
Linux gives you a lot more control over what runs on the system (e.g. it's trivial to run without a GUI, which I don't think is even possible on Windows), so if you want an ideal synthetic benchmark environment, it would be a better option.
But a clean Windows install (important: no virus scanner or other preinstalled crapware) with most services disabled should be good enough as well.
Related
I'm looking for a way to boost my team's productivity, and one way to do that would be to shorten the time it takes to compile & unit test & package & deploy our Java EE application which is getting bigger and bigger.
The trivial solution that I know of is to set up a powerful computer with N processors (N ~= num of developers) and a blazingly fast disk system and a lot of memory, and run everything on this computer and connect to it via X remotely. It would certainly be much faster than compiling on our laptops, but still cheaper and easier to maintain than to buy each developer his/her own supercomputer.
Is there another way to solve this problem? For example, could we run our IDEs locally and then tell it to remote compile java source? Can Netbeans / Eclipse / IntelliJ / etc. do this? Or is there a special tool that enables remote java compilation, also that makes use of multiple processors? It need not be free/open source.
Unfortunately our laptops MUST run a (company managed) Windows Vista, so another reason to go for the separate server computer is to let us use linux on it and finally get rid of the annoying managed environment.
EDIT: to sum up the answers so far, one way to shorten build times is to leave compilation for the developers individually (because compiling is supposed to be fast), skip running unit tests and hot-deploy (without packaging) to the container.
Then, when the developer decides to check his/her code in, a continuous integration server (such as Hudson) is triggered to clean & build & run tests & package & deploy.
SOLUTION: I've accepted Thorbjørn's answer since I think that's going to be the closest to which way I'm planning to proceed. Although out of curiosity I'm still interested in solving the original problem (=remote Java compiling)...
You essentially need two workflows.
The OFFICIAL build, which checks out the sources, builds the whole thing from scratch, runs all the unit tests, and then builds the bits which will eventually ship to the customer after testing.
Developer hot-deploying after each source code change into the container the IDE knows about.
These two can actually be vastly different!
For the official build, get Jenkins up and running and tell it to watch your source repository and build whenever there is a change (and tell those who break the build). If you can get the big computer for building, use it for this purpose.
For the developers, look into a suitable container with very good IDE deployment options, and set that up for usage for each and every developer. This will VERY rapidly pay off! JBoss was previously very good for exactly this purpose.
And, no, I don't know of an efficient remote java compilation options, and I don't think this is what you should pursue for the developers.
See what Joel thinks about Build Servers: http://www.joelonsoftware.com/articles/fog0000000023.html
If you don't like Jenkins, plenty others exist.
(2016 edit: Hudson changed to Jenkins. See https://stackoverflow.com/a/4974032/53897 for the history behind the name change)
It's common to set up a build server , e.g. running hudson to do the compiling/packaging/unit-testing/deploying.
Though you'd likely still need the clients to at least perform a compile. Shifting to using a build server, you might need to change the work process too if you arn't using a build server now - e.g. if the goal is to take load off the client machines, your developers will check code in , automatic unit tests gets run, instead of running unit tests first, then checking in.
You could mount each developer dir with ntfs on the powerful machine and then create External Tool Configuration in Eclipse (GUI access), that would be triggering build on external server.
JavaRebel can increase productivity also. It eliminates the need for redeployments..
You can recompile a single file and see the changes being applied directly on the server.
When things start getting too big for efficient builds, it may be time to investigate breaking up your code into modules/JARs (how it breaks apart would depend on many project specifics and how your team tends to work). If you find a good setup, you can get away with less compiling (dont always need to rebuild the whole project) and more/quicker copying/jaring to get to the point where you can test new code.
What your project need is a build system to do the building, testing and packaging for you. Hudson is a good example of such a continuous integration build system.
Can anyone point me out to a project out there that I can download and run it and it would load / stress test itself and then provide me with reports? I want the project to be as big as it can and to involve as many components in java as it can, also i need it free... or to some very good already made results over the web that I can already take a look and get decision. Thanks!
main issue to benchmark is which would run it faster / better, solaris or linux
Linux and Solaris are not much different seen from your perspective, and I do not believe that the benchmark you ask for exists. A much better approach is to take the application you want to run - which hopefully should be platform independent already - and deploy to the architectures you want to test and then attach with jvisualvm and apply your standard test suite.
This will give you quite a good look at the performance without skewing with heavy profiling.
My guess is that for identical configurations you will see that Linux is slightly better than Solaris as the amount of unused memory available for disk caching will strongly influence the performance of the system. Also note that expert system tuning can also make a big difference, but that I believe that you are most interested in the "out of the box" performance.
It really depends on the aspects of usage you want to benchmark.
I did this for database applications, in this area TPC could be helpful.
I would recommend google: benchmark java numeric|transaction|rendering|olap
Depending on the characteristics of your use case.
Edit: regarding your comment of a java app running on an applicationserver, check from the backend db server whats the maximal throughput? TPC, then write a multithreaded benchmark client to checkout whats the business logics performance. The last step would be to involve webservers using Apache JMeter. This procedure allows you to tune all relevant parameters from OS over DB-Poolsizes etc.
Is Java completely Platform Independent ?
if not then, what care needs to be taken to see that your code written in Java can run on Multi Platforms. Basically it should work on Targeted Platforms like Windows (Various versions), Linux (all flavors), Mac and Solaris.
While in practice, most compiled byte code is platform independent, my experience in my 12 years of developing on the Java platform has taught me that there are still idiosyncrasies from platform to platform.
For example, while developing a Java 1.4 Swing application for PC and MacOSX the behavior of dialogs was different if the parent frame is null.
Another example might be with working with the file system and files in general. The Java API has methods to help shield the developer from differences in path separators (/ vs \). When writing to a file, it important to use the FileWriter API as intended so that return characters and such are generated properly for the platform that it is being written on.
So while the motto is "write once, run anywhere" my experience has been for production envs it is write once, test, everywhere.
As a result, having strong unit and integration tests can help with this, as you can execute those tests on the various platforms you want to distribute your software.
Despite some minor issues here and there, it is cool to see your code running on Linux, Unix, Windows and MacOSX (BSD Unix) using the same JARs.
As djacobson pointed out, the answer is a qualified "yes". For the most part, Java developers don't have to worry about platform dependencies. However, you may run into problems when you're dealing with APIs that handle traditional OS and platform functions.
When dealing with File I/O, for example, it's easy to make your code platform dependent by ignoring the differences between file/path separators across platforms (i.e. using '\' rather than File.separator).
For the most part, yes. Because Java compiles to bytecode that's executed by its virtual machine, it can generally be expected to behave the same way regardless of the system sitting under the virtual machine.
However. Not even virtual machines are immune to bugs. A quick Google search turns up the following, for example:
http://www.ibm.com/developerworks/java/library/j-diag0521.html
Differences in behavior can vary from JVM to JVM. Hopefully you won't end up with code that depends on any of these cases... but careful research is worthwhile to know what the limitations of your infrastructure are.
You problem will not be executing your code, but more likely the assumptions you have to make about file paths, available external commands (if you need them), necessary file permissions and other external factors that don't really fall under the "Java" problem domain. Unless you're planning on using native code (via JNI) extensively, Java will not be your problem, your environment will.
Which brings us back to the old adage: "write once, test everywhere".
Threading priorities is one thing to consider. Other OS like Solaris for example has more thread priorities than windows. So if you are working heavily on multi-threading, OS is something that may affect the program's behavior.
The main thing to be concerned with is UI code, to make sure that it is represented properly on all the platforms you will be running on.
Another source of possible issues is deploying to different app servers. There might be incompatibility issues between them.
Java other than that is platform independent, This is also one of its weaknesses, since you are coding to a common denominator and many features of each individual OS are not available.
There are very few and they should be pretty obvious. like System.getProperty("os.name") is clearly OS dependant or it wouldn't work. The most common one is System.exec() as it calls another application which is on your system, again you should know if the application you are calling works the same on every system or not (unlikely).
Along with the above concerns, the main problem I had was actually building on different platforms, which may not be what your asking, but may be something to watch out for.
OS X is especially guilty of this when using the Apple Distribution of Java (why anyone would want to put out their own packaging of Java I don't know but that is a separate argument, and on OSX i dont think you have a choice but to use their java). The Libraries that you may or may not be relying on are in completely different directories, eg libraries instead of lib if my memory serves me correctly. And the IBM java I think packages Classes in different Jars in some cases. Ridiculous!!
Hope that helps.
I created a Java application which is the only application active on a workstation. (Similar to a kiosk system)
The problem is that the application has to be up and running as fast as possible after starting the computer.
I'm wondering which one of the major operating systems can be configured to provide the shortest startup time?
I'm using 3rd party audio and graphics libraries so my choices are limited to Windows XP/Vista, Linux and Solaris.
Currently on my dual-boot machine Fedora takes a little longer than Vista, but on the other hand I don't have much experience with tuning boot time of Linux. So if someone knows that Linux could have much better chances of a quick startup then I would put my time in there.
I'd also appreciate general hints on tuning boot times and Java startup times.
I would look at BootChart to optimise your Fedora boot time. If you're running one app, then you can certainly remove a lot of the services that Fedora would normally come configured with.
I would also point out that perhaps for the amount of time you're going to spend optimising this, you may be better off investing in the appropriate hardware (e.g. SSDS and similar, if boot time is governed by your disk). Optimising can be a major time sink.
If you're running your application inside of a kiosk like machine where you don't need any other applications running, and you know which drivers/modules you'll need to load ahead of time, I think your best boot time will come from Linux.
It will just take some time to fine tune your boot process to load all the proper software in the fastest time possible.
For such a task a fine tuned Linux is best suited. You can take a look at some more customizable distro, where you can control which drivers and applications get in.
Debian is highly modularized and customizable, so you can get really good boot speed.
Another option can be Gentoo - there you can strictly choose what to compile and include.
Linux with SSD drives.
I'd also suggest a linux distro. E.g. Gentoo with initng (initng.org). Initng parallelizes the startup process. There are other startup system with which your system will be up in a few seconds.
And of course, fast hdds and enough ram for java ;)
My guess would be Windows XP embedded. I've found that Java apps start up fairly quickly under Windows, particularly if you use a client VM.
It is extremely likely that your 3rd party vendors will support XP embedded (particularly if you are a big customer to them). It is very similar to normal XP, just cut down.
If you're making a kiosk type app, why do you care about boot time?
Fedora can be easily optimized if you want to only run a single java application. There are many services which are pre-configured during boot time and they can be omitted. You could also go for SSD drives to improve the boot-time of the system, and at the same time if you spend some time on optimizing the boot chart, it would solve your problem.
I'm coming from the .Net camp where virtualization was much more prevalent do to the need to run on server software and system-wide entities such as the GAC.
Now that I'm doing Java development, does it make sense to continue to employ virtualization? We were using VirtualPC which, IMO, wasn't the greatest offering. If we were to move forward, we would, hopefully, be using VMWare.
We are doing web development and wouldn't use virtualization to test different flavors of server deployment.
Pros:
Allows development environments to be
identical across the team
Allows for isolation from the host
Cross-platform/browser testing
Cons:
Multiple monitor support is lacking (not in VMWare?)
Performance degradation - mostly I/O
Huge virtual disks
One possible advantage is that you could technically test out the same program on a variety of operating systems and on a vartiety of JVMs.
Contrary to popular opinion, Java is not 100% portable, and it is very possible to write nonportable code. In addition, there are subtle versions between libraries. If you work with different JVMs, there could also be differences.
However, since Java IDEs are heavyweight, running an IDE within the VM may not be fun.
Java does support some forms of remote deployment, might be beneficial to explore those while still doing the IDE work locally.
I don't like developing in a VM. The good thing is, in contrast to what you're writing as cons, that multiple monitors are supported by VMWare and the huge disk thing isn't really a problem since VMWare runs surprisingly smoothly from USB hard disks.
Running the heavyweight IDEs for Java, as Uri said, won't be much fun in a VM. But then, running Visual Studio in a VM isn't really fun as well. So if you were happy with VS in a VM, then give it a try for Java, because the cons aren't as strong as you might think :)
You said your doing java web development so it makes sense to test your application using different web browsers on different operating systems. VMware will be useful for this.
The Netbeans IDE is operating system independent, so you can have developers working on different operating systems with out any trouble.
I run eclipse inside a VirtualBox instance and it works fine. I've used VMWare in the past and that's fine too.
I like having my development environment segmented away from whatever the rest of my PC is doing (playing games, surfing the web, reading email, etc...)
I work from home so virtualization provides necessary separation of work/play. It also allows me to upgrade each environment separately and have much more control over the environment.
Also I can safely try something new and revert if the install goes "wonky". Sorry for the highly technical term. ;-)
Edit: It also allows me to satisfy the corporate VPN access requirements without subjecting my home environment to excessive corporate influence.
If you need a VM to verify that the app will also run on a different OS, you can cover quite some ground by using a continuous integration server and start/run VM instances on that machine (ie. Linux / Windows / OSX). Then unpack the latest build, run unit-tests on the delivered classes.
Then run automated integration tests. You will then have to report the results back to the CI-environment.
If the integration tests are good, this can catch a lot of the common multi-platform mistakes right after they are committed to the SCM.