I am using Eclipse WTP for a project. It requires few libraries to be used. Few are small in size and few are larger. My question is, what happens when we use external jars in servlets ? If I am importing a heavy-weight library in a Servlet, does it impacts webpage load time ?
Or Java just compiles my program including libraries to give results. I understand heavy-weight jar will take time to load once even on my local machine, but I can manage them to be initiated only once (by creating a separate class and initiate static variables and use them from other classes). But seems like, this can't be done in Servlets and every time a page is loaded, servlet has to load all those heavy jars.
Is it good to use Guava and Solrj in Servlets ? Do they slow down (asking because I feel Solrj is slowing down webpage load time) ?
Including a jar in-and-of itself does not slow down servlet run time. However, using a particular tool/class/functionality in a jar may slow down the servlet, depending on what you are trying to do.
I recommend using a profiler to analyze your code and actually determine what it causing the slowdown. Here's a quote from Martin Fowler's Refactoring:
I had speculated with various members of the team (Kent and Martin deny participating in the speculation) on what was likely wrong with code we knew very well. We had even sketched some designs for improvements without first measuring what was going on.
We were completely wrong. Aside from having a really interesting conversation, we were doing no good at all.
The lesson is: Even if you know exactly what is going on in your system, measure performance, don't speculate. You'll learn something, and nine times out of ten, it won't be that you were right!
Related
In order to learn more about testing, we're going to use a profiler on a larger project (to actually get some values and measurements) and since we don't have any large project ourselves, we're forced to use something else. Any good suggestions? Maybe testing JUnit perhaps? (not "With" JUnit)?
Edit:
Not looking for any specific data, just... something... The problem is that all of this is so new so it gets kinda confusing. The point is to get slightly accustomed to testing tools such as a profiler. In other words, there shouldn't be too necessary to know much about the actual program since the program don't really matter and the data gained isn't too significant either and is mostly supposed to merely demonstrate that you can actually get stuff out of testing. So it's a bit confusing how I should proceed since I am not used to big actual programs.
Can I just download normal java files and just run/profile them with NetBeans (or similar) without having to do or care about a bunch of stuff?
Well, I've got my standard scenario. It's in C++, but it shouldn't take more than a day or two to recode it in Java.
Caveat: The scenario is not about measuring, per se, but about performance tuning, which is not at all the same thing.
It makes the point that serious code often contains multiple performance problems, and if you're really trying to make it go fast, profilers are not necessarily the best tools.
It depends on what type of data you want to profile. But the best way to get a "larger project" if you don't have one, is to find some open source project on the web that fit with what you want.
Edit: I never profile with NetBeans, so I can't tell you for this tool, but if you don't care about the tool, you can start trying with VisualVM (included with the JDK), it's a tool for monitoring the JVM. It's very usefull, and if you already run java application (like NetBeans) you'll not need to download extra applications.
Description of the tool taken on their website: VisualVM monitors application CPU usage, GC activity, heap and permanent generation memory, number of loaded classes and running threads.
VisualVM website
If you really want to profile with some source code, a little java application with a main will do the job, but again it depends on what data/amout of data you want to profile. Maybe you can find some "test applications" written in java on the web.
I have a project whose core domain is dramatically changing. It's possible to use 50% of the core functionality from this site and just add the 50% new functionality, but I am starting to consider that maybe it might be faster to simply redo the product in Rails. Development speed is very important.
There are some things I really like about java - the performance and scalability are very good. I am not a crappy Java developer, so my apps tend to run very well - better than the Rails sites I've seen. I've always accepted the idea that people probably just throw a little more money at the problem when it comes to using Rails, which probably works itself out in the end because of the insane productivity benefits.
I am actually quite agile with Java. I know it will still take me longer to add a basic entity to the system, but I am quick at it and I don't mind it that much. At least it's easy and straight-forward to do.
What I do mind is:
having to start/stop the server just to fix a route, lazy load exception, controller is going to wrong view, etc.
putting up with the fact that unit/integration tests sometimes have different results than the production environment (because annotations on controllers can't be tested, or lazy-loading exceptions occur during asynchronous service calls, or things like that). Knowing if your Jackson is marshaling your data properly is another Tomcat-only thing because it's handled by Spring. There are lots of things that go wrong after you have tested all that you can, and this frankly annoys the crap out of me.
putting up with the occasional maven/classloader problem that doesn't rear its ugly head until you deploy into tomcat. It gives the false impression that everything is "a-okay" when you are in your IDE.
having to put more effort to do database migrations than the ruby people ever have to.
putting up with framework bugs in Spring that block (it's happened about 5 times on this project since 2009) or Hibernate. I also don't like upgrading Spring Security and having them constantly change the configuration, apis and tag libraries over and over again. This is annoying.
wasting so much time uploading 58 MB war files to the server! These take me 12 minutes to upload whenever I need to deploy changes. If I forgot to do 'mvn clean' before I upload, Spring might complain that 2 beans exist with the same name because I moved one to a new package... and then I have to re-upload the whole stupid war file again. Why isn't "clean" run by default whenever you do 'mvn package' for?!?! Sometimes these frameworks and tools use the stupidest default settings. This is just so common in the Java world.
Having to spend hour(s) to figure out where a framework wants to plug-in your own custom implementation for something. This is very annoying. You can spend 2 hours sifting through Google and crappy documentation trying to figure out how to override Spring Security's authentication mechanism for example... and then spend only 5 minutes writing the actual implementation. Of course, they wrote paragraphs upon paragraphs explaining the architecture and how awesome it is, but nobody cares. For something so common, why not just give example source code and be done with it?
Waiting 10-15 seconds for Spring to start up whenever you want to run your integration tests. This is a drag.
There are a few things I like about Java though. Role-based access is very easy to do with Spring Security. Authentication is never that big of a gain, but I like the implementation inside of Spring.
I also like Spring's form-backing objects and #ModelAttribute. These are huge wins when it comes to controllers, and I don't know if Rails can do these things. I honestly never liked passing request parameters around in every action - Spring MVC is actually a lot easier to use when it comes to this common bloat.
Being able to cache really massive structures in memory and have them stay in memory when you start the application is also highly desirable, especially for this application actually. I have an in-memory thesaurus and grammar checker that needs to get called hundreds of times per request, so in memory is pretty much the fastest option for me.
Even still, I think I could rebuild what I have in 2-3 weeks, and then add all of the new features in a few weeks using rails.
On the bright side, all of the really well-designed css, html and javascript could be ported over with very little problems.
I'd appreciate some advice on the subject before I continue.
PS: I could also go to Spring-ROO... but that would also be a considerable rework. I was never using JPA - I was using Hibernate directly. I am also not using JSP's - I am using Freemarker.
It takes more time to get good at Ruby, and Rails. I worked as an independent contractor as Spring and Hibernate expert myself, but I felt strangled by java and it's web frameworks so I decided to learn Ruby on Rails.
I would advice you to learn Ruby, from what I read you would probably master it, although get pretty frustrated with the very different way the use the ORM. I had issues with it, used to working on aggregate roots in Hibernate to the ActiveRecord one class one table kind of pattern. But hey, you could easily try out MongoDB to have some real fun.
Ruby is
less code
it's fast and scalable (slower than java on the specific tasks, but you get rid of stacks of layers.)
the problems are more often; which gem should I use. Luxorious!
a unique, big, sharing and caring open source community
nice frameworks, as Rails and Sinatra
powerful.
fun!
Would I advice you to do the project you describe in Ruby.
NO.
Not if speed of development matters. You will be slower, trust me. There's a lot to learn, it's conventions are not familiar to a java programmer and when you get stuck, lots of hours fly by.
The best option would be to hire a senior ruby developer to pair up with you and teach you. Be a good apprentice and you'll learn fast. Faster than me, I had to learn most by myself, which is really inefficient.
Good luck!
Check out Playframework. Its fun to develop, and you can use your Java experience to develop features way quicker (given than you have 2 weeks) than any other Java-based frameworks out there.
You do not have to start/stop a server. You fix the code in Eclipse and hit refresh on the browser. No dealing with WAR files till you have to actually deploy in production. Do everything from within Eclipse. Easily perform TDD process if thats what you want as you develop code. From an architecture standpoint, it is a fully stateless, RESTful framework from the get-go. Fully JPA compliant (even for NoSQL like Mongo), so you will not have to write complex JDBC code. On the front-end, it has a full featured templating engine, using Groovy as a templating language.
I can go on and on, but I'd recommend going through the site and take a look.
You should take a look at Grails.
You can continue to leverage a lot of your Java code but use a scripting language (Groovy) and many of the paradigms of Rails. E.g. lots of time saved by using convention rather than configuration.
Grails is used by some pretty big web sites E.g. BSkyB the UK satellite broadcaster.
It doesn't really help with some of the startup speed aspects. If you really prize development speed that much - get a faster machine or buy an SSD and fit in your machine. If you work for a big company - sell it to your manager as the cheaper option (E.g. buy a $2000 machine rather than spend 3 weeks rewriting something to save 10 minutes a day).
Java will scale better in the long run than Rails. The Hotspot technology in the JVM is one of the wonders of modern technology.
Also worth checking out is Tapestry5. It allows you to make code changes on the fly (no server restart required) and is easily the fastest & leanest framework to develop with in Java I've used.
I would still give Spring Roo a shot, it will the same rework as with Ruby on Rails or Grails or even less, but you will still stay with something that you are familiar with, which is often the biggest consideration
It has the scaffolding concepts of Ruby on Rails and Grails, but it gives you zero lock in code, just simple, well written (massive use of AOP is matter of taste though) of Spring + Hibernate / JPA (I think you can use Freemarker for the views, Roo has a miriad of plugins, but I'm not 100% sure)
I am curious about what automatic methods may be used to determine if a Java app running on a Windows or PC is malware. (I don't really even know what exploits are available to such an app. Is there someplace I can learn about the risks?) If I have the source code, are there specific packages or classes that could be used more harmfully than others? Perhaps they could suggest malware?
Update: Thanks for the replies. I was interested in knowing if this would be possible, and it basically sounds totally infeasible. Good to know.
If it's not even possible to automatically determine whether a program terminates, I don't think you'll get much leverage in automatically determining whether an app does "naughty stuff".
Part of the problem of course is defining what constitutes malware, but the majority is simply that deducing proofs about the behaviour of other programs is surprisingly difficult/impossible. You may have some luck spotting particular patterns, but on the whole you can't be confident (and I suspect it's provably impossible) that you've caught all possible attack vectors.
And in the general sphere, catching 95% of vectors isn't really worthwhile when the attackers simply concentrate on the remaining 5%.
Well, there's always the fundamental philosophical question: what is a malware? It's code that was intended to do damage, or at least code that doesn't do what it claims to. How do you plan to judge intent based on libraries it uses?
Having said that, if you at least roughly know what the program is supposed to do, you can indeed find suspicious packages, things the program wouldn't normally need to access. Like network connections when the program is meant to run as a desktop app. But then the network connection could just be part of an autoupdate feature. (Is autoupdate itself a malware? Sometimes it feels like it is.)
Another indicator is if a program that ostensibly doesn't need any special privileges, refuses to run in a sandbox. And the biggest threat is if it tries to load a native library when it shouldn't need one.
But all these only make sense if you know what the code is supposed to do. An antivirus package might use very similar techniques to viruses, the only difference is what's on the label.
Here is a general outline for how you can bound the possible actions your java application can take. Basically you are testing to see if the java application is 'inert' (can't take harmful actions) and thus it probably not mallware.
This won't necessarily tell you mallware or not, as others have pointed out. The app could still do annoying things like pop-up windows. Perhaps the best indication, is to see if the application is digitally signed by an author you trust; if not -- be afraid.
You can disassemble the class files to determine which Java APIs the application uses; you are looking for points where the java app uses the OS. Since java uses a virtual machine, there are well defined points where a java application could take potentially harmful actions -- these are the 'gateways' to various OS calls (for example opening a socket or reading a file).
Its difficult to enumerate all the APIs, different functions which execute the same OS action should require the same Permission. But java's docs don't provide an exhaustive list.
Does the java app use any native libraries -- if so its a big red flag.
The JVM does not offer the ability to run arbitrary code, or use native system APIs; in particular it does not offer the ability to modify the registry (a typical action of PC mallware). The only way a java application can do this is via native libraries. Typically there is no need for a normal application written in java to use native code (unless it needs to use devices).
Check for System.loadLibrary() or System.load() or Runtime.loadLibrary() or Runtime.load(). This is how the VM loads native libraries.
Does it use the network or file system?
Look for use of java.io, java.net.
Does it make system calls (via Runtime.exec())
You can check for the use of java.lang.Runtime.exec() or ProcessBuilder.exec().
Does it try to control the keyboard / mouse?
You could also run the application in a restricted policy JVM (the instructions/tools for doing this are not as simple as they should be) and see what fails (see Oracle's security tutorial) -- note that disassembly is the only way to be sure, just because the app doesn't do anything harmful once, doesn't mean it won't in the future.
This definitely is not easy, and I was surprised to find how many places one needs to look at (for example several java functions load native libraries, not just one).
I am building an app that scrapes information from web pages. To do that I have chosen to use an html scraper called Jsoup because it's so simple to use. Jsoup is also dependent on Apache Commons Lang libray. (Together they make up a total of 385kB ).
So Jsoup will be used to Download the page and parse it.
My question is if the use of these simplifying libraries, instead of using Androids built-in libraries, will make my app slower? (in terms of downloading data and parsing).
I was thinking that the internal libraries would be optimized for Android.
The next release of jsoup will not require Apache Commons-Lang or any other external dependencies, which brings down the jar size to around 115K.
Internally, jsoup uses standard Java libraries (URL connection, HashMap etc) which are going to be reasonably well Android optimised.
I've spent a good amount of time optimising jsoup's parse execution time and data extractor methods; and certainly if you find any ways to improve it, I'm all ears.
If the question is, "Will external libraries INHERENTLY make my app slower than if I wrote the same code myself?", the answer is generally, "Yes, but not very much."
It will take the JVM some time to load an external library. It's likely that the library has functions or features that you aren't using, and loading these or reading past them will take some time. But in most cases this difference will be trivial, and I wouldn't worry about it unless you are in a highly constrained environment.
If what you mean is, "Can I write code that will do the same function faster than an external library?", the answer is, "Almost certainly yes, but is it worth your time?"
The odds are that any external library you use will have all sorts of features that you don't need but are included to accomodate the needs of others. The authors of the library don't know exactly what every user is up to so they have to optimize in a general way. So if you wrote your own code, you could make it do exactly what you need and nothing more, and be optimized to exactly what you are up to.
Whether it's worth the trouble in your particular case is the big question.
The external libraries will also use the internal libraries that are optimized for Android. I guess the real question is: would your custom implementation be faster than the generic implementation of these libraries?
In most cases, third-party libraries solve the problem that you want to solve, but also other problems that you might not need to solve, and it's this part that might hurt performance. You have to find the balance between reinventing the wheel and using optimized code just for your basic needs.
Additionally, if these libraries weren't designed with the Android platform in mind, make sure to test them extensively.
It's the classical build-vs-buy argument.
If run-time performance is really important for your application then you should consider rolling out your own implementation or optimizing the library (assuming it's open source.) However, before you do that you should know good or bad the performance of the existing library is. You won't know that unless you actually use it and get some data.
As a first step I would recommend using the library and collect data regarding it's performance OR ask someone who has already used this library on Android for performance numbers. The library may be slow but if it's acceptable then I guess it's better than rolling one on your own.
Keep in mind when you create your own implementation it will cost your time and money (design, coding, testing and maintenance.) So you are trading off runtime performance for reuse and reduced development cost.
EDIT: Another important point is that performance is a function of many things. For example, the hardware, the Android version and the network. If your target device is running 2.1 or less and you may get a boost in performance by using 2.2. On the other hand, if you want to target all versions you have to adopt a different strategy.
What are common Java vulnerabilities that can be exploited to gain some sort of access to a system? I have been thinking about it recently, and havent been able to come up with much of anything - integer overflow - maybe? race condition - what does it give you?
I am not looking for things like "sql injection in a web app". I am looking for a relationship similar to buffer overflow - c/c++.
Any security experts out there that can help out? Thanks.
Malicious Code injection.
Because Java (or any language using an interpreter at runtime), performs linkage at runtime, it is possible to replace the expected JARs (the equivalent of DLLs and SOs) with malicious ones at runtime.
This is a vulnerability, which is combated since the first release of Java, using various mechanisms.
There are protections in places in the classloaders to ensure that java.* classes cannot be loaded from outside rt.jar (the runtime jar).
Additionally, security policies can be put in place to ensure that classes loaded from different sources are restricted to performing only a certain set of actions - the most obvious example is that of applets. Applets are constrained by the Java security policy model from reading or writing the file system etc; signed applets can request for certain permissions.
JARs can also be signed, and these signatures can be verified at runtime when they're loaded.
Packages can also be sealed to ensure that they come from the same codesource. This prevents an attacker from placing classes into your package, but capable of performing 'malicious' operations.
If you want to know why all of this is important, imagine a JDBC driver injected into the classpath that is capable of transmitting all SQL statements and their results to a remote third party. Well, I assume you get the picture now.
After reading most of the responses I think your question has been answered in an indirect way. I just wanted to point this out directly. Java doesn't suffer from the same problems you see in C/C++ because it protects the developer from these types of memory attacks (buffer overflow, heap overflow, etc). Those things can't happen. Because there is this fundamental protection in the language security vulnerabilities have moved up the stack.
They're now occurring at a higher level. SQL injection, XSS, DOS, etc. You could figure out a way to get Java to remotely load malicious code, but to do that would mean you'd need to exploit some other vulnerability at the services layer to remotely push code into a directory then trigger Java to load through a classloader. Remote attacks are theoretically possible, but with Java it's more complicated to exploit. And often if you can exploit some other vulnerability then why not just go after and cut java out of the loop. World writable directories where java code is loaded from could be used against you. But at this point is it really Java that's the problem or your sys admin or the vendor of some other service that is exploitable?
The only vulnerabilities that pose remote code potential I've seen in Java over the years have been from native code the VM loads. The libzip vulnerability, the gif file parsing, etc. And that's only been a handful of problems. Maybe one every 2-3 years. And again the vuln is native code loaded by the JVM not in Java code.
As a language Java is very secure. Even these issues I discussed that can be theoretically attacked have hooks in the platform to prevent them. Signing code thwarts most of this. However, very few Java programs run with a Security Manager installed. Mainly because of performance, usability, but mainly because these vulns are very limited in scope at best. Remote code loading in Java hasn't risen to epidemic levels that buffer overflows did in the late 90s/2000s for C/C++.
Java isn't bullet proof as a platform, but it's harder to exploit than the other fruit on the tree. And hackers are opportunistic and go for that low hanging fruit.
I'm not a security expert, but there are some modules in our company that we can't code in java because it is so easy to de-compile java bytecode. We looked at obfuscation but if you want real obfuscation it comes only with a lot of problems (performance hit/loss of debug information).
One could steal our logics, replace the module with a modified version that will return incorrect results etc...
So compared to C/C++, I guess this is one "vulnerability" that stands out.
We also have a software license mechanism built-in in our java modules, but this can also be easily hacked by de-compiling and modifying the code.
Including third party class files and calling upon them basically means you are running unsecure code. That code can do anything it wants if you don't have security turned on.