We Keep Coding

Java Web Application for 5000~ Users

For the first time (hopefully not the last) in my life I will be developing an application that will have to handle a high number of users (around 5000) and manage lots of data. I have developed an application that manages lots of data (around 100~ GB of data, not so much by many of your standards), but the user count was pretty low (around 50).
Here is the list of tools / frameworks I think I will be using:
Vaadin UI framework
Hibernate
PostgreSQL
Apache Tomcat
Memcached (for session handling)
The application will mainly be run inside a company network. It might be run on a cluster of servers or not, depends on how much money the company wants to spend to make its life easier.
So what do you think of my choices and what should I take caution of?
Cheers

The answer, as with all performance/scaling related issues is: it depends.
There is nothing in your frameworks of choice that would lead me to think it wouldn't be able to handle a large amount of users. But without knowing what exactly you want to do or what your budget is, it's impossible to pick a technology.
To ensure that your application will scale/perform, I would consider the following:
Keep the memory footprint of each session low. For example, caching stuff in the HttpSession may work when you have 50, but not a good idea when you have 5000 sessions.
Do as much work as you can in the database to reduce the amount of data that is being moved around (e.g. when looking at tables with lots of rows, ensure that you've got paging that is done in the database (rather than getting 10,000 rows back to Tomcat and then picking the first 10...)
Try to minimise the state that has to be kept inside the HttpSession, this makes it easier to cluster.
Probably the most important recommendations:
Use load testing tools to simulate your peak load and beyond and test. JMeter is the tool I use for performance/load-testing.
When load testing, ensure:
That you actually use 5000 users (so that 5000 HttpSessions are created) and use a wide range of data (to avoid always hitting the cache).
EDIT:
I don't think 5000 users is not that much and you may find that (performance-wise) you only need a single server (depends on the size of the server and the results of the load testing, of course, and you may consider a clustered solution for failovers anyway...) for handling the load (i.e. not every one of your 5000 users will be hitting a button concurrently, you'll find the load going up in the morning (i.e. everyone logs in).

You might want to consider an Apache HTTP server in front of your Tomcat servers. Apache will provide: compression, static caching, load-balancing and SSL.

Any reason for not using Spring? It has really became an de-facto standard in the enterprise java applications.
Spring provides an incredibly powerful and flexible collection of technologies to improve your enterprise Java application development that is used by millions of developers.
Spring is lightweight and can stay as a middle layer, connecting the vaadin and hibernate, there by creating a clean separation of layers. The spring transaction management is also superior to the one on hibernate. I will suggest you go for it until you have a strong reason stopping you.

Since you asked people to weigh in, I won't hold back my opinion. ORMs in general, and Hibernate in particular, are an anti-pattern. I know, I've worked in shops that use Hibernate over the past 9 years. Knowing what I know now, I will never use it again.
I highly recommend this blog post, as it puts it more succinctly than I can:
ORM is an anti-pattern
But forgive me if I quote the bit from that blog about ORMs and anti-patterns:
The reason I call ORM an anti-pattern is because it matches the two
criteria the author of AntiPatterns used to distinguish anti-patterns
from mere bad habits, specifically:
It initially appears to be beneficial, but in the long term has more
bad consequences than good ones
An alternative solution exists that is proven and repeatable
Your other technology choices seem fine. Personally, I lean more toward Jetty than Tomcat. There's a reason that Google embeds it in a lot of their projects (think GWT and PlayN); it's a younger codebase and I think more actively developed now that Eclipse has taken it over. Just my humble opinion.
[UPDATE] One more link, very long read but when making architectural decisions, reading is good.
Object/Relational Mapping: The Vietnam of Computer Science

I recommend Glassfish for application server because Apache Tomcat can serve simple content. And Glassfish has full implementation of the Java EE specification.

Depending on you specification and future goals I would perhaps leave the normal version of tomcat and go for Apache TomEE or my personal preference of jBoss. As I understand it EJB's are not very well supported in the normal tomcat version and that is probalby something sweet to have when you want to create a couple of services, some clustered singleton service and other stuff. But this is just my personal pref of course and if your specification will not allow a more advanced EE server then you should stick with the slick tomcat.

RoR web tier with Java middle/backend

I'm looking for some advice/feedback about using different languages in a layered architecture. Right now the architecture in question is an all java stack (db, middle tier, and web tier) using Spring to tie everything together. We've been tossing around the idea of using Ruby on Rails for the web tier, but I'd like to know if anyone has done this before (maybe with different languages/technologies). The idea behind switching to RoR is to increase development speed in the web tier, however my initial analysis is that it will also decrease development speed in the middle tier.
Some of the questions I'm trying to get data points on are:
Have any small development teams actually used a hybrid language approach in a similar architecture?
Why did you use that approach? What worked well and what didn't?
How would you handle interfaces and models (maybe use something like Google protocol buffers or Apache Thrift)?
Are there any other benefits or pitfalls to using an approach like this?

While I have no experience of RoR, we have taken a similar approach with Django as a front end layer, where we would use java based services for high volume transaction processing back ends.
We define the DB in a django model, develop the web front end with django views/ jquery and obviously get a for free admin site.
On the java side we run a hibernate reverse engineering task to generate DB access classes for the java code. Where required we use home crafted JSON over AMQP RPC mechanism to communicate between python and java components.
There are ups and downs to this approach. For us as we were not using any big java frameworks, it gave us access to quick / lightweight web front end to our solution as well as the ease of modelling our database using the Django model format (which I like a lot).
On the down side we did encounter some tricky stuff with hibernate reverse engineering configuration (which is complex if you are doing anything that is not bog-standard - especially in relation to some Django ManyToMany relations).
I notice you mention that you already have a Spring based architecture with a web tier. I'm no expert here but that may present you with opportunities worth exploring such as groovy/grails or the SpringMVC which in version 3 claims to be much simplified and faster to develop against. This may give you the wins you need.
A compromise might be to look at some RoR inspired Java tools like Rife.
Bottom line is that you will have to make the call in light of what your own application requirements and team capabilities are. It worked for us and was I think the correct decision for us but if your team is already tooled up with Java & Spring skills then Spring 3 might be the way to go (but ask a Spring expert about that :-) )
Rob

This presentation will probably be interesting to you: Rails on Spring - Using JRuby as the Secret Sauce

recommendation for choosing a new web development stack

I work in a medium to small team ( 10 people ) developing and supporting several web enterprise applications.
We have a dozen of them built with a house-made framework with asp-classic working against ms-sql server.
We are evaluating the migration to a new development stack.
We'd like it to be open (free) and simple.
I've been looking around the java web frameworks, but all of them seem to be extremely overbloated for our needs (with the possible exception of http://www.playframework.org/, which I couldn't study yet...)
We are thinking about porting our own framework to this new stack, rather than adopting a whole new stack that we are unaware of ...
so far now, we though about the following possibilities
plain java - jsp - jsf
groovy - gsp (no grails at all)
jruby (no rails at all)
we feel really comfortable working with dynamic languages (well, as dynamic as classic asp can be) and with a lean and understandable framework...
I see no small and simple web frameworks for java, like there are for php or ruby...
I really like groovy, but I see no web implementations outside of grails... Besides the language documentation doesn't seem to be quite complete (I might be looking in the wrong place, perhaps)
php could be an option, but I think it would be hard to advocate for it in my current work...
any other option, advice, pros and cons?
thanks a lot
--
edit
some related link Can anyone recommend a simple Java web-app framework?

I'd suggest you take another look at Grails. It does use hibernate and spring under the covers, but for most situations, you don't need to know the details of those frameworks. There's a large community and lots of documentation/blogs/mailing lists for support, as well as a thriving plugin community with over 300 plugins solving pretty much any need.
If you're still put off by grails, you could look into the play framework. I don't have any experience with it, but there has been some traffic recently around it on hacker news and the like. I know it uses groovy for the templating language.

I cannot recommend anything, but strongly recommend that you consider these things:
Rapid development. Basically you want to save a page file, and reload it in the browser. Instantly! It can be done, do not settle for long deployment times.
Plain, readable text files!
Convention coding instead of explicit coding - big XML files will eventually drive one or more developers insane. The less, the better.
Good tool support (just having syntax coloring may be a big help)
Consider the long term support of your choice. You are basically remarrying with your software - will it still be maintained in 10 years? By whom? Will you have alternatives (JSR's are great - look at the amount of servlet engines)?
And WHEN you choose - get the source code for it, and ensure that it builds correctly. It will never be easier than now, and some day you WILL need to fix something inside. On short notice! (You may even consider allocating resources for donating documentation/patches/time to the open source project you are building your business on).
EDIT: A few more things:
You want to be able to verify things at compile time. One of the things that make it possible to build cathedrals in Java is that the static typechecking prevents a lot of nasty runtime errors. "Oh, THAT method? Well, it's not here, sorry. Boom!"
You want good error reporting. Built in! Try throwing a NullPointerException deep, deep down and see what 1) the user and 2) the developer is told about it. Anything that requires going to a log file to get the details WILL cause calls at 3 AM eventually.
Look into scalability from the start. Any non-trivial customer will need to and the world goes to multicores, so you might as well think about it already now. What will you do when all the magic pixie performance dust has been used and it just isn't enough: The application requires more than a single box.
And read this: http://www.pragprog.com/titles/mnee/release-it

You're forgetting about the other major player in this field: the LAMP stack (linux, Apache, MySQL and mod_perl). All components are free, there are many books available on LAMP development and each of these components, and there are vast numbers of libraries and components already available.
Apache: the Definitive Guide
Learning Perl: by SO's brian d foy
Practical mod_perl

If you are afraid of Grails and need Java, try Stripes and read the excellent Stripes book (http://www.stripesbook.com/blog/). You can buy the eBook pdf for $23. The book covers the framework in amazing detail. Stripes is a very strong, lightweight MVC framework that deals with all the common problems of web development (templates, url mapping, form validation, security, internationalization, testing) but it won't automagically create the database layer for you unless you want it to by using Stripernate. You can also use Groovy with it. You can use it standalone or with Spring.

I've had great success in simple web projects using Spring MVC with JSTL JSPs. Spring MVC is a framework that can be kept pretty darn simple (1 additional XML file used for configuration). You can eschew all the fancy options and just specify a set of JSPs that you want to associate with view names, then forward to those views by specifying their names in the controller.
Spring MVC can also easily scale up and be as complex as you need, letting you switch from JSTL to JSTL with Tiles, or Struts, or JSF, or Wicket. It can also handle complex web flows using the Spring Web Flow project. But for most projects I just keep it simple -- build a JSTL JSP, create a controller that provides the objects that JSP needs, and associate them by having the controller return that view. Once you get the project set up and you're familiar with the configuration, it takes maybe a couple minutes to wire a new page into place.

If you like Groovy but don't like Grails you could try Gaelyk, which is a lightweight Groovy framework. However, AFAIK you can only use Gaelyk if you're hosting the app on the Google App Engine
If your apps won't be hosted on GAE, and you really don't want to use Grails, another option is to use Groovlets, Groovy template servlet, GSPs.
However, personally I think it's a big mistake to dismiss Grails. It really is a great framework, and you can go a long way without knowing much about Spring and Hibernate. One of your complaints is a lack of Grails documentation. I think you must have been looking in the wrong place, because in addition to all the books available, there's a very extensive reference document and a lot of other documentation available on the website. Finally, there's a very active mailing list.

My platform of choice is JRuby - Rails (3) because of its very rich and powerful ecosystem, but mainly because:
* very easy to use
* many MANY libraries
* fast support via IRC
* deep documentation
You can also check out Scala + Lift Web Framework ( imho best static typed language, nice framework )

Does it make sense to use a framework for a simple java web app?

I've done lots of java web development using jsps and servlets, and I have found this approach to be straightforward and flexible. Some of the groundwork involved though - such as managing database connections - is rather tedious, and it takes a fair amount of work just to get a new web app off the ground.
I'm therefore considering using a framework for the first time, but my impression of frameworks is that they're mostly used for large J2EE applications and involve a lot of complex configuration. What I'm looking for is something simple that (after the initial learning curve) will enable me to get up and running with a new web app as quickly as possible.
So my question is - does it make sense to use a framework for a simple java web app?
Note that I'm not asking which framework to use (if indeed a framework is recommended), as it has already been asked here.

If you don't use a web framework you'll usually end up writing one - poorly.

It makes a lot of sense. My team has spent the better part of five years with our open source stack and no matter and we have a "seed" project (works like appfuse) that we use to create all new web apps. Even the simple two pagers from the pov of maintaining the app, it looks like every other app, just smaller.
The short is you won't get any return on the investment right now, but you will as the project evolves and you maintain it.

Yes, I would use a web framework for the following reasons:
Increased navigation capabilities and controls. Even though you may not need them, they are there for you to use should you require them at any time
As others have indicated, apps grow over time and you will feel the need for the framework at some point in time in the future. When you need to add additional pages and navigation
Features that allow you to plug into other frameworks such as security and DB access frameworks. Spring is a prime example in the Java world. You never have to use Spring but it plugs in so well with Struts, Spring MVC, Hibernate, Acegi etc. It ends up saving you the hassle of doing all of the plumbing on your own.
Support!!! Good frameworks almost always have a vibrant community to support them and ask and answer questions.
It may seem like too much hassle initially but definitely saves you much time in the future

I'd say it actually matters quite a lot which framework you go for. Something like Spring MVC is fairly unobtrusive in your code and allows you existing stuff to run quite a lot as-is. Other frameworks have much more specific ideas about how you should do things.

Yes it does make sense. Apps can grow and change and might require something a framework can provide easily in the future.
For example at my workplace we have simple jsp / servlet app. It needs to be rewritten just because of what I explained above. If someone would have taken the time to just get the framework setup we would be in better shape today.

Yes it makes sense. However implicit in your question is that the wrong (for you) framework can be more pain than it's worth - and that's true. There is a world of difference between some heavy J2EE framework and something light and cheerful like grails.

What's the alternative? Rolling your own? Embedding all the navigation and logic in the JSPs?
I agree with those who say that a web framework is worth it. There are literally hundreds now (e.g., Struts, JSF, Spring, Wicket, etc.). Pick one that suits you.

There are two kinds of applications:
1) the kind that you throw away and never use again, and therefore should not worry about modularity, maintainability and clarity.
2) the real kind.
It may sometimes seem that your app may never have to grow, scale, or service a larger feature set / user base than you currently are planning for....I assure you, that perception is always wrong.
Frameworks, specifically things like Struts for MVC in Java, Spring for MVC and Dependency Injection, Hibernate for Object-Relational Modeling are all extremely valuable tools that lead to modularity, maintainability and clarity in your code. So, to answer your original question.... Yes, emphatically.

No it doesn't unless:
you don't have any java experience
in your team
you are doing just a prototype that
must be ready in hours
you don't trust your developers
enough to let them write your
application
you don't plan to let your
developers learn and improve
Danger in using frameworks include but not limited to:
you'll get all framework defects on
top on your own
you'll be unable to do realistic estimates until you
know perfectly the framework
your team won't learn how to program a
web server
you'll find your team passing more
and more time on google instead that
solving problems by writing code.
The only thing that is worst that choosing a open source framework is having a separate R&D team in the company who should create the "big complete ultimate company framework".

but my impression of frameworks is that they're mostly used for large J2EE applications and involve a lot of complex configuration
Not necessarily true. The good frameworks are built to scale well so that they will take you from small apps to very large apps. Many of the frameworks today are moving towards zero configuration so you will find them easier and easier to use.
You are right that it does take an initial effort to learn the framework itself but that investment pays for itself in the very first application you build. And meta frameworks like AppFuse makes it even easier to get started since it pre-configures the frameworks for you.

Frameworks make sense for most applications. It may be necessary for you to build your own or adopt some other framework. The most important questions is the granularity of your data structures. What I mean by this is do you need to just enter data or do you need to parse, compile and execute dynamic code?
If you think of a frame works as a gauge and to the left is an all ready existing frame work open source or closed and to the right is all custom framework then super impose your code on top of that frame work where the level of complexity increases to the right. The farther right you go the more of a stuggle you will have with the frame work (IMO).
However, you can also slowly migrate from existing to custom framework.
There is also the question about your business. If you work for a business that does not look at software as it's core compentency such as a bank or hospital then you need to way that into how much of a frame work you want to build.
Bottom line a framework of some type will always be useful.

Using a framework may add some overhead to your web-app development due to the learning curve. However, depending on the framework you choose, you may be able to realize the following benefits:
Scalability
Maintainability
Clear Division of your model (e.g. MVC)
Out-of-the-box components(SessionManagement,authentication,etc )
Third party plugins
Modularity inherited from the framework
Many others.
Besides, the learning curve is a variable factor. Some frameworks may be easier to learn than others depending on your skills.

It definitely makes sense to look into using a framework even if for no other reason than it'll give you a new area of knowledge. If you've got the time to invest in learning a framework then you may well find in time that it's easier and faster to use that for new projects no matter how simple they are.
Also, if I could vote for SamBeran's post then I would. Getting started with a framework will involve several moments of "oh great! I don't need to write all that boilerplate again!".

I'd say use one - like you said, it's rather tedious to get stuff up and running for even a simple java web app. If the framework helps you get the job done faster and provides the services you need, I'd say it makes sense.

Using java (desktop) codebase in a webapp

I have a rather large (80k loc) java desktop app that talks to a database. We're now looking at exposing some parts of the database via a web application, using the existing codebase and preferably not having to modify it.
I have good separation between the data access, business logic and presentation layers, but we haven't used enterprise java beans or anything like that (if that's important).
What's the best way forward? Which of the java web frameworks will be best suited to the problem? Learning curve isn't terribly important, since I haven't done any java development on the web...

To be true, it depends what you already have, and how well is the design of your current desktop application. You might not be able to use any or may be minimal of your existing code without modifying it, if its designed badly, and everything is tightly coupled.
Assuming that you are having a system with a good design, everything is de-coupled well enough. You can look into Stripes to make your presentation for the web, and use your existing data access and business code. I wish you all the luck.
Few other goodies to look into are, Groovy on Grail, Wicket.
I don't recommend anything like Seam and Spring they are more of a container and sophisticated large frameworks, which give you almost everything, solution for almost all of your problems. As you mentioned that you already have a complete system, and you just need to make a web interface to publish it for the web, these are not recommended, IMO.
JSF, is a good framework, but it might drive you nuts and has a big learning curve, according to few folks.

The two frameworks I would recommend would be Grails and Struts 2.
Grails comes with a whole bunch of stuff that it configures under the covers including Hibernate and Spring. It makes generating dynamic pages to send to the browser ridiculously easy. What you are probably going to need to do is set up controllers that call Grails services which reference your existing code as you probably don't want Grails managing your database interactions. The disadvantage with Grails is not so much that it is written in Groovy, which is easy to learn for Java programmers, but that the IDE support for Groovy is still maturing. Still if you want quick productivity this is the route to go down.
Struts 2 offers a clean command pattern framework implementation that talks to JSPs (or velocity or FreeMarker templates) on the front end. To use this you would configure actions to call your existing code. You may want to investigate adding Spring to the mix depending on what you need to do.
There are other choices but these are two that I have had some success with.