We Keep Coding

Modern HTML/CSS/JS layout together with a Java backend

Introduction
I am basically coming from two sides of the development teams. I started building web applications using PHP. This way, I had to implement everything by hand. Forms, graphical elements, how everything is tied together (i.e. what happens when you click a button).
On the other hand, some time ago I entered the Java Web world, namely JSF. Which is in my opinion a nice way to build a "serious" applications. (why I think so, read below)
Problem
I would like to build a web app that is made of a nice and useable frontend that is designed after my specifications (basically doing the HTML and graphical layout as a horizontal prototype first - as far as possible -, then make a complete webapp out of it). But on the other hand I would like to do all the business logic in Java because of its many nice frameworks out there - Hibernate, Morphia, Lombok, Dozer, just to name a few I met and fell in love with.
I could build the frontend with JSF myself but I still think, that I'm too limited because I have to use the built-in components that can only be customized to a certain amount. What if I want to use my own design patterns and UI elements and behavior but still want them to work together with a Java "backend"?
What would be the "best" choice of tools to achieve my goal? Choose a different web framework like Wicket or similar?
My idea: Build a web application in Java that understands the good old GET commands and translates it to the Java methods. Sort of a RESTful service - although I'm not a great fan of making everything RESTful in the frontend - unless you tell me, that's fine and reliable.

Have you considered / looked at Google Web Toolkit for what you're trying to craft?
GWT is Java behind the scenes yet still can get you where you want to go, if I understand what you're asking.
Just an update:
If GWT isn't your thing, take a look at the Groovy / Grails route.
Just some elaboration here based on a comment...
With Groovy/Grails you don't actually have to merge your front and back end. I have seen that done, but its not necessarily always 100% required.
For example, you could create a web app in Grails where your AJAX functions could be formed and stored with the UI code being delivered out to the client for Great-UI-Justice. For added fun, you can even instruct Grails to include a JS framework such as jQuery to help you out with the UI.
Now on the back end, you should be able to take your Java back end and apply it directly with little to no change. However, I would recommend looking into adjusting anything you've done so far to take advantage of the Grails methodology.

There are many JSF libraries like ICEfaces,
RICHfaces etc, that make use of things like AJAX, etc.
In my personal opinion it's always a good idea to have a RESTful API and treat your web page just like any other client ;)

What are my options for building a rich web application?

I'm working on a client-side Java application for which I want to create a web-based user interface (which will be served up to a browser from the local app). At the risk of oversimplifying, its a microblogging tool that will support functionality similar to Twitter and Facebook. Its open source, and you can read more on our wiki.
Its important that this app is extremely easy to use, and I also want it to look good relative to contemporary web applications (such as the examples available on the Chrome web store). Since I am not a web designer, I'm hoping to use tools that give me an attractive UI "out of the box". Oh, and I need to keep the distributable file reasonably small (un him so whatever I use needs to be reasonably lightweight.
So far, I've looked at Sproutcore and GWT.
Sproutcore apps definite have the appearance I'm hoping for, but I'm not that familiar with JavaScript, and after some experimentation is not entirely clear to me how I will achieve some of the more sophisticated functionality that I think I'm going to need.
Being Java, GWT is more within my comfort zone, however I had a hell of a time just getting my development environment set up properly. It seems that GWT wants to dictate the structure of my application, but since the user interface is only a small part of it I'm not willing to build the entire app around GWT.
Something else I've been thinking about is using CoffeeScript (which seems more palatable him and him him him him him than JavaScript, and there are Maven builders for it), perhaps together with JQuery-UI.
What other options should I consider?

To recommend two big javascript frameworks Sencha and Cappuccino.
These have an extensive UI toolkit out of the box. The latter has an MVC framework build in, the former has a general JavaScript library build in. Both come highly recommended.
Also as recommended jQuery UI is a reasonable toolkit. I would personally couple Sencha for its rich UI toolkit and a lightweight MVC framework like Backbone. I'm afraid this does require learning JavaScript well.
You can also instead drive your JavaScript UI toolkit mainly from the server and skip the clientside MVC. For that sencha should surfice as a UI toolkit, but if you want meaningful functionality you will need some more structure. I would also recommend RequireJS for modular file management.

Take a look on Vaadin its based on GWT but simpler to use. I use it since 2 years and i'am not a designer too.

Take a look at jQuery & jQuery UI (jQuery Modile if you are looking to deploy to hand held devices)
Also, since you mentioned that you are not a designer, you might want to implement the front end in an MVVM pattern to separate the UI from the business logic of your applications. Knockout is a nice way to do with with decelerative data binding

I don't like writing plain JavaScript, so I'll tell you little bit about Java based frameworks. Recently, I've picked Google Web Toolkit.
Pros:
plenty of materials
Eclipse + Google plugin
Very intuitive API
3rd party libraries
Fancy widgets + optimized JavaScript
lot of generics
very active community
awesome async callbacks
Cons:
very slow debugging, that will drive you mad sooner or later
lot of deprecated code for so young framework
Mixed feelings:
You don't have to write declarative UI (you can do Swing-like development), but your code grows bigger and less maintainable, so it is quite probable that you'll eventually learn some declarative UI like UI Binder
Visual designer for GWT is somehow nice, but very unstable (eclipse) :(
Some considerations:
Do some prototyping with Vaadin. Try some debugging. Then try GWT and debugging on similar sized project. Consider debugging as the VERY important decision factor, because GWT debugging was the most frustrating thing for me.
Consider JSF + some nice rich component framework for JSF like RichFaces, IceFaces, PrimeFaces etc (but you app won't be "light" and "small" anymore if you really care). Be sure to be perfectly comfortable with environment (IDE + plugins), nothing beats JSF there (I'd pick RichFaces + Seam today, if full Java EE is option).

If you don't want to learn javascript I recommend you GWT with Sencha-GWT (GXT), you can develop all your app in java language, and also debug your code in java.
It is very easy to create a web app with this technology, and provides almost all funcionality available in javascript version.
Also you can mix it with hibernate and giled to persist your data, and giled helps you to serialize hibernate objects.

I have been writing an application in SproutCore for the front-end and a Java backend. In my mind this works perfectly.
The backend can stop worrying about HTML and JavaScript contents on the client side, if the request is XHR or plain request/response. I can implement the backends using plain old Java Servlets, and only worry about the data that is passed between the client and the server (JSON for the most part).
The client becomed fast and responsive, and SproutCore has a rich amount of components that you can mix and match.
So in other words with SproutCore (and Cappuccino and other full-blows MVC JavaScript frameworks) you can let your client do what it does best : Render and display HTML, CSS and JavaScript, while leaving your server to worry about what it does best: retrieve, store and update data. If you are interested, the source code for this project is available with the GPLv3 licence: https://github.com/joachimhs/EurekaJ
I have written an article about the differnt approaches between server-side MVC and client-side MVC here: http://haagen.name/2011/05/24/The_RIA_MVC_Model.html.
Also SproutCore comes with the MIT licence, so its avaiable for use without (m)any restrictions.

Google Web Toolkit -- only for Gmail type applications?

I am trying to build a web application, similar in feel to something like GMail, with one persistent AJAXy page.
I am told that GWT is the way to do this most effectively. However, I also want a framework that I can use for other things besides single AJAXy pages. Is GWT more scalable than this, or should I spend the time and build this thing in ASP.net MVC or Django?
I should mention, I'm trying to LEARN a framework that will be suitable for me to do whatever in.

You can use GWT also for parts of the page (a single widget or two), but yes, you are right,
even then it is geared towards the case where you do not refresh the page it lives on.

GWT works fine for both cases. I've written a couple applications at work with it, and I haven't had any trouble with multi-page navigation. As Thilo also mentioned, you can use as much or as little of GWT for manipulating your page(s) as you need or want.

You can do both styles of applications in ASP.NET MVC or GWT.
GWT makes it easier for GMail-type applications eg: web applications which mimic desktop application functionality.
On the other side, ASP.NET MVC is easier for dynamic sites. EG: This site is built on ASP.NET MVC, and if you look at it, its behaves more like a dynamic and interactive web site rather than a desktop-like application.
So its just easier to do web sites in one, and web applications in the other. So I'd think about what type of app you'd rather spend your time on and choose the framework accordingly - Realising, of course, that you can do both types of applications in either.

The nice thing of GWT is that in the end all you have is HTML and JavaScript. You can combine it with any other technology.
You can use GWT for the one page that needs ajaxy stuff and the rest with something more traditional.
Or you could even write everything with another framework and then put a GWT widget that needs more AJAX behaviours in a dedicated location of your page.
If you want to stay in the Java world you can use JSP,JSF, Struts, Wicket, or any other framework.
If you want you could even just combine GWT with a .NET or PHP. Just don't use the GWT RPC in that case (use JSON or some simple XML encoding instead).

Is Google Web Toolkit similar to AWT and Swing

I've looked breifly into GWT and like the idea that I can develop in Java and have the application compile down to HTML and JavaScript. Is the concept behind GWT and AWT and Swing the same or different?

It is programmed very similarly(patterned after Swing) and the code is 100% java (compiles with a standard Java compiler without errors), but the way it works is very different. Instead of compiling into a Java app, it compiles into Javascript that is sent to your browser.
This ability to program good active Javascript without actually coding Javascript and HTML is pretty nice.
Also, since it programs much like swing, you can do stuff like adding listeners that effect other controls pretty easily.

GWT is very much similar to Swing in its usage of Widgets, Panels and the EventListeners it provides. A different way to look at GWT is to think of Javascript and HTML as Assembly language and GWT as a sort of High level language which generates Javascript and HTML. With GWT its easy to develop desktop-like apps for the web using the same tools you would use for building a desktop app

Define concept.
AWT/Swing are used for desktop Java apps or applets. They both require JVM to run.
GWT is used to translate Java code to Javascript. This only runs on Javascript engines, i.e. browser.
The API design, as stated above, is similar to Swing. You get the same Panels, Buttons and other Component classes as in Swing.

AWT and Swing are for programming stand-alone applications (and to a lesser extent, applets). GWT is supposed to make programming web applications similar to stand-alone apps. I wouldn't program a non-web app using GWT.

GWT is conceptually similar to Swing, but is more a replacement for JSPs than anything else.

GWT is a javascript toolkit that allows you to write the javascript in Java. The only real similarities to Swing and AWT are that they are toolkits for creating user interfaces and they use Java. The end product is different and the real purpose for using them is different. GWT gives you the ability to generate a ajax user interface for a web browser while the other 2 give you a console (or applet) java app. In my mind the real reason for using GWT is to get a quick ajax interface up for prototyping purposes. But I dont think its really production ready- ie I dont thing Google uses it in their own webapps. A better choice for more robust ajax webapps is http://developer.yahoo.com/yui/ or http://script.aculo.us/.

disclamer: While I do work at IT Mill, this is just for information to the original poster. This isn't marketing spam.
If you like the idea of being able to write Java and get a webpage out of that, you might be interested in IT Mill Toolkit. It's a toolkit for making RIA software on top of a J2EE stack, and it uses GWT heavily.
One of the basic ideas are that the code generated by GWT is just as prone to client-side forging attacks as any other JS/Ajax-traffic. IT Mill Toolkit makes sure that the data is validated server-side too.
The toolkit is also designed in such a way that if you are familiar to Swing, you should have no problems picking it up.

Best server-side framework for heavy AJAX Java application

There are zillions of Java web application frameworks.
95% were designed before the modern era of AJAX/DHTML-based development, and that means these new methods are grafted on rather than designed in.
Has any framework been built from the ground up with e.g. GWT + Extjs in mind?
If not, which framework has adapted best to the world of forms with dynamic numbers of fields and pages that morph client-side?

Echo2 / Echo3 by Nextapp (www.nextapp.com) is totally awesome.
Advantages over GWT:
1) It is not limited to a sub-set of java like GWT
2) It is easier (in my estimation) to learn
3) Has extremely robust design studio for almost drag and drop designing.
4) It is very fast, and works very well on all platforms browsers
5) You can write your application using either java script or java
6) It has great and straight forward methods for handling events and actions.
Personally I think that for any web-application in which you are trying to integrate java and speedy delivery I wouldn't hesitate to pick Echo3 or Echo2.

If you're starting from scratch. I'd have to say Google Web Toolkit. I have to say it is incredibly powerful. You get keep using most of your Java tools. Plus, you don't have to duplicate code that exists on both the server and the client, it just gets compiled differently for each area.

I'd consider REST-style frameworks as well as the other recommendations here- Restlet or Jersey may be good choices for the backend, while you use something like JQuery or GWT on the front end. Both frameworks can easily produce JSON, and the REST style provides a nice clean line of demarcation between your client application and your server source; I find that JSF can make that demarcation pretty muddy.

I use JSF and IceFaces. Although JSF has a few limitations, IceFaces seems to work pretty well and has ironed out a few of the problems with JSF.
I haven't used a really good AJAX Java framework as yet, although Echo2 looks interesting.

I like the stripes framework. It lets you use whatever javascript toolkit you want.
Here is their documentation on AJAX

GWT is quite powerful and easy to use (all Java, no Javascript/HTML/CSS coding). If Google has their way it will be a dominant framework/tool in web applications development, and for good reason. It already works with Google Gears (which allows offline access to web apps) - and more than likely will be optimized to work within Google Chrome.

DWR
I use this to dynamically populate drop downs, and even filter them on the fly based on user input in other places on the form.

I like the combination of JBoss Seam and Richfaces, especially with the JBoss tools that are extentions to Eclipse - makes building these sort of RIA's incredibly easy.
Wikipedia contains some useful comparisons:
Comparison of JavaScript frameworks
List of AJAX Frameworks
Your choice depends on several different factors including whether you want the "work" done client-side (most javascript frameworks) or server-side (echo2 etc.). Other things worth looking at are tools like OpenLaszlo that provide Flash (I think) out of the box, but drop back to DHTML if there is no Flash player present.
Unfortunately I think the decision comes down to balancing several competing cocerns. Check out the comparisons and try them out - most come with online demo's for you to try.

Aptana has a server side frame work called Jaxer. This is from their site:
Jaxer's core engine is based on the same Mozilla engine that you'll find in the popular Mozilla Firefox browser. This means that the execution environment you use on both the client and the server are the same. It's Ajax all the way through and through. That means you only need one set of languages -- the languages that are native to the browser -- to create entire applications.
This framework is open source and has a very nice IDE based on Eclipse. Aptana is also working on a Javascript implementation for ActiveRecord called ActiveRecordJS. Potentially you could use this both client and server side with their framework.

GWT is one of the best AJAX framework that I used ever. Most important thing about this framework is that its maintained by Google. And Everyone know "Who is Google ?"
GWT is used by many products at Google, including Google AdWords and Google
Wallet. It's open source, completely free, and used by thousands of
enthusiastic developers around the world.
GWT provide rich widgets that can be used to built any application. Almost all the widgets they have.
Another important point is GWT is continuously developing and its also have stable release which is very good thing. Another thing Google has also released GWT-Material which is again a very good thing because everyone is moving toward material.
I hope this will help you!!!