Dear Stack Overflow Community,
I am a Java programmer in front of a task of building a complex, data-driven, web application (SaaS) and I'm searching for technologies to use. I have already made some decisions and I believe I'm proficient enough to build the appliaction using just the technologies I have decided for (I'm definitely not saying it would be perfect, just that it would be working). However, I'd like to make my task easier and that's why I need your help.
Brief description of the project
Back-end
The application will be heavily data-driven, meaning that everything will be stored in a self-descripting database. This means the database itself will be entirely described with metadata and the application will not know what data it reads and writes. There won't probably be any regular entities (in terms of JPA #Entity) because the application won't know the structure of the data; it will obtain it from the metadata. Only the metadata will have a pre-determined structure. To put it simply, the metadata is the alpha-omega of the application because it will tell the application WHEN and WHAT to display and HOW to display it.
The application will probably utilize stored procedures to perform some low-level tasks on the data, such as automatical auditing, logging and translating to user's language, thus most likely eliminating any possibility to use ORM frameworks because there won't be just simple CRUD operations. Therefore, JDBC seems like my only option (doesn't it?).
Front-end
The UI will be "dumb" in terms that it will not know what data it is displaying (to some extent, of course). It will just know how to display it based on the metadata which it will obtain from the database. All UI controls (like menu items, buttons, etc) will be created based on current application's state and the UI will NOT know what the controls do. This means that clicking a menu item or a button will just send an identifier of associated action to the back-end and the server will decide what to do.
My goals
My main goal is to have the application as lightweight as possible with as least dependencies as possible. Because the application will be very complex, I'd like to avoid any heavy framework(s) because there is a very high probability that I'd need to customize a lot of its functionality.
What I have already decided for
Please object to the following decisions only if you think they're absolutely non-viable for my application, as I have already implemented some core functionality using these technologies:
Servlets on Tomcat, Guice DI, AOP (AspectJ)
I believe all of these technologies are lightweight enough and I don't need to learn J2EE.
GWT with GIN-jection on the front-end
Seems like the best option for me because I'm very familiar with Java and Swing and don't want to write any Javascript, PHP or learn a new language. GIN is a little brother of Guice and I will be using the same syntax and principles on both the client and server.
MSSQL RDBMS
This is actually a requirement from company management as I'd much rather like to go with an open-source solution. Too bad for me..
Maven 2
I think no-one can object to this :)
What I need help with
DB communication
I think that ORM is ruled out (is it?) so I need to use JDBC. Do you think Spring JDBC is lightweight and flexible enough for my use? I would often need to "blindly" read data from database, mapping it to some generic entity (because I won't assume any pre-determined structure), and then send the data using some generic DTO to the client along with the metadata telling it what data it is and how to display it. Or do you know any alternatives? Or should I do this myself?
Client/Server communication
GWT and its GWT-RPC mechanism seems not very suited for sending the generic data I need. Although I'm convinced that it's doable using GWT-RPC, are there any alternatives? But I definitely want to use GWT.
Security
Do you know any security libraries / frameworks that would help me? I'm aware of the existence of Spring-security; do you think it's flexible enough for my use or I'd be better off implementing that myself? Also, is Spring's IoC an integral part of the Spring framework, or would I be able to continue to use Guice?
Anything else that you think might be useful?
I really appreciate any advice and suggestions because I wouldn't dare to try to make such decisions myself. Please ask me if you need more information.
Thank you in advance!
eQui
I think you are over-engineering the solution. Take a look at
http://thedailywtf.com/Articles/Programming-Sucks!-Or-At-Least,-It-Ought-To-.aspx
If everything is driven by the DB, you are going to have immense difficulty making things happen in the UI, and you aren't going to be able to use many of the tools that make UI development easier.
I also suggest you take a look at Spring Roo, if your application is mainly just updating data in a database.
UI framework and implications for client/server communication
You say that any UI action will triger the backend (and potentially the DB). This mean that UI interraction will be somewhat slow anyway, and more than that will require a round trip to the server.
GWT is especially suited to avoid as much as possible round trips to the server and do all UI work on client side. In this model, only information that will transit from client to server is real data, and not UI metadata. GWT will do the job, but you'll be using a somewhat low level tool, needed for advenced optimisation you'll be unable to perform anyway...
Framework like ZK or Vaadin seems more suited to what you want to do. The client side has nice widgets with a rich UI, but you manipulate the UI from the server side. The framework manage client/server communication for you (no need of REST, RPC or javascript). The main limitation of theses framework is scalability, with all theses chatty round trip. But because your requirement impose that chatty behaviour anyway, you could really benefit from the abstraction they provide, are they are at not cost in your case.
I have tried both GWT and Zk to do some proof of concept for my company. We ended choosing GWT, because of it's hability to be embedded nicelly into any existing UI and to fine tune what you do... In particular avoid as much as possible rountrip to server. But ZK is really easier and faster in term of developmeent hours.
The side effect is that would totally solve your client/server communication concern, leting the framework performing it in an optimized way (Zk is able to intelligently regroup several UI event before sending them to server).
DB and ORM
For DB design, i tend to think that using fine granularity things in DB will make it very very slow. If each widget is one or several rows in the database you'll have to perform many lookup to perform the simpliest thing.
Problem is if your UI is just a little complex with a few dozen of elements (a few button, checkboxes, labels and widgets), compositing a screen will require lot of requests to the DB. Rendering just one page might be very slow and scalability would be very very bad.
I know this because i worked on somewhat generic bug tracking system with similar (but simpler) requirements than yours and we had exactly this problem.
So i would try to describe UI in some templating or XML format. Maybe you'll not show this data to the user, providing it with a nice abstraction, but instead of performning many queries for just one screen, you'll save the whole screen as one blob.
A really dumb and basic implementation of this would be to store HTML/CSS/PNG file in your DB and load it as needed, with user being responsible for making theses HTML file by hand. Of course this would be terrible for the user. That's why you need a nice and fancy editor UI editor that would work on an intermediary format of your own. Another dumb implementation would be some sort of wiki templating. This is not what you need, you need more. But you have the idea, I would seek in that direction...
For the maintenance and debugging too, this would be far easier to the whole UI description to a few file, to understand what is really implemented than to read lot of tabuled data in your prefered SQL editor. Users would have they export/import format to easily version, backup or experiment.
Security
I would say by hand... Because you have a generic UI generated by user it seem likely that the security will be generic too and dependant of database content.
Hope it help...
For the backend, i implemented a program which had a similar interaction with the database. the code was database structure oblivious, instead, it read a config file describing the db and could construct complex sql queries based on this information. most of the code is proprietary, but one bit of it got pushed into an open source project called sqlbuilder. may be useful to you on the backend.
I think you're on the right track, with your tool seclection. Your 100% data driven model is going to be hard to maintain. But I understand that's a requirement not an option. Normal source control is going to fail you becuase of the ui application logic all being in the meta-data. You'll need some good test databases and some way to maintain them, such as regularily mysqldump them out and check them in to souce code control to handle all the differences, etc..
You're wise to stay away from various ORM solutions and just use JDBC for this type of app.
Let me give you some warnings about GWT. On the surface it will abstract all the uglyness of html, javascript away and give you clean heirarchy's BUT...
1) If the abstraction fails you how do you easily debug?
2) Do you want any of your site to be visible to Google or other search engines, if yes GWT is not for you
3) Do you want to use any HTML5 technogies or do you want to be stuck in IE 5 compatability mode?
So...
I think you'll be much better off Implementing the UI as simple HTML controls with a small set of jQuery ajax interactions with the server. You can define an input type in your database, your serverlet can generate an input tag and then you have two options you can have some standard event bindings in jquery to tell your server that button1. is clicked, or that select2 has changed, etc.
Your server can send back javascript to change the state of the ui - simply load the javascript inside a div so it runs on the client. or 2) You let the input submit the data to the server and do an old school page refesh and the servlet build the next ui screen based on the database.
Building an interface dynamically in HTML from a database is easy and straight foward compared to doing the same in SWING or Windows Forms. You just have to write out a big text string, been doing that since 1999.
That approach is going to be much more lightweight - simpler to debug, understand and modify in the long run than going with the "GWT automatically compiles to unreadable javascript that doesn't run in my browser for some unknown reason" solution.
Related
We want to write a UI that consists of HTML, Javascript (JQuery) and CSS. Although the initial starting point will be served up by a web server, there won't be any sever side templating. The browser will interact with the server via a restful interface and render its UI.
What are the risks of this approach?
Ideally I'd like a nice, straightforward javascript OO api which underneath makes http calls to the server to get JSON representations of resources. Any suggestions as to how this could be structured?
Anyone have experience with browser side templating?
Is there a framework to make this style of development easier?
We will also be defining the server side resources and my thoughts are to follow ruby on rails conventions. For example, if you define a Users resource in routes.rb, you have 7 uri templates. Any thoughts?
By the way, the server side functionality will be developed in java.
I have plenty of experience with this approach. I can guarantee you that it works - how well in the long run, I don't know yet but I'm extremely happy with it (as a developer).
You do need to make sure that you've mastered Javascript. Read up on the state of the art, at least check Douglas Crockford's work, and most notably JSLint.
As for frameworks, this is where your vision comes in. We've built one from scratch because we need a combination of tools that existing frameworks don't and because we think we have the vision and expertise to carry it through. You have to compare the pro's and con's. If you use an existing framework you have very little control over it's direction or the speed at which bugs are found and fixed. If you build one yourself you could run the risk of making wrong decisions and ending up with a framework that doesn't quite work.
I have noticed that in our applications the custom server side code is only very small. This means the importance of the backend is only very small (validation, sanity, authorization). We use PHP, but simply because we have loads of experience with PHP.
There are definitely risks. In the startup and early transition I have noticed that 'lesser' programmers have trouble catching up. There is a very steep learning curve for anyone not too familiar with Javascript and it's many elegances.
Another risk is performance. We're advising our customers to use Google Chrome, simply because
And then there is compatibility. The idea of a framework is that it's able to hide this complexity. Luckily browsers are increasing in-tune in accordance to standards but backwards compatibility with (for instance) IE6 is incredibly difficult.
I would advise against using jQuery. I find jQuery more of a 'plugin' than an actual framework. jQuery really shines when you have a website and you want to sprinkle on some fanciness. It has some very good general tools (DOM manipulation and all that) but it's very lacking in the business-modeling area.
I would also advise against an OO approach. For some very small number of domains OO is the perfect solution. For most businesses, it's not. And Javascript is capable of so much more than just OO.
The #1 problem (and, perhaps, the only problem) is search engine. It is not sure how well will your content be recognized/crawalble/searchable. The underyling cause is that the search engine is not necessarily going to understand your content (since it is only revealed once Javascript gets executed).
Other than that, it is a great approach. I tried it several times and it works great (assuming you're not intimidated by Javascript). The resulting web-site is usually much more responsive than traditional web-sites since the server -> client traffic is quite small - only the raw data is transmitted. All the UI stuff is generated, by Javascript, on the client side.
Is it wise to develop a prototype GUI before designing other part of the system?
I am using Java for this small project. It will be a program with GUI and database connection. Say the database has table A and B, the user can choose which table to interact with. The program then display the contents of, say, table A in the GUI, and allows the user to change the content and submit the changes, or delete, or insert.
I think GUI should be developed first before any back-end development starts. There are couple of reason to do this:
You gain clarity on how model objects should interact.
Usability poses lots of restrictions on the way you want to pull data. You will probably want to develop and architect after you're 100% sure what constraints are there.
On business point, managers like to have a dumb function UI before any development start. Many times, the feedback leads in major changes in back-end assumptions. Which is a lot less pain than the case when you get a change request after the back-end development is over.
My personal experience goes that simultaneous development of GUI and back-end is a bit messy. Plus GUI provides solid expectation of behavior from back-end. Moreover, this approach makes sure all the developers, your client and your manager on the same page.
I agree with Joel Spolsky that it is a great idea to write a functional spec before writing code. Part of that spec should include a collection of screen mockups. #O.D. is right, Balsamiq is a great tool. It has saved me a lot of time in the past.
Once you have a functional spec in place that the business users are happy with, you will then have a better idea of how to design your system to meet the requirements. e.g. is high performance a requirement, domain model vs simple crud etc.
Then you should start by taking a single use case and building a vertical slice of your application. Build a GUI, service layer, persistence layer, database schema in one iteration. This will hopefully point out any problems with your design and give you the chance to modify it before you start building out the horizontal functionality.
I'd say yes and no.
No because you should design you application to be modularized enough so that your logic and data do not depend on UI design.
Yes because it is always smart to design everything before you actually start implementing it.
So what I mean is that you should make a concept, but not let your UI concept 'tie your hands' when you implement your logic. So if your managers clients don't like your conceptual UI, you can always change it without actually changing your application logic.
Well showing you GUI brfore starting to program is a very a good Idea, specially that you enable the enduser (Customer) to check if the UI is up to his expectations, which can save you lots of time.
In order to do that you dont necessarily need to develope a "real" prototype, you can use programms which enable you to fast design the UI of your App, including a minimal workflow simulation instead of full funcionality.
i had a very good experience with: Balsamiq can really recommend it
Writing spec before your code is always a good idea, because it makes you think. But most specs I have seen are not that good. And if the spec is too technical, users will at the end sign-off your spec without really understanding what are they going to get.
I have seen best results when either presenting the User Manual to the client, or by discussing mockups of the system one scenario at a time.
Note that half-baked mockups won't do the trick. You need your mockups to be fully populated with relevant data (Ever tried to discuss some screens with accounting while the numbers on the screen don't match? There's no way at all you could explain to them these are only dummy numbers...)
And the caveat of using mockups is that users will more often than not believe the app is "almost finished", whatever you do or say. It must be some subconscious thing, I'm not sure. But to avoid that, most of specialized tools have either only "black&white" look and feel or multiple skins you can switch to and from.
There is a pretty complete list of mockup tools here. Many of them are free:
http://c2.com/cgi/wiki?GuiPrototypingTools
My own tool is pretty popular: http://MockupScreens.com, I created it a long time ago exactly because of my own frustration with above mentioned problems.
I am thinking of working on an online homework submission system where students can upload their text documents that can viewed by somebody(Teacher/Professor) with viewing rights. This person can annotate/mark up/highlight this document and then can pass this document on for further review. After the review is done, this document is sent back to the student with proper feedback.
There are a lot of moving pieces here and I am having a tough time trying to figure out where to start. I am using this project as an exercise to design a scalable/secure/fast web-application using entirely open source tools that supports desktop like user experience but besides the UI-framework(Vaadin) and language(Java), I am having a little bit of trouble figuring out the roadmap needed to make some progress with this.
It will be great if the SO gurus can mentor me through this or provide me with a nudge in the right direction.
Edit: Thanks for the response. This is the standard three-tiered architecture that was described in the response. I need massive scalability and since the application will be primarily document centric and I may have to retrofit search at some later date, I would like to steer away from RDBMS. Since we have a massive amount of users posting their documents at any given time(lets say *.txt) files for the time being, I would need some kind of message queue to process this massive information influx. There has to be some kind of a fast transformation layer that takes the documents in all its formats and displays it in a format suited for annotations and markup....and the list goes on. Starting with the Domain model and moving down would be ideal but I am a little bit of a skeptic.
I'm a Java guy. How I would approach this problem will be,
Figure out the data model. What are the Objects that the system will be using.
Based on that design your database. You can use ORM frameworks like Ibatis or Hibernate ( Ibatis has a code generator which generates pretty much all the DAO's to access the tables. It gives you the CRUD methods and you can add on top of it).
Once you are done with that, then you can start designing your service layer. Its not a good practice to expose the DAO's directly to the controllers( MVC pattern). This is where your business logic should go in.
The choose one of the web frameworks available. The most popular java web frameworks are Spring MVC. You can also try Google Guice.
Final step will be to design your front end. I think your project will include a lot of javascript. So look into JQuery or EXT JS
I hope this helps you to get started with.
Not really a "full answer" but hopefully some food for thought.
I am using this project as an exercise to design a scalable/secure/fast web-application using entirely open source tools...
That's a great thing to be doing - so my question would be why worry about the architecture? I admit this is a bit strange coming from someone who see's themselves as an advocate for architecture, but "learning architecture" and learning new technologies can both be demanding. Maybe you just want to get up-to speed with one first, then tackle the other?
Architecting, designing and building a system that is "scalable/secure/fast" isn't trivial. In a "real-world" business driven case you'd have a system context to fit in with, and non-functional requirements to hit: these might suggest a different technology stack. Different key drivers will have a huge affect on how you approach things and what decisions get made - and of course the architecure would be built around that.
Edit:
I'd start with what was more important to you; then as I start on the "other" topic I'd be constantly checking what I'd learnt / my assumptions.
Depending on the two subjects there might be some "synergies" which suggest a different approach. Part of me wants to recommend doing enough on one to get a good basic grasp and then bring the other one up to a similar level - then move on. That way you're kind of more consistent.
Another part of me says - just do what's most interesting!
Should I plumb the depths of a given technology stack before moving forward or get the blueprint right from a tool agnostic standpoint?
As a start, you'd definately want to get a broad but shallow view; maybe go into some specific areas in a little more detail if they were of particular importance to you.
Or is my blueprint going to change because of the technology constraints?
By this I'm assuming you mean that: if you discover a "given technology works in such a way" and how it works it different form what you were expecting - then will this affect the blueprint?
It could do - that's why a broad but shallow view of the relevant technology should help you avoid the worst / most obviously bad mistakes. I'm also assuming that you have enough overal experience to know what you're looking at when you start doing this - that you can recognise an area that needs more detailed study before committing to a design.
At a high level, all you're looking to do is ensure you're not restricting options further down the path.
The application is vb.net front end and sql server express backend. The networks are always cabled LANs.
Installations are small with only a few users, none of whom would have any technical knowledge.
Very little technical support is ever called for and I'd like to keep it that way.
I don't know Java or Objective C or HTML/CSS/Javascript which as far as I can see seem to be the choices for smartphone development on Android, iphone or web based application
I want users to be able to access as much of the functionality of the application as possible for the least effort both in terms of coding and acquiring new skills on my part.
I don't know where to start or which would provide the easiest path.
I don't know how to make the database available to smartphones whilst keeping it physically secure in a small office.
If all things were equal I'd probably learn towards HTML/CSS?Javascript as it seems to be the most widely applicable.
On the other hand maybe I should wait for win phone 7?
To reach the largest number of users in a device independent manner then delivery via browser is going to give you the best results for the least effort.
If you have designed you existing application with a Data Access Layer, a Business Rules Layer and a User Interface layer, this may be as simple as creating an ASP.NET UI for mobile/internet/intranet users.
If your appliciaction is not designed this way, then my approach would be to seperate out the code in you existing into these three layers, or at the very least seperate the UI layer out of the existing code. Then it just a matter of implementing a UI layer for each access method you plan to use.
That way you end up with a lot less code to maintain, and when the businees rules or backend data changes you only have to do the change in one place for all you User Interfaces.
Well, .NET Compact Framework is already avaialbe on WinMobile, so you defenitely should give it a try if you're free to choose which mobile OS to target.
If not, I suppose that for task like this it would really be better to use web interface. If you don't now HTML/CSS/JS - as for me it's not a problem but a great chance to learn new interesting trendy things! :)
I would go with a simple html app designed for a mobile screen.
Android or iphone will only get a % of your users. If you want to get them all, you would need to write in both (and then blackberry and winmo are SOL).
So without seeing the application, it is very hard to know how much work converting vn.net to something you can get at from a web browser would be... but I don't think it would be much worse than a port to android or iphone, and it will allow a much bigger market to view.
Either way, you will need to learn something new. Learning is good though, right?
I want to create a process control application. Events update the database and that should be reflected on the GUI.
Although I personally prefer Linux, the hard fact it that 100% of the potential customers I can imagine run Windows.
Ok, for Windows I am comfortable with C++ Builder.
I suppose I could switch to NetBeans and use Java just in case someone ever wants it cross-platform.
browser based is probably the easiest way to cross-platform (barring some disagreements between browsers).
The thing about browser-based is I am not sure how to implement it. Would I auto-refresh the page every second or so? Can a database change be propagated upwards via PHP and refresh the screen? A very basic question, but I am new to this sort of thing, coming from an embedded background.
If all else is equal, which is easier for me to implement and maintain?
If it's real-time control, and you have to respond within a very narrow time band, then web based and Java based probably won't do it. If it's real-time control problem you ought to look elsewhere for a solution.
You can certainly use the web, Java and PHP to display results as they are produced, but the actual control and persistence to a database should be done with different technology.
I'd also be careful about writing to a database. It should be an asynchronous, "write behind" capability rather than the naive, "connect to a relational database and do an INSERT" sort of thing. I think that will be too slow.
If it is desired to be multiuser apllication I prefer web applications. Easy to make changes, easy to deploy. No problem with firewall settings and etc.
About propagation of changes from server to client. No way. But you can utilize AJAX tu "ping" on server and checking if somethink is changed. And if somethink is changed then load id and change view. Facebook/Google use somethink like this for chat/googletalk and so on.
About browser differencies. You can use CSS framework, JavaScript framework and most of problems with diferencies between browsers are solved.
Edit: If it is about seconds I think that PHP, Python or somethink really easy and fast is good part on server side. Or C++ CGI module. And on database side SQLite. Lightweight and fast solution for not much complex data. ANd not to big amount of data.