We Keep Coding

Package vs project separation in java

First off, I'm coming (back) to Java from C#, so apologies if my terminology or philosophy doesn't quite line up.
Here's the background: we've got a growing collection of internal support tools written for the web. They use HTML5/AJAX/other buzzwords for the frontend and Java for the backend. These tools utilize a lightweight in-house framework so they can share an administrative interface for security and other configuration. Each tool has been written by a separate author and I expect that trend to continue, so I'd like to make it easy for future authors to stay "standardized" on the third-party libraries that we've already decided to use for things like DI, unit testing, ORM, etc.
Our package naming currently looks like this:
com.ourcompany.tools.framework
com.ourcompany.tools.apps.app1name
com.ourcompany.tools.apps.app2name
...and so on.
So here's my question: should each of these apps (and the framework) be treated as a separate project for purposes of Maven setup, Eclipse, etc?
We could have lots of apps appear here over time, so it seems like separation would keep dependencies cleaner and let someone jump in on a single tool more easily. On the other hand, (1) maybe "splitting" deeper portions of a package structure over multiple projects is a code smell and (2) keeping them combined would make tool writers more inclined to use third-party libraries already in place for the other tools.
FWIW, my initial instinct is to separate them.
What say you, Java gurus?

I would absolutely separate them. For the purposes of Maven, make sure each app/project has the appropriate dependencies to the framework/apps so you don't have to build everything when you just want to build a single app.

I keep my projects separated out, but use a parent pom for including all of the dependencies and other common properties. Individual tools / projects have a name and a reference to the parent project, and any project-specific dependencies, if any. This works for helping to keep to common libraries and dependencies, since the common ones are already all configured, but allows me to focus on the specific portion of the codebase that I need to work with.

I'd definitely separate these kind of things out into separate projects.
You should use Maven to handle the dependencies / build process automatically (both for your own internal shared libraries and third party dependencies). There won't be any issue having multiple applications reference the same shared libraries - you can even keep multiple versions around if you need to.
Couple of bonuses from this approach:
This forces you to think carefully about your API design for the shared projects which will be a good thing in the long run.
It will probably also give you about the right granularity for source code control - i.e. your developers can check out and work on specific applications or backend modules individually

If there is a section of a project that is likely to be used on more than one project it makes sense to pull that out. It will make it a little cleaner as well if you need to update the code in one of the commonly used projects.

If you keep them together you will have fewer obstacles developing, building and deploying your tools.
We had the opposite situation, having many separate projects. After merging them into one project tree we are much more productive and this is more important to us than whatever conventions happen to be trending.

Wanting to distribute program as a framework, but worried about number of dependencies

I have a project right now that straddles the line on framework and pluggable program, and am worried about the sheer number of dependancies that this program rely's on.
Currently I have this:
Commons lang - Mainly for string utils and array utils
slf4j - Logging facade
slf4j-log4j - Redirects logging to log4j for GUI (note that the GUI is a module)
log4j - Log4j itself for the above reason
jpersist/EJP - Database abstraction layer
PircBot - IRC layer
A JDBC driver
Mozilla Rhino - For Javascript plugins
In all that totals 7, even without the GUI unless you don't want any logging. For me who's trying to pass this off as "lightweight", this seems like way too much.
So my questions:
Should I limit the amount of frameworks that I am using?
How should I distribute it? Would an independent jar for being used in other programs and a big combined jar for a single program be okay?
Is this many dependencies normal?

It does seem like quite a lot. Regardless of the issue of specifying numerous libraries, you're restricting your users wrt. the third-party libraries they can use in their project to the ones you specify.
Can you specify implementation-agnostic libraries ? e.g commons-logging, which will delegate to existing logging frameworks under the covers. If your users are already using something other than log4j, then this will permit them to carry on without having to switch.
Secondly, is your framework doing too much ? Instead of providing a chat implementation, why not provide a suitable API such that clients can plug in their own chat/notification mechanism. That way your framework becomes more generic and (again) your clients can choose what/how to implement features. A rich client API will give your users many more options and extend the usefulness of your framework.

Should I limit the amount of frameworks that I am using?
If you are really using/needing them, not really. I would just try to avoid overlapping libraries and adding a library if you're only using 1% of it.
How should I distribute it? Would an independent jar for being used in other programs and a big combined jar for a single program be okay?
Many projects are distributed as a zip/tar.gz distro. But for a framework, making it available as a Maven artifact would be a great plus (in which case, make log4j and the log4j binding optional).
Is this many dependencies normal?
Firstly, you don't have that many dependencies. Secondly, there is IMO nothing wrong with reusing a logging facade, a persistence library, utility classes, etc (not using such libraries and writing your own code to replace them would be stupid). Thirdly, most users don't care, especially if you are delivering nice features (instead of spending time reinventing the wheel and, ultimately, creating bugs).

the scale of your project ie what it accomplishes and in what environment it will be used will balance out against how many dependencies and ease of configuration in general when people assess the suitability of your project.
you haven't really hinted at what your project attempts to achieve so it's difficult to say whether you have a bloated stack. however, for something reasonably useful I personally wouldn't have a problem with most of those jars.
the only thing that rings alarm bells is the database layer and the jdbc driver. if your project is a 'framework' i fail to see how a particular jdbc driver fits the model, and persistence in general does not quite fit the model of a framework.

That might seem like a lot of dependencies, but I don't think it is in reality. Certainly, there doesn't seem to be much gratuitous duplication of functionality. Most of the dependencies are doing things that you'd otherwise need to implement yourself.
For me who's trying to pass this off as "lightweight", this seems like way too much.
Maybe you need to adjust your rhetoric. :-)
Seriously, if those dependencies are necessary, the only way you will be able to get rid of them is to either code equivalent functionality yourself (bad idea) or drop the corresponding functionality (maybe a bad idea). Which is more important to you; being lightweight or being functional?
EDIT
Functional is key in the end. I can have my own custom implementation of everything but it would be full of bugs I guess. However I would like to keep it small as small and easy do attract people.
Well you clearly understand the issues. The decision is yours to make. (But don't forget that while some people are put off by "bloat", others are attracted by lots of functionality.)
I suppose that there is a half-way solution. Keep the functionality, but make it optional and provide some way that people can configure it in / out. Of course, the downside is that this means that you have to test multiple permutations of configuration options, and it makes installation / configuration more complicated for your users.

I think you worry to much :) The number of dependencies is not relevant, the maturity of them it is. If you will drop functionality/usability/flexibility/etc just because you want to keep the number of dependencies "low" it would be a loss for you (and your clients).

Java equivalent for database schema changes like South for Django?

I've been working on a Django project using South to track and manage database schema changes. I'm starting a new Java project using Google Web Toolkit and wonder if there is an equivalent tool. For those who don't know, here's what South does:
Automatically recognize changes to my Python database models (add/delete columns, tables etc.)
Automatically create SQL statements to apply those changes to my database
Track the applied schema migrations and apply them in order
Allow data migrations using Python code. For example, splitting a name field into a first-name and last-name field using the Python split() function
I haven't decided on my Java ORM yet, but Hibernate looks like the most popular. For me, the ability to easily make database schema changes will be an important factor.

Wow, South sounds pretty awesome! I'm not sure of anything out-of-the-box that will help you nearly as much as that does, however if you choose Hibernate as your ORM solution you can build your own incremental data migration suite without a lot of trouble.
Here was the approach that I used in my own project, it worked fairly well for me for over a couple of years and several updates/schema changes:
Maintain a schema_version table in the database that simply defines a number that represents the version of your database schema. This table can be handled outside of the scope of Hibernate if you wish.
Maintain the "current" version number for your schema inside your code.
When the version number in code is newer than what's the database, you can use Hibernate's SchemaUpdate utility which will detect any schema additions (NOTE, just additions) such as new tables, columns, and constraints.
Finally I maintained a "script" if you will of migration steps that were more than just schema changes that were identified by which schema version number they were required for. For instance new columns needed default values applied or something of that nature.
This may sounds like a lot of work, especially when coming from an environment that took care of a lot of it for you, but you can get a setup like this rolling pretty quickly with Hibernate and it is pretty easy to add onto as you go on. I never ended up making any changes to my incremental update framework over that time except to add new migration tasks.
Hopefully someone will come along with a good answer for a more "hands-off" approach, but I thought I'd share an approach that worked pretty well for me.
Good luck to you!

as I'm looking for the same heres what I've achieved so far.
We first used dbdeploy. It manages the most stuff for you but you will have to write all the transition scripts by yourself! That means every change you make has to be in its own script which you will have to write from scratch. Not very handy, but works very reliable.
The second thing I encountered is liquibase. It stores the configuration in one single xml file. Not very intuitive to read, but managable. Plus there is an Intellij Idea plugin for it. At the moment of writing it still has some minor issues, but as the author assured me, they will be fixed soon.
The perfect solution would be to get south working with your java environment. That really would be a tool to marry :D

Maybe try flyaway. Seems like a good alternative.

I've been thinking about using django-jython just for db migrations in our legacy Java application. The latest Jython version is 2.5.4rc1, but I think I can mitigate the risk by just using it for South migrations.
Especially since I can use inspectdb to generate the models for me. And then replace parts of the Java with Python "seamlessly".

If you're using hibernate, then checkout liquibase
http://www.liquibase.org/databases.html
It's been around for 10 years so it's pretty solid. It may support other ORM's, just have a dig around on their website. Checkout the liquibase+hibernate extension here:
https://github.com/liquibase/liquibase-hibernate

appfuse vs roo - what would you use

Appfuse vs. Roo, what would you use and why?
What are the sweet spots of each.

As per the answer I gave to TheServerSide thread on this issue...:
AppFuse aims to provide a single initial scaffold of your new project. This is similar to Maven archetypes or Eclipse's "new project" features in that you run them once at the start of a new project and then you maintain the scaffolded code going forward. The scaffold system has no further involvement in your project once you've run it once.
Roo, on the other hand, provides a round-trip aware active code generator for your long-term usage on a given project. As such Roo offers value both at initial creation time as well as whenever you are modifying the project going forward.
In practical terms this means as you evolve your project, Roo will automatically maintain certain files. To take a simple example, when you add (or remove) a field, Roo will update the toString, getters/setters, JSP pages etc for you automatically. It also offers commands so you can add new capabilities later. So if you need to add security six months after you created the project, you just "security setup". Or if you need to send emails, you just "http://static.springsource.org/spring-roo/reference/html/command-index.html#command-index-email-sender-setup". There are similar commands for many other capability areas well, such as Spring Web Flow, JUnit, Selenium, common JPA providers etc. You just defer the decision as long as you like, and Roo will only add those capabilities at the time you ask for them (and it will also automatically use those new capabilities in your project).
There are many other differences as well. Roo allows extension via user-developed add-ons, it offers a highly usable shell, it allows you to incrementally build a new project and add features only when required, it extensively supports the latest versions of the major Spring technologies, it comes with a SpringSource-developed (and therefore endorsed) application architecture and so on.
A read of the Roo Reference Guide's Introduction Chapter or simply completing the ten minute test project will illustrate they are very different in approach.

My notes on AppFuse and Roo:
AppFuse
Is a fully working template application/project.
Traditional DAO <-> Service <-> Controller architecture
Easy to get started with maven archetypes
Great documentation and tutorials
Not really up to date. Spring 3 final is soon to be released and AppFuse is based on 2.5 (?)
Spring Roo
Spring Roo on the other hand is a tool that speeds up development by using code generation.
Getting started with a new, fully configured project takes 1 minute
Creates rich domain objects where CRUD are weaved into the domain objects using AOP instead of traditional DAOs/services
Hard to grasp if you are new to Spring
Documentation is not that good yet
Really cool! I.e. add Spring Security to your project with just one line of code!

Telosys (a lightweight code generator) is also a good alternative. See
http://www.telosys.org/
It produces very clean code (without adherence like ApectJ)
and the templates are customizable
There's a stack of templates designed to generate Spring MVC web apps (and many others to generate code for other kinds of frameworks).

for me Appfuse but it is not up to date, but the spring roo using aspectj and there are parts of code you should not touch and I do not like that.

Spring Roo
Pros.
1.
Customizable : You add and remove diffrent framework and addon as per your requirement.
Database Reverse Enginnering : Create CRUD applications if you have database schema ready.
Strong Spring community support.
NOSQL MongoDB support
Can create required add-ons.
Cons :
Require in depth knowledge of Aspect oriented and Spring stack.
Require little more time to learn spring roo as compared appfuse.
Appfuse:
Pros:
Good one to start small and mid-size enterprise application with
struts, JSF and Spring
Complete open source code.
Enough documentation.
Twitter-bootstrap ready.
Cons:
Customize application but not like Spring roo.

For starters, roo looks more over engineered, with code generation, use of aspect oriented programming and more.

Appfuse doesn't seem to be maintained anymore seeming that the last version was released on May, 2008.
Roo it's right now a little bleeding edge, because of the use of the still-unreleased version 3 of Spring Framework, but that will change, and that version brings a lot of interesting changes to the table.
It also upsells you on more of the Spring technologies portfolio, such as STS and tcServer, and makes it dead easy to use Spring Security and Spring WebFlow.

I am going with ROO.
I am already using
Spring
Spring ORM/JDBC
Spring MVC
Spring Remoting
STS (Tool-suite)
So my preference is SpringSource products, AS I am already familiar with API style, documentation, conventions of SpringSource and even their coding practices once I extended/implemented security-framework's code.. ;-)
so, my advice is go with the tool/framework which is more natural to you..
Cheers,

AppFuse -> change to SpringFuse
I more prefer use SpringFuse

The question is some years old and in the meanwhile there are new productivity tools I want to point out:
Generjee. Generjee is a full-online tool. You define online your requirements and the (optional) data model. Then you get the generated code as a download.
The generated code is independent from generjee. The tool is useful for starting development projects from an integrated full-stack code base.
generjee will generate for you:
JPA code according to your data model
JSF code to create, read, edit, filter, sort and export data
user management, registration and login
your defined user roles and specific access permissions for this roles
I18N support
file upload support
Forge. To describe it in some words, it is "like Roo". But Forge is not as strong based on Spring and AspectJ as Roo.

AppFuse has integration with Tapestry, Wicket and other Web frameworks which Roo does not - yet

The AppFuse project was shut down in April 2016. Its founder, Matt Raible, recommends using JHipster as an alternative.

How to modularize a (large) Java App?

I have a rather large (several MLOC) application at hand that I'd like to split up into more maintainable separate parts. Currently the product is comprised of about 40 Eclipse projects, many of them having inter-dependencies. This alone makes a continuous build system unfeasible, because it would have to rebuild very much with each checkin.
Is there a "best practice" way of how to
identify parts that can immediately be separated
document inter-dependencies visually
untangle the existing code
handle "patches" we need to apply to libraries (currently handled by putting them in the classpath before the actual library)
If there are (free/open) tools to support this, I'd appreciate pointers.
Even though I do not have any experience with Maven it seems like it forces a very modular design. I wonder now whether this is something that can be retrofitted iteratively or if a project that was to use it would have to be layouted with modularity in mind right from the start.
Edit 2009-07-10
We are in the process of splitting out some core modules using Apache Ant/Ivy. Really helpful and well designed tool, not imposing as much on you as maven does.
I wrote down some more general details and personal opinion about why we are doing that on my blog - too long to post here and maybe not interesting to everyone, so follow at your own discretion: www.danielschneller.com

Using OSGi could be a good fit for you. It would allow to create modules out of the application. You can also organize dependencies in a better way. If you define your interfaces between the different modules correctly, then you can use continuous integration as you only have to rebuild the module that you affected on check-in.
The mechanisms provided by OSGi will help you untangle the existing code. Because of the way the classloading works, it also helps you handle the patches in an easier way.
Some concepts of OSGi that seem to be a good match for you, as shown from wikipedia:
The framework is conceptually divided into the following areas:
Bundles - Bundles are normal jar components with extra manifest headers.
Services - The services layer connects bundles in a dynamic way by offering a publish-find-bind model for plain old Java objects(POJO).
Services Registry - The API for management services (ServiceRegistration, ServiceTracker and ServiceReference).
Life-Cycle - The API for life cycle management (install, start, stop, update, and uninstall bundles).
Modules - The layer that defines encapsulation and declaration of dependencies (how a bundle can import and export code).
Security - The layer that handles the security aspects by limiting bundle functionality to pre-defined capabilities.

First: good luck & good coffee. You'll need both.
I once had a similiar problem. Legacy code with awful circular dependencies, even between classes from different packages like org.example.pkg1.A depends on org.example.pk2.B and vice versa.
I started with maven2 and fresh eclipse projects. First I tried to identify the most common functionalities (logging layer, common interfaces, common services) and created maven projects. Each time I was happy with a part, I deployed the library to the central nexus repository so that it was almost immediately available for other projects.
So I slowly worked up through the layers. maven2 handled the dependencies and the m2eclipse plugin provided a helpful dependency view. BTW - it's usually not too difficult to convert an eclipse project into a maven project. m2eclipse can do it for you and you just have to create a few new folders (like src/main/java) and adjust the build path for source folders. Takes just a minute or two. But expect more difficulties, if your project is an eclipse plugin or rcp application and you want maven not only to manage artifacts but also to build and deploy the application.
To opinion, eclipse, maven and nexus (or any other maven repository manager) are a good basis to start. You're lucky, if you have a good documentation of the system architecture and this architecture is really implemented ;)

I had a similar experience in a small code base (40 kloc). There are no °rules":
compiled with and without a "module" in order to see it's usage
I started from "leaf modules", modules without other dependencies
I handled cyclic dependencies (this is a very error-prone task)
with maven there is a great deal with documentation (reports) that can be deployed
in your CI process
with maven you can always see what uses what both in the site both in netbeans (with a
very nice directed graph)
with maven you can import library code in your codebase, apply source patches and
compile with your products (sometimes this is very easy sometimes it is very
difficult)
Check also Dependency Analyzer:
(source: javalobby.org)
Netbeans:
(source: zimmer428.net)

Maven is painful to migrate to for an existing system. However it can cope with 100+ module projects without much difficulty.

The first thing you need to decide is what infra-structure you will move to. Should it be a lot of independently maintained modules (which translates to individual Eclipse projects) or will you consider it a single chunk of code which is versioned and deployed as a whole. The first is well suited for migrating to a Maven like build environment - the latter for having all the source code in at once.
In any case you WILL need a continuous integration system running. Your first task is to make the code base build automatically, so you can let your CI system watch over your source repository and rebuild it whenyou change things. I decided for a non-Maven approach here, and we focus on having an easy Eclipse environment so I created a build enviornment using ant4eclipse and Team ProjectSet files (which we use anyway).
The next step would be getting rid of the circular dependencies - this will make your build simpler, get rid of Eclipse warnings, and eventually allow you to get to the "checkout, compile once, run" stage. This might take a while :-( When you migrate methods and classes, do not MOVE them, but extract or delegate them and leave their old name lying around and mark them deprecated. This will separate your untangeling with your refactoring, and allow code "outside" your project to still work with the code inside your project.
You WILL benefit from a source repository which allows for moving files, and keeping history. CVS is very weak in this regard.

I wouldn't recommend Maven for a legacy source code base. It could give you many headaches just trying to adapt everything to work with it.
I suppose what you need is to do an architectural layout of your project. A tool might help, but the most important part is to organize a logical view of the modules.

It's not free but Structure101 will give you as good as you will get in terms of tool support for hitting all your bullet points. But for the record I'm biased, so you might want to check out SonarJ and Lattix too. ;-)