We Keep Coding

Remote MySQL DB and eclipse

I am currently trying to implement a simple servlet that has to communicate with our database.
I have no real prior experience with databases, so I was wondering how I should I go about this? I have downloaded the mysql-connector-java-5.1.40 from dev.mysql.
Going over some of the directions on the web for setting up the connection, it seems to only be for local mysql, but what of remote? The remote's user and pass is demo/demo; of course I would also need to log into the the remote server with my credentials. How do I go about connecting to this remote db?
Edit: So I believe I successfully connected to the DB, at least I can see it in my eclipse under data sources and the tables are present (company and stock_prices), however my eclipse still says I have an unsuitable driver even though I do have one associated with it.

The proper way of consuming a database resources in a web container (or in an application server) is through the javax.sql.DataSource abstraction. So you should configure a data source in your container. For tomcat it's as simple as creating a file named context.xml in your war's META-INF folder with the following content (replace address and credentials with your own):
<Context>
<Resource name="jdbc/[YourDatabaseName]"
auth="Container"
type="javax.sql.DataSource"
username="[DatabaseUsername]"
password="[DatabasePassword]"
driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql:/[yourserver]:3306/[your-db]"
maxActive="20"
maxIdle="20"/>
</Context>
Then when you want to perform a DB operation:
you either look up the data source:
DataSource ds =(DataSource) new InitialContext().lookup("java:comp/env/jdbc[YourDatabaseName]");
or simply use dependency injection for managed components like servlets:
#Resource(name="jdbc/YourDataSource")
Datasource ds;
The you just get a connection from the datasource in order to execute statements to the database.
The DB driver can be put in one of two places:
the war's lib folder
tomcat's lib folder
It's recommended to put it in tomcat's lib, because drivers are singletons and if you have several apps with different versions of the driver in the same container bad things will happen.
How do I go about connecting to this remote db?
Connecting to a remote DB is the same as connecting to alocal DB. Just pass the correct DB address in the connection string.

When to put configuration in file.properties or Jndi

For a long time in many IT services, I see some complex process to manage Java EE application configuration depending of the environments:
- custom tools, with Database or not, to manage replacement in the properties file (unzip war, replace, zip war...)
- Externalize properties file in obscure directory in the server (and some process to update it some time) and some time with a JNDI configuration...
- maven profile and lot of big properties files
But for database connection everybody use jndi datasource.
Why this is not generalized for all configurations that depend of environment ?
Update : I want deal with other variable than datasource, there is no question about datasource : it's in configured in JNDI for Java EE application. After if you want hack JNDI...

Setting up database connectivity (like user name, password, URL, driver etc.) somewhere in the application server has several advantages over doing it yourself in the WAR:
The app server can be a central point where the DB is configured, and you might have several WARs running on that server sharing a DB. So you need to set it up only once.
The DB settings, especially the credentials (username, password) are stored somewhere in the app server instead of somewhere in the WAR. That can have security implications (for instance, restricting access to that file is easier done than in a WAR archive).
You can set up one JNDI path to retrieve a DataSource instance pointing to the DB and do not need to worry about username and password anymore. If you have multiple app servers (one live system, one test system, several developer machines) with different DB URLs and credentials, then you can just configure that in each app server individually and deploy the WAR files without the need to change DB settings (see below).
The server might provide additional services, like connection pools, container managed transactions, etc. So again, you don't have to do it on your own in the WAR.
This is true for other services provided by the app server as well, for example JavaMail.
There are other cases where it you want to configure something that is specific to one web application and does not rely on the environment (the app server), like logging (although that may be set up in the app server, too). In those cases you might prefer using static config files, for instance log4j.properties.
I want to illustrate the third bullet point a bit further ...
Suppose you have one WAR in three app servers (developer machine, test server, live server).
Option 1 (DB setup in WAR)
Create a database.properties :
db.url=jdbc:mysql://localhost:3306/localdb
db.user=myusername
db.pass=mysecretpassword
#db.url=jdbc:mysql://10.1.2.3:3306/testdb
#db.user=myusername
#db.pass=mysecretpassword
#db.url=jdbc:mysql://10.2.3.4:3306/livedb
#db.user=myusername
#db.pass=mysecretpassword
Before you deploy it somewhere, you need to check if your settings are pointing to the right DB!
Also, if you check this file in to some version control system, then you might not want to publish your DB username/password to your local machine.
Option 2 (DB setup in App Server)
Imagine you have configured the three servers with their individual DB settings, and each of them registers the DB with the JNDI path java:database/mydb.
Then you can retrieve the DataSource like so:
Context context = new InitialContext();
DataSource dataSource = (DataSource) context.lookup("java:database/mydb");
This is working on every app server instance and you can deploy your WAR without the need to modify anything.
Conclusion
By moving the configuration to the app server you'll have the advantage of separating settings depending on the environment from your app code. I would prefer this whenever you have settings involving IP addresses, credentials, etc.
Using a static .properties file on the other hand is simpler to manage. I would prefer this option when dealing with settings that have no dependencies to the environment or are app specific.

Environment configuration management?

There is a team develops enterprise application with web interface: java, tomcat, struts, mysql, REST and LDAP calls to external services and so on.
All configuration is stored in context.xml --tomcat specific file that contains variables available via servlet context and object available via JNDI resources.
Developers have no access to production and QA platforms (as it should be) so context.xml is managed by support/sysadmin team.
Each release has config-notes.txt with instructions like:
please add "userLimit" variable to context.xml with value "123", rename "DB" resource to "fooDB" and add new database connection to our new server (you should know url and credentials) named "barDb"
That is not good.
Here is my idea how to solve it.
Each release has special config file with required variable names, descriptions and default values (if any): even web.xml could be used.
Here is pseudo example:
foo=bar
userLimit=123
barDb=SET_MANUAL(connection to our new server)
And there is a special tool that support team runs against deployment artifact.
Look at it (text after ">" is typed by support guy):
Config for version 123 of artifact "mySever".
Enter your config file location> /opt/tomcat/context/myServer.xml
+"foo" value "bar" -- already exists and would not be changed
+"userLimit" value "123" -- adding new
+"barDb"(connection to our new server) please type> jdbc:mysql:host/db
Saving your file as /opt/tomcat/context/myServer.xml
Your environment is not configured to run myServer-123.
That will give us ability to deploy application on any environment and update configuration if needed.
Do you like my idea? What do you use for environment configuration management? Does there is ready-to-use tools for that?

There are plenty of different strategies. All of them are good and depends on what suit you best.
Build a single artifact and deploy configs to a separate location. The artifact could have placeholder variables and, on deployment, the config could be read in. Have a look at Springs property placeholder. It works fantastically for webapps that use Spring and doesn't involve getting ops involved.
Have an externalised property config that lives outside of the webapp. Keep the location constant and always read from the property config. Update the config at any stage and a restart will be up the new values.
If you are modifying the environment (i.e. application server being used or user/group permissions) look at using the above methods with puppet or chef. Also have a look at managing your config files with these tools.
As for the whole should devs be given access to prod, it really depends on a per company basis. For smaller companies where the dev is called every time there is a problem, regardless of whether that problem is server or application related, then obviously devs require access to the box.
DevOps is not about giving devs access to the box, its about giving devs the ability to use infrastructure as a service, the ability to spawn new instances with application X with config Y and to push their applications into environments without ops. In a large company like ours, what it allows is the ability for devs to manage the application they put on a server. Operations shouldn't care what version is on their, thats our job, their job is all about keeping the server up and running.

I strongly disagree with your remark that devs shouldn't have access to prod or staging environments. It's this kind of attitude that leads to teams working against each other instead of with eath other.
But to answer your question: you are thinking about what is typically called continuous integration ( http://en.wikipedia.org/wiki/Continuous_integration ) and moving towards devops. Ideally you should aim for the magic "1 click automated deployment". The guys from Flickr wrote a lot of blogs (and books) about how they achieved that.
Anyhow .. there's a lot of tools around that sector. You may want to have a look a things like Hudson/Jenkins or Puppet/Chef.

Best practice? JNDI, Hibernate and Tomcat

I've got a web application, hosted with tomcat, which uses hibernate to talk to a database.
I'm looking at how I can easy the pain of configuration as I migrate from dev, to test and to prod.
I've seen JNDI mentioned a lot and at first glance it seems like a good idea. You configure a jndi resource on each tomcat instance and the web context just uses it.
However after examining it further it seems that in order to have a JNDI I've got to have all my database objects + hibernate in the tomcat lib files in order for this to work. This sounds scary to me, what if I want to deploy another context that uses a different version of hibernate?
Also, am I not just swapping the pain of maintaining configuration for the pain of breakages caused by mismatches between the installed jndi resource classes and the ones in my context.
Ideally I think what I'm wanting is to just say in tomcat. There is a database called X, it is at this server and has this user/pass.
I'd appreciate your thoughts on the best way to handle the need for different config in different environments without having an extra step after each deploy to update the config files.
Cheers,
Peter

You have confused things a bit, I believe.
JNDI is just a name assigned to a datasource pool. This datasource uses a JDBC driver which in global Tomcat classpath, but that about the only shared resource in the whole setup.
Datasource has connection URL, username, password and options for connections defined, which may differ per server, but application doesn't care about it -- all it knows is the JNDI name, e.g. "jdbc/myDatasource".
All hibernate JARs, and well as any other JARs and whats not are to be packaged within the WAR. They are "visible" only within the WAR, and therefore you can have multiple applications using conflicting versions of libraries deployed to the same Tomcat.
No need to pollute lib/ directory of Tomcat. This is a bad practice, as you correctly observed.

can Web/Meta-Inf/Context.xml read in Tomcat from some properties file

I've context.xml in my web/meta-inf/ folder containing database connection (pool) details. Now i want the database details for my application to be provided by end user in some properties file and context.xml reading the db connection info from the properties file instead of hard coing them in the file directly.
is it possible for tomcat to replace the placeholders from some properties file?
i read about context-manager but not sure where to place that.
Please provide your inputs.
Thanks
Abhishek

You can do it in a better way.
In our case we had different databases for different profiles, like dev, UAT, pre-prod, support etc.
So what I did was, I put my context.xml in its default location <TOMCAT_HOME>/conf.
Had the following in the resource config of context.xml:
<Resource name="jdbc/someDS"
auth="Container"
type="javax.sql.DataSource"
driverClassName="oracle.jdbc.OracleDriver"
url="${appName.db.url}"
username="${appName.db.user}"
password="${appName.db.password}"
maxActive="30"
maxIdle="10"
defaultAutoCommit="true"
maxWait="-1"/>
I created three tomcat servers - server-dev, server-uat, server-support
and passed the corresponding values for each database in the vm arguments of tomcat in eclipse/intellij's tomcat server config (like shown below):
-DappName.db.url=jdbc:oracle:thin:#<DB_SERVER>:1521:<SID> -DappName.db.user=DB_USER -appName.db.password=DB_PASSWORD
The advantage is, the developers then don't need to change the database every-time, they just need to start a particular server. This was very handy and saved a lot of time during our development.

Adding the JNDI resources to Context.xml isn't enough. You will need to define their use in WEB-INF/web.xml also. Check this for a step by step.

As I understand it, your goal is to have application configuration outside of your .war to allow a system administrator to configure the system.
One way to achieve that is to not place the context.xml in your .war file but to distribute that file alongside with your .war.
This file then has to be placed placed in CATALINA_HOME/conf/Catalina/HOSTNAME/APPLICATIONPATH.xml (e.g. CATALINA_HOME/conf/Catalina/localhost/myapp.xml).
This way, the database connection information can be edited directly in the external context configuration file without changing the .war file, you won't need placeholders in that scenario.
This is not the most comfortable way for the user as he/she has to edit an xml file but it should be feasible for most system administrators...
You can find more information on tomcat context configuration at tomcat.apache.org/tomcat-5.5-doc/config/context.html

I'm not sure it's possible to load the details from a properties file but you can have the details in the central server.xml file rather than context.xml. Once you've achieved that, you could probably externalize the connection details using a standard XML entity reference.
Instead of putting your database connection details in the context.xml, put them in the server.xml under a section, and then add a ResourceLink element to the context.xml that creates a link between the global data source and your context.
Unfortunately the documentation is fairly weak for what you're trying to achieve.
Follow the instructions on the "Resource Definitions" section of this page (about half way down), and pay particular attention to the (tiny) ResourceLink section below it:
http://tomcat.apache.org/tomcat-5.5-doc/config/globalresources.html

If I don't have database connection information at build time, I'll just configure a resource reference and tell the admin what name they need to give the connection pool. They can configure everything in the Tomcat admin console and I never have to know about it. This also makes it easier to build a single WAR file and use it in multiple environments since the database configuration details aren't part of the WAR.
See also:
Apache Tomcat 6.0 JNDI Resources
If you do know the details at build time and want to bake them into the WAR, try Ant filters.